{ "id": 54759, "title": "Thank you and Credits for the images used in this challenge", "url": "/competitions/data-science-bowl-2018/discussion/54759", "author_name": "", "post_date": "2018-04-17T13:37:56.321733200Z", "votes": 19, "comment_count": 15, "views": 0, "content": "

This challenge could not have happened without the efforts of so many!

\n\n

Big thanks to all you competitors, who made it so interesting and educational and positive (and competitive!) You are making the world a better place by encouraging so many to work on challenging puzzles like these (especially you, @Heng CherKeng!)

\n\n

As noted in this thread (https://www.kaggle.com/c/data-science-bowl-2018/discussion/47864), all contributors of images to this challenge graciously provided them as CC0 (public domain) so that others could freely make use of them in the future. This is really phenomenal; it makes life so much easier for researchers, and we do not take it for granted!

\n\n

Now that the challenge is over, we want to publicly thank the until-now mystery contributors of these images. They are:

\n\n\n\n

Now, the Kaggle & Booz Allen teams did the major organization and logistics. My entire lab did the annotations. Which took a really long time, to say the least. Allen Goodman and Juan Caicedo designed the metric. Allen answered some of your questions on the forum. Kyle Karhohs in my lab was the chief organizer of the images, gathering them from the biological community and organizing the annotation effort, which was no small feat!

\n\n

This was an intense experience and I'm looking forward to the announcement of the official winners within a few weeks!

\n\n

UPDATE (how to cite the data set): The data set will live on at the BBBC, Broad Institute Broad Bioimage Benchmark Collection; we plan to release updated versions as well, with corrections: https://data.broadinstitute.org/bbbc/BBBC038/ Until the data set is published in a paper, please follow citation instructions there!

", "messages": [ { "id": "315693", "postDate": "04/17/2018 13:37:56", "content": "

This challenge could not have happened without the efforts of so many!

\n\n

Big thanks to all you competitors, who made it so interesting and educational and positive (and competitive!) You are making the world a better place by encouraging so many to work on challenging puzzles like these (especially you, @Heng CherKeng!)

\n\n

As noted in this thread (https://www.kaggle.com/c/data-science-bowl-2018/discussion/47864), all contributors of images to this challenge graciously provided them as CC0 (public domain) so that others could freely make use of them in the future. This is really phenomenal; it makes life so much easier for researchers, and we do not take it for granted!

\n\n

Now that the challenge is over, we want to publicly thank the until-now mystery contributors of these images. They are:

\n\n\n\n

Now, the Kaggle & Booz Allen teams did the major organization and logistics. My entire lab did the annotations. Which took a really long time, to say the least. Allen Goodman and Juan Caicedo designed the metric. Allen answered some of your questions on the forum. Kyle Karhohs in my lab was the chief organizer of the images, gathering them from the biological community and organizing the annotation effort, which was no small feat!

\n\n

This was an intense experience and I'm looking forward to the announcement of the official winners within a few weeks!

\n\n

UPDATE (how to cite the data set): The data set will live on at the BBBC, Broad Institute Broad Bioimage Benchmark Collection; we plan to release updated versions as well, with corrections: https://data.broadinstitute.org/bbbc/BBBC038/ Until the data set is published in a paper, please follow citation instructions there!

", "rawMarkdown": "This challenge could not have happened without the efforts of so many! \n\nBig thanks to all you competitors, who made it so interesting and educational and positive (and competitive!) You are making the world a better place by encouraging so many to work on challenging puzzles like these (especially you, @Heng CherKeng!)\n\nAs noted in this thread (https://www.kaggle.com/c/data-science-bowl-2018/discussion/47864), all contributors of images to this challenge graciously provided them as CC0 (public domain) so that others could freely make use of them in the future. This is really phenomenal; it makes life so much easier for researchers, and we do not take it for granted!\n\nNow that the challenge is over, we want to publicly thank the until-now mystery contributors of these images. They are:\n\n* Riki Eggert, King's College London\n* Donna McPhie, McLean Hospital\n* Andrew Bradley and Gustavo Carneiro\n* Mariko Taga, Columbia University\n* Matthew Stachler, Brigham and Women's Hospital\n* Chris Lee, MIT\n* Alexander Chamessian, Ji Lab, Duke University\n* Florian Barthelemy, Miceli Lab, Center for Duchenne muscular dystrophy, UCLA\n* Lorraine Montel, Ecole Normale Superieure\n* Glyn Nelson, Newcastle University\n* Tim Becker, Fraunhofer EMB\n* Maria Frias, Foster Lab, Hunter College\n* Philipp Keller\n* Christian Marinaccio, Northwestern University\n* Vasiliy Chernyshev, Skoltech\n* several biologists who wished to remain anonymous\n* And of course, the Carpenter lab, Broad Institute\n\nNow, the Kaggle & Booz Allen teams did the major organization and logistics. My entire lab did the annotations. Which took a really long time, to say the least. Allen Goodman and Juan Caicedo designed the metric. Allen answered some of your questions on the forum. Kyle Karhohs in my lab was the chief organizer of the images, gathering them from the biological community and organizing the annotation effort, which was no small feat!\n\nThis was an intense experience and I'm looking forward to the announcement of the official winners within a few weeks! \n\nUPDATE (how to cite the data set): The data set will live on at the BBBC, Broad Institute Broad Bioimage Benchmark Collection; we plan to release updated versions as well, with corrections: https://data.broadinstitute.org/bbbc/BBBC038/ Until the data set is published in a paper, please follow citation instructions there!", "votes": null }, { "id": "315725", "postDate": "04/17/2018 14:22:16", "content": "

Nice!

\n\n

Kyle Karhohs was amazing. He helped with every part of the competition and made sure everything went smoothly.

", "rawMarkdown": "Nice!\n\nKyle Karhohs was amazing. He helped with **every** part of the competition and made sure everything went smoothly.", "votes": null }, { "id": "315822", "postDate": "04/17/2018 16:27:55", "content": "

Anne, thank you, your team and kaggle team so much for this great competition. In my experience in terms of potential application of the winning results to real systems (may be even in cellprofiler) this is definitely the best competition

\n\n

Recently in https://www.reddit.com/r/science/comments/7xpwc0/science_ama_series_dr_anne_carpenter_institute/duakmjr/, you mentioned

\n\n\n\n

Is it still valid? :)

", "rawMarkdown": "Anne, thank you, your team and kaggle team so much for this great competition. In my experience in terms of potential application of the winning results to real systems (may be even in cellprofiler) this is definitely the best competition\n\nRecently in https://www.reddit.com/r/science/comments/7xpwc0/science_ama_series_dr_anne_carpenter_institute/duakmjr/, you mentioned \n\n- How well should one's Kaggle model perform in order to get hired into your research group without references, an education, or a background check?\n\n- You can have Allen Goodman’s job if you beat his score on the leaderboard :D\n\nIs it still valid? :)", "votes": null }, { "id": "315956", "postDate": "04/17/2018 21:52:30", "content": "

Thanks for sharing! Could you please tell more about annotation: did you use CellProfiler? how many annotations per one image were made (different people)? how much time was spent per one image on average? will you share true labels for the stage 2 test set?

", "rawMarkdown": "Thanks for sharing! Could you please tell more about annotation: did you use CellProfiler? how many annotations per one image were made (different people)? how much time was spent per one image on average? will you share true labels for the stage 2 test set?", "votes": null }, { "id": "315980", "postDate": "04/17/2018 23:32:48", "content": "

I think some teams have annotated external data as well, would be great for this effort to be combined with this dataset as well.

\n\n

We have developed a simple annotation tool and annotated relatively small number of images (comparing to the training dataset) but we would be happy to share it under the CC0 license. We have been using the open data shared in the forum thread, but it's still necessary to check datasets individually for license for original images, I'd expect some may be more restrictive comparing to CC0.

\n\n

We also found adding relatively small number of images of the new type (around 10 or so) significantly improves prediction quality for other images of the similar type with our model (UNET based, deep watershed transform like with centers predictions. Sorry the forum post still pending). While I don't think we are reached the state to have an universal nuclei detector, it's possible to fine tune the model to the new image type with the reasonable effort. Integrating the raw model predictions (masks, watershed energy levels, predicted centers, places where multiple models disagree) and processing algorithms with the annotation tool would make the annotations process even easier/faster.

\n\n

If results of such fine tuning annotations are shared between groups under the open license, this would allow to achieve the goal of having an universal nuclei detector much close.

\n\n

And thanks for organizing the competition, it was a great pleasure to participate.

", "rawMarkdown": "I think some teams have annotated external data as well, would be great for this effort to be combined with this dataset as well.\n\nWe have developed a simple annotation tool and annotated relatively small number of images (comparing to the training dataset) but we would be happy to share it under the CC0 license. We have been using the open data shared in the forum thread, but it's still necessary to check datasets individually for license for original images, I'd expect some may be more restrictive comparing to CC0.\n\nWe also found adding relatively small number of images of the new type (around 10 or so) significantly improves prediction quality for other images of the similar type with our model (UNET based, deep watershed transform like with centers predictions. Sorry the forum post still pending). While I don't think we are reached the state to have an universal nuclei detector, it's possible to fine tune the model to the new image type with the reasonable effort. Integrating the raw model predictions (masks, watershed energy levels, predicted centers, places where multiple models disagree) and processing algorithms with the annotation tool would make the annotations process even easier/faster.\n\nIf results of such fine tuning annotations are shared between groups under the open license, this would allow to achieve the goal of having an universal nuclei detector much close.\n\nAnd thanks for organizing the competition, it was a great pleasure to participate.", "votes": null }, { "id": "316026", "postDate": "04/18/2018 03:19:01", "content": "

Thanks to the organizers (especially Anne and Allen) for putting on an inspiring competition. I've never worked with this data before (never even took bio 101), so for me it was from 0 to 90 in 90 days.

\n\n

Without the contribution of images, this contest doesn't happen, so thanks to them as well.

\n\n

The community was great. Inspired first (and probably most) by @Allen's original post and open answering of questions and @Heng's boundless energy (he is an idea machine). @Anne for putting up with all the complaining from the peanut gallery, while cheering the contributors on. (She may have single handedly won the argument for emoji's on Kaggle.)

\n\n

Finally 3 cheers for the winners with a creative application of some non-obvious techniques and what looks to me to be a fantastic score.

", "rawMarkdown": "Thanks to the organizers (especially Anne and Allen) for putting on an inspiring competition. I've never worked with this data before (never even took bio 101), so for me it was from 0 to 90 in 90 days. \n\nWithout the contribution of images, this contest doesn't happen, so thanks to them as well.\n\nThe community was great. Inspired first (and probably most) by @Allen's original post and open answering of questions and @Heng's boundless energy (he is an idea machine). @Anne for putting up with all the complaining from the peanut gallery, while cheering the contributors on. (She may have single handedly won the argument for emoji's on Kaggle.)\n\nFinally 3 cheers for the winners with a creative application of some non-obvious techniques and what looks to me to be a fantastic score.", "votes": null }, { "id": "316279", "postDate": "04/18/2018 15:53:02", "content": "

A great thanks to the organizers, @Anne, @Allen and members from Kaggle and Booz Allen teams for organizing the competition. My team and I enjoyed and learned a lot from it. Looking forward to the next competition from you guys!

", "rawMarkdown": "A great thanks to the organizers, @Anne, @Allen and members from Kaggle and Booz Allen teams for organizing the competition. My team and I enjoyed and learned a lot from it. Looking forward to the next competition from you guys!", "votes": null }, { "id": "316333", "postDate": "04/18/2018 18:58:47", "content": "

Nice work!

", "rawMarkdown": "Nice work!", "votes": null }, { "id": "316616", "postDate": "04/19/2018 14:05:26", "content": "

That would be wonderful!

\n\n

We would definitely welcome more CC0 images. Where do you suggest placing them/organizing them? Our BBBC is one option but it's really just a simple prototype; the NEUBIAS community is working on a better database for images+annotations for biology.

\n\n

In our experience, it's very difficult to find images that are CC0 (most require credit of some sort - which is sensible from an academic point of view, but a nightmare if we try to collect and distribute images from hundreds of labs!)

", "rawMarkdown": "That would be wonderful! \n\nWe would definitely welcome more CC0 images. Where do you suggest placing them/organizing them? Our BBBC is one option but it's really just a simple prototype; the NEUBIAS community is working on a better database for images+annotations for biology. \n\nIn our experience, it's very difficult to find images that are CC0 (most require credit of some sort - which is sensible from an academic point of view, but a nightmare if we try to collect and distribute images from hundreds of labs!)", "votes": null }, { "id": "316618", "postDate": "04/19/2018 14:05:56", "content": "

Aw, thanks! :blush:

", "rawMarkdown": "Aw, thanks! :blush:", "votes": null }, { "id": "316625", "postDate": "04/19/2018 14:22:30", "content": "

We are debating whether to release true labels here: https://www.kaggle.com/c/data-science-bowl-2018/discussion/54764#315764

\n\n

I just wrote a post to describe the annotation strategy here: \nhttps://www.kaggle.com/c/data-science-bowl-2018/discussion/54900

", "rawMarkdown": "We are debating whether to release true labels here: https://www.kaggle.com/c/data-science-bowl-2018/discussion/54764#315764\n\nI just wrote a post to describe the annotation strategy here: \nhttps://www.kaggle.com/c/data-science-bowl-2018/discussion/54900", "votes": null }, { "id": "316629", "postDate": "04/19/2018 14:30:05", "content": "

No, we think it best that Allen stay employed; despite his avatar always looking disgusted with us, he's a good guy and put up with a lot during this challenge, so he's resilient too!

\n\n

Keep an eye on the CellProfiler forum for job openings in the field: forum.cellprofiler.org

", "rawMarkdown": "No, we think it best that Allen stay employed; despite his avatar always looking disgusted with us, he's a good guy and put up with a lot during this challenge, so he's resilient too! \n\nKeep an eye on the CellProfiler forum for job openings in the field: forum.cellprofiler.org", "votes": null }, { "id": "316920", "postDate": "04/20/2018 07:35:32", "content": "

@Anne @Allen Thank you so much for organizing such an amazing event! We have had a great fun joining it. This event was really great because all the talented and skillful engineers and scientists discussed and improved many various methods to achieve better results. Personally, I have learned a lot of new skills and ideas here.

", "rawMarkdown": "Anne @Allen Thank you so much for organizing such an amazing event! We have had a great fun joining it. This event was really great because all the talented and skillful engineers and scientists discussed and improved many various methods to achieve better results. Personally, I have learned a lot of new skills and ideas here.", "votes": null }, { "id": "318893", "postDate": "04/24/2018 17:37:43", "content": "

Is there a formal citation for the dataset, in case it is used for published academic research? I want to give credit where it's due!

", "rawMarkdown": "Is there a formal citation for the dataset, in case it is used for published academic research? I want to give credit where it's due!", "votes": null }, { "id": "318973", "postDate": "04/24/2018 23:40:47", "content": "

Anne thank you very much for organizing one of the most intense and enjoyable contest on Kaggle. The dataset is amazing in that it balanced usability and usefulness all in a small enough size for experimentation.

\n\n

I want to give a shout out to @Heng for all his knowledge sharing and @Konstantin lopuhin for helping with a processed data for those of us using paid GPU to save time and money. Thanks to all the people that shared really useful research papers and data sources.

\n\n

Once again Allen and Anne, thanks for answering our never ending questions :-)

", "rawMarkdown": "Anne thank you very much for organizing one of the most intense and enjoyable contest on Kaggle. The dataset is amazing in that it balanced usability and usefulness all in a small enough size for experimentation. \n\nI want to give a shout out to @Heng for all his knowledge sharing and @Konstantin lopuhin for helping with a processed data for those of us using paid GPU to save time and money. Thanks to all the people that shared really useful research papers and data sources.\n\nOnce again Allen and Anne, thanks for answering our never ending questions :-)", "votes": null }, { "id": "319739", "postDate": "04/26/2018 18:43:47", "content": "

Thank you for asking! The data set will live at the BBBC, Broad Institute\nBroad Bioimage Benchmark Collection; we plan to release updated versions as well, with corrections: https://data.broadinstitute.org/bbbc/BBBC038/

", "rawMarkdown": "Thank you for asking! The data set will live at the BBBC, Broad Institute\nBroad Bioimage Benchmark Collection; we plan to release updated versions as well, with corrections: https://data.broadinstitute.org/bbbc/BBBC038/", "votes": null } ], "comments": [ { "id": 315725, "author_name": "agoodman", "author_url": "", "post_date": "04/17/2018 14:22:16", "content": "

Nice!

\n\n

Kyle Karhohs was amazing. He helped with every part of the competition and made sure everything went smoothly.

", "votes": null, "replies": [] }, { "id": 315822, "author_name": "shvetsiya", "author_url": "", "post_date": "04/17/2018 16:27:55", "content": "

Anne, thank you, your team and kaggle team so much for this great competition. In my experience in terms of potential application of the winning results to real systems (may be even in cellprofiler) this is definitely the best competition

\n\n

Recently in https://www.reddit.com/r/science/comments/7xpwc0/science_ama_series_dr_anne_carpenter_institute/duakmjr/, you mentioned

\n\n\n\n

Is it still valid? :)

", "votes": null, "replies": [ { "id": 316629, "author_name": "drannecarpenter", "author_url": "", "post_date": "04/19/2018 14:30:05", "content": "

No, we think it best that Allen stay employed; despite his avatar always looking disgusted with us, he's a good guy and put up with a lot during this challenge, so he's resilient too!

\n\n

Keep an eye on the CellProfiler forum for job openings in the field: forum.cellprofiler.org

", "votes": null, "replies": [] } ] }, { "id": 315956, "author_name": "", "author_url": "", "post_date": "04/17/2018 21:52:30", "content": "

Thanks for sharing! Could you please tell more about annotation: did you use CellProfiler? how many annotations per one image were made (different people)? how much time was spent per one image on average? will you share true labels for the stage 2 test set?

", "votes": null, "replies": [ { "id": 316625, "author_name": "drannecarpenter", "author_url": "", "post_date": "04/19/2018 14:22:30", "content": "

We are debating whether to release true labels here: https://www.kaggle.com/c/data-science-bowl-2018/discussion/54764#315764

\n\n

I just wrote a post to describe the annotation strategy here: \nhttps://www.kaggle.com/c/data-science-bowl-2018/discussion/54900

", "votes": null, "replies": [] } ] }, { "id": 315980, "author_name": "dmytropoplavskiy", "author_url": "", "post_date": "04/17/2018 23:32:48", "content": "

I think some teams have annotated external data as well, would be great for this effort to be combined with this dataset as well.

\n\n

We have developed a simple annotation tool and annotated relatively small number of images (comparing to the training dataset) but we would be happy to share it under the CC0 license. We have been using the open data shared in the forum thread, but it's still necessary to check datasets individually for license for original images, I'd expect some may be more restrictive comparing to CC0.

\n\n

We also found adding relatively small number of images of the new type (around 10 or so) significantly improves prediction quality for other images of the similar type with our model (UNET based, deep watershed transform like with centers predictions. Sorry the forum post still pending). While I don't think we are reached the state to have an universal nuclei detector, it's possible to fine tune the model to the new image type with the reasonable effort. Integrating the raw model predictions (masks, watershed energy levels, predicted centers, places where multiple models disagree) and processing algorithms with the annotation tool would make the annotations process even easier/faster.

\n\n

If results of such fine tuning annotations are shared between groups under the open license, this would allow to achieve the goal of having an universal nuclei detector much close.

\n\n

And thanks for organizing the competition, it was a great pleasure to participate.

", "votes": null, "replies": [ { "id": 316616, "author_name": "drannecarpenter", "author_url": "", "post_date": "04/19/2018 14:05:26", "content": "

That would be wonderful!

\n\n

We would definitely welcome more CC0 images. Where do you suggest placing them/organizing them? Our BBBC is one option but it's really just a simple prototype; the NEUBIAS community is working on a better database for images+annotations for biology.

\n\n

In our experience, it's very difficult to find images that are CC0 (most require credit of some sort - which is sensible from an academic point of view, but a nightmare if we try to collect and distribute images from hundreds of labs!)

", "votes": null, "replies": [] } ] }, { "id": 316026, "author_name": "jbfarrar", "author_url": "", "post_date": "04/18/2018 03:19:01", "content": "

Thanks to the organizers (especially Anne and Allen) for putting on an inspiring competition. I've never worked with this data before (never even took bio 101), so for me it was from 0 to 90 in 90 days.

\n\n

Without the contribution of images, this contest doesn't happen, so thanks to them as well.

\n\n

The community was great. Inspired first (and probably most) by @Allen's original post and open answering of questions and @Heng's boundless energy (he is an idea machine). @Anne for putting up with all the complaining from the peanut gallery, while cheering the contributors on. (She may have single handedly won the argument for emoji's on Kaggle.)

\n\n

Finally 3 cheers for the winners with a creative application of some non-obvious techniques and what looks to me to be a fantastic score.

", "votes": null, "replies": [ { "id": 316618, "author_name": "drannecarpenter", "author_url": "", "post_date": "04/19/2018 14:05:56", "content": "

Aw, thanks! :blush:

", "votes": null, "replies": [] } ] }, { "id": 316279, "author_name": "hengck23", "author_url": "", "post_date": "04/18/2018 15:53:02", "content": "

A great thanks to the organizers, @Anne, @Allen and members from Kaggle and Booz Allen teams for organizing the competition. My team and I enjoyed and learned a lot from it. Looking forward to the next competition from you guys!

", "votes": null, "replies": [] }, { "id": 316333, "author_name": "thatguyaryan", "author_url": "", "post_date": "04/18/2018 18:58:47", "content": "

Nice work!

", "votes": null, "replies": [] }, { "id": 316920, "author_name": "edmoon", "author_url": "", "post_date": "04/20/2018 07:35:32", "content": "

@Anne @Allen Thank you so much for organizing such an amazing event! We have had a great fun joining it. This event was really great because all the talented and skillful engineers and scientists discussed and improved many various methods to achieve better results. Personally, I have learned a lot of new skills and ideas here.

", "votes": null, "replies": [] }, { "id": 318893, "author_name": "jwjohnson314", "author_url": "", "post_date": "04/24/2018 17:37:43", "content": "

Is there a formal citation for the dataset, in case it is used for published academic research? I want to give credit where it's due!

", "votes": null, "replies": [ { "id": 319739, "author_name": "drannecarpenter", "author_url": "", "post_date": "04/26/2018 18:43:47", "content": "

Thank you for asking! The data set will live at the BBBC, Broad Institute\nBroad Bioimage Benchmark Collection; we plan to release updated versions as well, with corrections: https://data.broadinstitute.org/bbbc/BBBC038/

", "votes": null, "replies": [] } ] }, { "id": 318973, "author_name": "sheriytm", "author_url": "", "post_date": "04/24/2018 23:40:47", "content": "

Anne thank you very much for organizing one of the most intense and enjoyable contest on Kaggle. The dataset is amazing in that it balanced usability and usefulness all in a small enough size for experimentation.

\n\n

I want to give a shout out to @Heng for all his knowledge sharing and @Konstantin lopuhin for helping with a processed data for those of us using paid GPU to save time and money. Thanks to all the people that shared really useful research papers and data sources.

\n\n

Once again Allen and Anne, thanks for answering our never ending questions :-)

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "315693": "This challenge could not have happened without the efforts of so many! \n\nBig thanks to all you competitors, who made it so interesting and educational and positive (and competitive!) You are making the world a better place by encouraging so many to work on challenging puzzles like these (especially you, @Heng CherKeng!)\n\nAs noted in this thread (https://www.kaggle.com/c/data-science-bowl-2018/discussion/47864), all contributors of images to this challenge graciously provided them as CC0 (public domain) so that others could freely make use of them in the future. This is really phenomenal; it makes life so much easier for researchers, and we do not take it for granted!\n\nNow that the challenge is over, we want to publicly thank the until-now mystery contributors of these images. They are:\n\n* Riki Eggert, King's College London\n* Donna McPhie, McLean Hospital\n* Andrew Bradley and Gustavo Carneiro\n* Mariko Taga, Columbia University\n* Matthew Stachler, Brigham and Women's Hospital\n* Chris Lee, MIT\n* Alexander Chamessian, Ji Lab, Duke University\n* Florian Barthelemy, Miceli Lab, Center for Duchenne muscular dystrophy, UCLA\n* Lorraine Montel, Ecole Normale Superieure\n* Glyn Nelson, Newcastle University\n* Tim Becker, Fraunhofer EMB\n* Maria Frias, Foster Lab, Hunter College\n* Philipp Keller\n* Christian Marinaccio, Northwestern University\n* Vasiliy Chernyshev, Skoltech\n* several biologists who wished to remain anonymous\n* And of course, the Carpenter lab, Broad Institute\n\nNow, the Kaggle & Booz Allen teams did the major organization and logistics. My entire lab did the annotations. Which took a really long time, to say the least. Allen Goodman and Juan Caicedo designed the metric. Allen answered some of your questions on the forum. Kyle Karhohs in my lab was the chief organizer of the images, gathering them from the biological community and organizing the annotation effort, which was no small feat!\n\nThis was an intense experience and I'm looking forward to the announcement of the official winners within a few weeks! \n\nUPDATE (how to cite the data set): The data set will live on at the BBBC, Broad Institute Broad Bioimage Benchmark Collection; we plan to release updated versions as well, with corrections: https://data.broadinstitute.org/bbbc/BBBC038/ Until the data set is published in a paper, please follow citation instructions there!", "315725": "Nice!\n\nKyle Karhohs was amazing. He helped with **every** part of the competition and made sure everything went smoothly.", "315822": "Anne, thank you, your team and kaggle team so much for this great competition. In my experience in terms of potential application of the winning results to real systems (may be even in cellprofiler) this is definitely the best competition\n\nRecently in https://www.reddit.com/r/science/comments/7xpwc0/science_ama_series_dr_anne_carpenter_institute/duakmjr/, you mentioned \n\n- How well should one's Kaggle model perform in order to get hired into your research group without references, an education, or a background check?\n\n- You can have Allen Goodman’s job if you beat his score on the leaderboard :D\n\nIs it still valid? :)", "315956": "Thanks for sharing! Could you please tell more about annotation: did you use CellProfiler? how many annotations per one image were made (different people)? how much time was spent per one image on average? will you share true labels for the stage 2 test set?", "315980": "I think some teams have annotated external data as well, would be great for this effort to be combined with this dataset as well.\n\nWe have developed a simple annotation tool and annotated relatively small number of images (comparing to the training dataset) but we would be happy to share it under the CC0 license. We have been using the open data shared in the forum thread, but it's still necessary to check datasets individually for license for original images, I'd expect some may be more restrictive comparing to CC0.\n\nWe also found adding relatively small number of images of the new type (around 10 or so) significantly improves prediction quality for other images of the similar type with our model (UNET based, deep watershed transform like with centers predictions. Sorry the forum post still pending). While I don't think we are reached the state to have an universal nuclei detector, it's possible to fine tune the model to the new image type with the reasonable effort. Integrating the raw model predictions (masks, watershed energy levels, predicted centers, places where multiple models disagree) and processing algorithms with the annotation tool would make the annotations process even easier/faster.\n\nIf results of such fine tuning annotations are shared between groups under the open license, this would allow to achieve the goal of having an universal nuclei detector much close.\n\nAnd thanks for organizing the competition, it was a great pleasure to participate.", "316026": "Thanks to the organizers (especially Anne and Allen) for putting on an inspiring competition. I've never worked with this data before (never even took bio 101), so for me it was from 0 to 90 in 90 days. \n\nWithout the contribution of images, this contest doesn't happen, so thanks to them as well.\n\nThe community was great. Inspired first (and probably most) by @Allen's original post and open answering of questions and @Heng's boundless energy (he is an idea machine). @Anne for putting up with all the complaining from the peanut gallery, while cheering the contributors on. (She may have single handedly won the argument for emoji's on Kaggle.)\n\nFinally 3 cheers for the winners with a creative application of some non-obvious techniques and what looks to me to be a fantastic score.", "316279": "A great thanks to the organizers, @Anne, @Allen and members from Kaggle and Booz Allen teams for organizing the competition. My team and I enjoyed and learned a lot from it. Looking forward to the next competition from you guys!", "316333": "Nice work!", "316616": "That would be wonderful! \n\nWe would definitely welcome more CC0 images. Where do you suggest placing them/organizing them? Our BBBC is one option but it's really just a simple prototype; the NEUBIAS community is working on a better database for images+annotations for biology. \n\nIn our experience, it's very difficult to find images that are CC0 (most require credit of some sort - which is sensible from an academic point of view, but a nightmare if we try to collect and distribute images from hundreds of labs!)", "316618": "Aw, thanks! :blush:", "316625": "We are debating whether to release true labels here: https://www.kaggle.com/c/data-science-bowl-2018/discussion/54764#315764\n\nI just wrote a post to describe the annotation strategy here: \nhttps://www.kaggle.com/c/data-science-bowl-2018/discussion/54900", "316629": "No, we think it best that Allen stay employed; despite his avatar always looking disgusted with us, he's a good guy and put up with a lot during this challenge, so he's resilient too! \n\nKeep an eye on the CellProfiler forum for job openings in the field: forum.cellprofiler.org", "316920": "Anne @Allen Thank you so much for organizing such an amazing event! We have had a great fun joining it. This event was really great because all the talented and skillful engineers and scientists discussed and improved many various methods to achieve better results. Personally, I have learned a lot of new skills and ideas here.", "318893": "Is there a formal citation for the dataset, in case it is used for published academic research? I want to give credit where it's due!", "318973": "Anne thank you very much for organizing one of the most intense and enjoyable contest on Kaggle. The dataset is amazing in that it balanced usability and usefulness all in a small enough size for experimentation. \n\nI want to give a shout out to @Heng for all his knowledge sharing and @Konstantin lopuhin for helping with a processed data for those of us using paid GPU to save time and money. Thanks to all the people that shared really useful research papers and data sources.\n\nOnce again Allen and Anne, thanks for answering our never ending questions :-)", "319739": "Thank you for asking! The data set will live at the BBBC, Broad Institute\nBroad Bioimage Benchmark Collection; we plan to release updated versions as well, with corrections: https://data.broadinstitute.org/bbbc/BBBC038/" }, "source": "meta" }