What do you think, should we release the list of stage 2 test images used for scoring?
\n\nPro: It seems like it would be useful for competitors to digest how/why their model did well (e.g., https://www.kaggle.com/c/data-science-bowl-2018/discussion/54739). And also make people feel better about their model performing badly on poison images that we would not intend a winning model to work nicely on. We'd include metadata so you know which images come from the same lab/experiment.
\n\nCon: It might be nice to keep it secret if we did want to rerun this competition or make it an ongoing thing (can Kaggle make it an ongoing leaderboard, for example?)
\n\nSo the choices are:
\n\nWhat do you think, should we release the list of stage 2 test images used for scoring?
\n\nPro: It seems like it would be useful for competitors to digest how/why their model did well (e.g., https://www.kaggle.com/c/data-science-bowl-2018/discussion/54739). And also make people feel better about their model performing badly on poison images that we would not intend a winning model to work nicely on. We'd include metadata so you know which images come from the same lab/experiment.
\n\nCon: It might be nice to keep it secret if we did want to rerun this competition or make it an ongoing thing (can Kaggle make it an ongoing leaderboard, for example?)
\n\nSo the choices are:
\n\nMy initial inclination is towards 1 in order to maximize knowledge gain by everyone right now, and who knows where that might lead. Would you be willing to share any thoughts on how you see things playing out from here both short and long-term? What applications appear most important, relevant, and promising with newfound deep learning results? We all would like to learn where state-of-the-art lies now with these kinds of images and it seems the stage 2 set is diverse enough to provide some nice \"watershed\" cases. In what kinds of images are the best methods really improving things and exactly where are they breaking down and making errors? For example, our unet-based model is currently doing poorly in the speckled / granular cases in which there can be a global scale resolution decision; that is, is the picture of three nuclei or several hundred. By retraining with updated data we could very likely do much better on them, but how important are such gains? At this time does it make sense to perhaps move towards building better models for specific staining and microscope modalities or do we press on further towards a one-model-fits-all global nuclei finder?
", "rawMarkdown": "My initial inclination is towards 1 in order to maximize knowledge gain by everyone right now, and who knows where that might lead. Would you be willing to share any thoughts on how you see things playing out from here both short and long-term? What applications appear most important, relevant, and promising with newfound deep learning results? We all would like to learn where state-of-the-art lies now with these kinds of images and it seems the stage 2 set is diverse enough to provide some nice \"watershed\" cases. In what kinds of images are the best methods really improving things and exactly where are they breaking down and making errors? For example, our unet-based model is currently doing poorly in the speckled / granular cases in which there can be a global scale resolution decision; that is, is the picture of three nuclei or several hundred. By retraining with updated data we could very likely do much better on them, but how important are such gains? At this time does it make sense to perhaps move towards building better models for specific staining and microscope modalities or do we press on further towards a one-model-fits-all global nuclei finder?", "votes": 4, "replies": [ { "id": 315931, "postDate": "2018-04-17T20:32:13.293Z", "content": "yes. having metadata? with each image is useful. what technique used to capture image, etc.
", "rawMarkdown": "yes. having metadata? with each image is useful. what technique used to capture image, etc.\n" }, { "id": 316207, "postDate": "2018-04-18T12:25:00.720Z", "content": "@Russ, those are interesting points.
", "rawMarkdown": "@Russ, those are interesting points. " }, { "id": 316677, "postDate": "2018-04-19T16:36:34.097Z", "content": "Agree! And we don't have solid answers to them, yet. I'm eager to dig into the winning results and see whether there is a subset of experimental types that we can consider reasonably solved with this round of solutions.
", "rawMarkdown": "Agree! And we don't have solid answers to them, yet. I'm eager to dig into the winning results and see whether there is a subset of experimental types that we can consider reasonably solved with this round of solutions." } ] }, { "id": 315721, "postDate": "2018-04-17T14:19:38.537Z", "content": "What about releasing the images (i.e. removing the poison images) but not releasing the annotations?
", "rawMarkdown": "What about releasing the images (i.e. removing the poison images) but not releasing the annotations?", "votes": 2, "replies": [ { "id": 315735, "postDate": "2018-04-17T14:31:36.143Z", "content": "Great point, will edit my post to add that in case this thread gets messy.
", "rawMarkdown": "Great point, will edit my post to add that in case this thread gets messy.", "votes": 1 }, { "id": 315749, "postDate": "2018-04-17T14:39:54.833Z", "content": "Usually when kaggle comps. are left open test labels (or anotations) are not released, so the submission process and feedback through scores is the same as in competition time. Also releasing true images list does not necessarily imply to remove them from test... I guess it will just depend on how you want to shape that future \"ongoin comp\".
", "rawMarkdown": "Usually when kaggle comps. are left open test labels (or anotations) are not released, so the submission process and feedback through scores is the same as in competition time. Also releasing true images list does not necessarily imply to remove them from test... I guess it will just depend on how you want to shape that future \"ongoin comp\".", "votes": 1 } ] }, { "id": 405978, "postDate": "2018-10-18T12:57:17.423Z", "content": "Hi everyone! \nThe data is now released here: https://data.broadinstitute.org/bbbc/BBBC038/\nWe do still plan to release another version, with many corrections to the data. Keep an eye on that page!
", "rawMarkdown": "Hi everyone! \nThe data is now released here: https://data.broadinstitute.org/bbbc/BBBC038/\nWe do still plan to release another version, with many corrections to the data. Keep an eye on that page!", "votes": 1, "replies": [ { "id": 406001, "postDate": "2018-10-18T13:34:28.403Z", "content": "Thanks Anne.
", "rawMarkdown": "Thanks Anne." } ] }, { "id": 319379, "postDate": "2018-04-25T23:10:45.187Z", "content": "I would vote for 1. There should be totally new images/annotations for another competition down the road.
", "rawMarkdown": "I would vote for 1. There should be totally new images/annotations for another competition down the road.", "votes": 1 }, { "id": 316274, "postDate": "2018-04-18T15:45:43.473Z", "content": "@Anne Carpenter Thank you for setting up the discussion. I would opt for 2. Actually, kaggler can always probe the data and know the \"estimated ground truth\" for all the options 2 and 3.
\n\nIf there is an intention of \"rerun this competition or make it an ongoing thing\", I suggest additional of some new black box data at the next round. Once \"black box data\" is seen, it become \"not so black\" anymore.
", "rawMarkdown": "@Anne Carpenter Thank you for setting up the discussion. I would opt for 2. Actually, kaggler can always probe the data and know the \"estimated ground truth\" for all the options 2 and 3.\n\nIf there is an intention of \"rerun this competition or make it an ongoing thing\", I suggest additional of some new black box data at the next round. Once \"black box data\" is seen, it become \"not so black\" anymore.", "votes": 1, "replies": [ { "id": 316676, "postDate": "2018-04-19T16:35:01.837Z", "content": "It's true, if we did another competition we absolutely would want to use only fresh images from new experiments anyway.
", "rawMarkdown": "It's true, if we did another competition we absolutely would want to use only fresh images from new experiments anyway. ", "votes": 1 } ] }, { "id": 315766, "postDate": "2018-04-17T15:09:33Z", "content": "There is a pro in leaving poison images secret as well. Some folks may want to perform unsupervised learning and try to identify which are the poisoned images and which are not. Albeit not the original intent of this competition, it is also an interesting vision machine learning topic.
", "rawMarkdown": "There is a pro in leaving poison images secret as well. Some folks may want to perform unsupervised learning and try to identify which are the poisoned images and which are not. Albeit not the original intent of this competition, it is also an interesting vision machine learning topic. ", "votes": 1 }, { "id": 315764, "postDate": "2018-04-17T15:07:54.123Z", "content": "Support Option 2 with an ongoing contest if possible. Note that if image identities are not released, but this is a made an ongoing contest, everyone is going to LB probe to find out which images change the score.
", "rawMarkdown": "Support Option 2 with an ongoing contest if possible. Note that if image identities are not released, but this is a made an ongoing contest, everyone is going to LB probe to find out which images change the score.", "votes": 1 }, { "id": 315733, "postDate": "2018-04-17T14:28:51.553Z", "content": "Both Pro and Con are true. It is frequent that Kaggle allows for some time submissions on past comps, what is both cool and really useful to learn and test future algorithms/frameworks.
\n\nI guess for the ones that took part in the competition it would be interesting to know.
\n\nAnyway, post competition submissions are usually in the context of courses/research, so no much point on taking shorcuts, or using aditional information like true images used. I mean, releasing that info doesn't harm and allows anyone seriously interested on using the data using it in same conditions we had. Or custom modify thos conditions, if their research justifies it.
\n\nSo... my opinion, yes, I think it would be interesting to know... Just my opinion, will be good to get some other points of view about it :-)
", "rawMarkdown": "Both Pro and Con are true. It is frequent that Kaggle allows for some time submissions on past comps, what is both cool and really useful to learn and test future algorithms/frameworks. \n\n I guess for the ones that took part in the competition it would be interesting to know. \n\nAnyway, post competition submissions are usually in the context of courses/research, so no much point on taking shorcuts, or using aditional information like true images used. I mean, releasing that info doesn't harm and allows anyone seriously interested on using the data using it in same conditions we had. Or custom modify thos conditions, if their research justifies it. \n\nSo... my opinion, yes, I think it would be interesting to know... Just my opinion, will be good to get some other points of view about it :-) ", "votes": 2 }, { "id": 316066, "postDate": "2018-04-18T05:41:03.553Z", "content": "What really matters IMHO is to have another competition on that topic in the future, say in a year or so. If keeping stage 2 annotations private helps, then I'd keep them private if I were you.
", "rawMarkdown": "What really matters IMHO is to have another competition on that topic in the future, say in a year or so. If keeping stage 2 annotations private helps, then I'd keep them private if I were you." }, { "id": 351547, "postDate": "2018-07-02T13:23:44.610Z", "content": "At this point, the team is inclined towards 1 (release everything) but given there is not a strong consensus we have fallen into option 3 (keep all secret) out of inertia.
", "rawMarkdown": "At this point, the team is inclined towards 1 (release everything) but given there is not a strong consensus we have fallen into option 3 (keep all secret) out of inertia. ", "replies": [ { "id": 356853, "postDate": "2018-07-14T16:08:55.417Z", "content": "If the train set masks are improved, is it possible you can release those? Also any additional images helps improve data ravenous NNs.
", "rawMarkdown": "If the train set masks are improved, is it possible you can release those? Also any additional images helps improve data ravenous NNs. " }, { "id": 359462, "postDate": "2018-07-20T06:41:02.873Z", "content": "Hi Anne, I didn't follow this topic recently. Did you already release the images with annotations? Thanks much.
", "rawMarkdown": "Hi Anne, I didn't follow this topic recently. Did you already release the images with annotations? Thanks much." }, { "id": 359748, "postDate": "2018-07-20T17:49:42.107Z", "content": "Nope, but actually it may happen in the next month or so! Keep an eye on this page for updates:\nhttps://data.broadinstitute.org/bbbc/BBBC038/
\n\nRight now it's just these posted:\nstage1_train.zip (82.9 MB)\nstage1_test.zip (9.5 MB)
\n\nBut I hope the others will be added soon.
", "rawMarkdown": "Nope, but actually it may happen in the next month or so! Keep an eye on this page for updates:\nhttps://data.broadinstitute.org/bbbc/BBBC038/\n\nRight now it's just these posted:\nstage1_train.zip (82.9 MB)\nstage1_test.zip (9.5 MB)\n\nBut I hope the others will be added soon." }, { "id": 359750, "postDate": "2018-07-20T17:57:08.613Z", "content": "Thanks Anne for the contribution to the community. I actually used the stage 1 data in my research to segment the NMR (nuclear magnetic resonance) peaks. It worked and in fact the result is better than I expected.
", "rawMarkdown": "Thanks Anne for the contribution to the community. I actually used the stage 1 data in my research to segment the NMR (nuclear magnetic resonance) peaks. It worked and in fact the result is better than I expected. ", "votes": 1 }, { "id": 359793, "postDate": "2018-07-20T19:22:39.957Z", "content": "Nice, that is quite surprising to me that it would be useful - I would love to see a link posted here if you publish it!
", "rawMarkdown": "Nice, that is quite surprising to me that it would be useful - I would love to see a link posted here if you publish it!" } ] }, { "id": 319460, "postDate": "2018-04-26T04:25:50.997Z", "content": "I want to make it as my thesis. So I vote 1.
", "rawMarkdown": "I want to make it as my thesis. So I vote 1." }, { "id": 319315, "postDate": "2018-04-25T19:08:12.877Z", "content": "Thanks for sharing in the community. I am with 2. More than that, I would like to see kind of ongoing leaderboard!
", "rawMarkdown": "Thanks for sharing in the community. I am with 2. More than that, I would like to see kind of ongoing leaderboard!" }, { "id": 316122, "postDate": "2018-04-18T08:14:10.663Z", "content": "I would vote for option 2. Eliminating poison images would enable people to improve their models on what really matters. Also the time generating predictions would be shorter. However I would not release annotations as that would introduce another factor which is not present in new unseen data. Grouping/clustering images should be better done based on the images alone and not the external annotations..
", "rawMarkdown": "I would vote for option 2. Eliminating poison images would enable people to improve their models on what really matters. Also the time generating predictions would be shorter. However I would not release annotations as that would introduce another factor which is not present in new unseen data. Grouping/clustering images should be better done based on the images alone and not the external annotations.." }, { "id": 315976, "postDate": "2018-04-17T23:18:10.200Z", "content": "I think it should be keep all secret and open a workshop at MICCAI where many researchers can try his own method. Hope the challenge will open a workshop at MICCAI 2018 and we can meet many kagglers :)
", "rawMarkdown": "I think it should be keep all secret and open a workshop at MICCAI where many researchers can try his own method. Hope the challenge will open a workshop at MICCAI 2018 and we can meet many kagglers :)" }, { "id": 315921, "postDate": "2018-04-17T20:01:49.100Z", "content": "Put raw image data on your server.\npublish url at kaggle\nupdate image data as it comes in.\n\"poison\" images are good. real world.
\n\nopen software resulted in great sw. Excel was last to go for me.\nopen data could result in great solutions.\nimagine snapping a picture of blood smear with your iPhone right at bed-side and getting a pretty good analysis.
", "rawMarkdown": "Put raw image data on your server.\npublish url at kaggle\nupdate image data as it comes in.\n\"poison\" images are good. real world.\n\nopen software resulted in great sw. Excel was last to go for me.\nopen data could result in great solutions.\nimagine snapping a picture of blood smear with your iPhone right at bed-side and getting a pretty good analysis." }, { "id": 315825, "postDate": "2018-04-17T16:28:37.533Z", "content": "Would be interesting to see the metadata. Is getting or labeling new images difficult? It seems like the con would be small if more data can be acquired for the next time.
", "rawMarkdown": "Would be interesting to see the metadata. Is getting or labeling new images difficult? It seems like the con would be small if more data can be acquired for the next time. ", "replies": [ { "id": 315843, "postDate": "2018-04-17T16:51:18.977Z", "content": "I agree. I think. I really see both sides to all of these questions. 😜
", "rawMarkdown": "I agree. I think. I really see both sides to all of these questions. 😜" } ] }, { "id": 315799, "postDate": "2018-04-17T16:02:56.933Z", "content": "I'm in favour of having the image identities released, purely from a learning perspective: we have two drastically different models and it would be nice to understand why one was favoured over the other. I think the reason must be down to the subset of data used for scoring.
\n\nAs mentioned in another comment, if you don't release the list and use the data for further competition you will be, in part, reducing it to a LB probing contest... Not in anyone's interest!
\n\nI don't think you need to release the annotations, if you want to keep options open in that regard in the future.
", "rawMarkdown": "I'm in favour of having the image identities released, purely from a learning perspective: we have two drastically different models and it would be nice to understand why one was favoured over the other. I think the reason must be down to the subset of data used for scoring.\n\nAs mentioned in another comment, if you don't release the list and use the data for further competition you will be, in part, reducing it to a LB probing contest... Not in anyone's interest!\n\nI don't think you need to release the annotations, if you want to keep options open in that regard in the future.", "replies": [ { "id": 315844, "postDate": "2018-04-17T16:53:13.120Z", "content": "I would expect that we’d use it for benchmarking rather than a competition. I don’t think is common outside of Kaggle-like competitions.
", "rawMarkdown": "I would expect that we’d use it for benchmarking rather than a competition. I don’t think is common outside of Kaggle-like competitions." } ] }, { "id": 315911, "postDate": "2018-04-17T19:36:07.970Z", "rawMarkdown": "", "votes": 1, "isDeleted": true } ], "comments": [ { "id": 315927, "author_name": "Russ Wolfinger", "author_url": "", "post_date": "2018-04-17T20:26:10.590000", "content": "My initial inclination is towards 1 in order to maximize knowledge gain by everyone right now, and who knows where that might lead. Would you be willing to share any thoughts on how you see things playing out from here both short and long-term? What applications appear most important, relevant, and promising with newfound deep learning results? We all would like to learn where state-of-the-art lies now with these kinds of images and it seems the stage 2 set is diverse enough to provide some nice \"watershed\" cases. In what kinds of images are the best methods really improving things and exactly where are they breaking down and making errors? For example, our unet-based model is currently doing poorly in the speckled / granular cases in which there can be a global scale resolution decision; that is, is the picture of three nuclei or several hundred. By retraining with updated data we could very likely do much better on them, but how important are such gains? At this time does it make sense to perhaps move towards building better models for specific staining and microscope modalities or do we press on further towards a one-model-fits-all global nuclei finder?
", "votes": 4, "replies": [ { "id": 315931, "author_name": "Bcottman", "author_url": "", "post_date": "2018-04-17T20:32:13.293000", "content": "yes. having metadata? with each image is useful. what technique used to capture image, etc.
", "votes": 0, "replies": [] }, { "id": 316207, "author_name": "William Green", "author_url": "", "post_date": "2018-04-18T12:25:00.720000", "content": "@Russ, those are interesting points.
", "votes": 0, "replies": [] }, { "id": 316677, "author_name": "Anne Carpenter", "author_url": "", "post_date": "2018-04-19T16:36:34.097000", "content": "Agree! And we don't have solid answers to them, yet. I'm eager to dig into the winning results and see whether there is a subset of experimental types that we can consider reasonably solved with this round of solutions.
", "votes": 0, "replies": [] } ] }, { "id": 315721, "author_name": "Allen Goodman", "author_url": "", "post_date": "2018-04-17T14:19:38.537000", "content": "What about releasing the images (i.e. removing the poison images) but not releasing the annotations?
", "votes": 2, "replies": [ { "id": 315735, "author_name": "Anne Carpenter", "author_url": "", "post_date": "2018-04-17T14:31:36.143000", "content": "Great point, will edit my post to add that in case this thread gets messy.
", "votes": 1, "replies": [] }, { "id": 315749, "author_name": "miguel perez", "author_url": "", "post_date": "2018-04-17T14:39:54.833000", "content": "Usually when kaggle comps. are left open test labels (or anotations) are not released, so the submission process and feedback through scores is the same as in competition time. Also releasing true images list does not necessarily imply to remove them from test... I guess it will just depend on how you want to shape that future \"ongoin comp\".
", "votes": 1, "replies": [] } ] }, { "id": 405978, "author_name": "Anne Carpenter", "author_url": "", "post_date": "2018-10-18T12:57:17.423000", "content": "Hi everyone! \nThe data is now released here: https://data.broadinstitute.org/bbbc/BBBC038/\nWe do still plan to release another version, with many corrections to the data. Keep an eye on that page!
", "votes": 1, "replies": [ { "id": 406001, "author_name": "Yibing Wu", "author_url": "", "post_date": "2018-10-18T13:34:28.403000", "content": "Thanks Anne.
", "votes": 0, "replies": [] } ] }, { "id": 319379, "author_name": "SamCC0815", "author_url": "", "post_date": "2018-04-25T23:10:45.187000", "content": "I would vote for 1. There should be totally new images/annotations for another competition down the road.
", "votes": 1, "replies": [] }, { "id": 316274, "author_name": "hengck23", "author_url": "", "post_date": "2018-04-18T15:45:43.473000", "content": "@Anne Carpenter Thank you for setting up the discussion. I would opt for 2. Actually, kaggler can always probe the data and know the \"estimated ground truth\" for all the options 2 and 3.
\n\nIf there is an intention of \"rerun this competition or make it an ongoing thing\", I suggest additional of some new black box data at the next round. Once \"black box data\" is seen, it become \"not so black\" anymore.
", "votes": 1, "replies": [ { "id": 316676, "author_name": "Anne Carpenter", "author_url": "", "post_date": "2018-04-19T16:35:01.837000", "content": "It's true, if we did another competition we absolutely would want to use only fresh images from new experiments anyway.
", "votes": 1, "replies": [] } ] }, { "id": 315766, "author_name": "Oscar Takeshita", "author_url": "", "post_date": "2018-04-17T15:09:33", "content": "There is a pro in leaving poison images secret as well. Some folks may want to perform unsupervised learning and try to identify which are the poisoned images and which are not. Albeit not the original intent of this competition, it is also an interesting vision machine learning topic.
", "votes": 1, "replies": [] }, { "id": 315764, "author_name": "SRT", "author_url": "", "post_date": "2018-04-17T15:07:54.123000", "content": "Support Option 2 with an ongoing contest if possible. Note that if image identities are not released, but this is a made an ongoing contest, everyone is going to LB probe to find out which images change the score.
", "votes": 1, "replies": [] }, { "id": 315733, "author_name": "miguel perez", "author_url": "", "post_date": "2018-04-17T14:28:51.553000", "content": "Both Pro and Con are true. It is frequent that Kaggle allows for some time submissions on past comps, what is both cool and really useful to learn and test future algorithms/frameworks.
\n\nI guess for the ones that took part in the competition it would be interesting to know.
\n\nAnyway, post competition submissions are usually in the context of courses/research, so no much point on taking shorcuts, or using aditional information like true images used. I mean, releasing that info doesn't harm and allows anyone seriously interested on using the data using it in same conditions we had. Or custom modify thos conditions, if their research justifies it.
\n\nSo... my opinion, yes, I think it would be interesting to know... Just my opinion, will be good to get some other points of view about it :-)
", "votes": 2, "replies": [] }, { "id": 316066, "author_name": "CPMP", "author_url": "", "post_date": "2018-04-18T05:41:03.553000", "content": "What really matters IMHO is to have another competition on that topic in the future, say in a year or so. If keeping stage 2 annotations private helps, then I'd keep them private if I were you.
", "votes": 0, "replies": [] }, { "id": 351547, "author_name": "Anne Carpenter", "author_url": "", "post_date": "2018-07-02T13:23:44.610000", "content": "At this point, the team is inclined towards 1 (release everything) but given there is not a strong consensus we have fallen into option 3 (keep all secret) out of inertia.
", "votes": 0, "replies": [ { "id": 356853, "author_name": "Bcottman", "author_url": "", "post_date": "2018-07-14T16:08:55.417000", "content": "If the train set masks are improved, is it possible you can release those? Also any additional images helps improve data ravenous NNs.
", "votes": 0, "replies": [] }, { "id": 359462, "author_name": "Yibing Wu", "author_url": "", "post_date": "2018-07-20T06:41:02.873000", "content": "Hi Anne, I didn't follow this topic recently. Did you already release the images with annotations? Thanks much.
", "votes": 0, "replies": [] }, { "id": 359748, "author_name": "Anne Carpenter", "author_url": "", "post_date": "2018-07-20T17:49:42.107000", "content": "Nope, but actually it may happen in the next month or so! Keep an eye on this page for updates:\nhttps://data.broadinstitute.org/bbbc/BBBC038/
\n\nRight now it's just these posted:\nstage1_train.zip (82.9 MB)\nstage1_test.zip (9.5 MB)
\n\nBut I hope the others will be added soon.
", "votes": 0, "replies": [] }, { "id": 359750, "author_name": "Yibing Wu", "author_url": "", "post_date": "2018-07-20T17:57:08.613000", "content": "Thanks Anne for the contribution to the community. I actually used the stage 1 data in my research to segment the NMR (nuclear magnetic resonance) peaks. It worked and in fact the result is better than I expected.
", "votes": 1, "replies": [] }, { "id": 359793, "author_name": "Anne Carpenter", "author_url": "", "post_date": "2018-07-20T19:22:39.957000", "content": "Nice, that is quite surprising to me that it would be useful - I would love to see a link posted here if you publish it!
", "votes": 0, "replies": [] } ] }, { "id": 319460, "author_name": "xiatian", "author_url": "", "post_date": "2018-04-26T04:25:50.997000", "content": "I want to make it as my thesis. So I vote 1.
", "votes": 0, "replies": [] }, { "id": 319315, "author_name": "Morteza Milani", "author_url": "", "post_date": "2018-04-25T19:08:12.877000", "content": "Thanks for sharing in the community. I am with 2. More than that, I would like to see kind of ongoing leaderboard!
", "votes": 0, "replies": [] }, { "id": 316122, "author_name": "rmldj", "author_url": "", "post_date": "2018-04-18T08:14:10.663000", "content": "I would vote for option 2. Eliminating poison images would enable people to improve their models on what really matters. Also the time generating predictions would be shorter. However I would not release annotations as that would introduce another factor which is not present in new unseen data. Grouping/clustering images should be better done based on the images alone and not the external annotations..
", "votes": 0, "replies": [] }, { "id": 315976, "author_name": "john1231983", "author_url": "", "post_date": "2018-04-17T23:18:10.200000", "content": "I think it should be keep all secret and open a workshop at MICCAI where many researchers can try his own method. Hope the challenge will open a workshop at MICCAI 2018 and we can meet many kagglers :)
", "votes": 0, "replies": [] }, { "id": 315921, "author_name": "Bcottman", "author_url": "", "post_date": "2018-04-17T20:01:49.100000", "content": "Put raw image data on your server.\npublish url at kaggle\nupdate image data as it comes in.\n\"poison\" images are good. real world.
\n\nopen software resulted in great sw. Excel was last to go for me.\nopen data could result in great solutions.\nimagine snapping a picture of blood smear with your iPhone right at bed-side and getting a pretty good analysis.
", "votes": 0, "replies": [] }, { "id": 315825, "author_name": "Tristan Delforge", "author_url": "", "post_date": "2018-04-17T16:28:37.533000", "content": "Would be interesting to see the metadata. Is getting or labeling new images difficult? It seems like the con would be small if more data can be acquired for the next time.
", "votes": 0, "replies": [ { "id": 315843, "author_name": "Allen Goodman", "author_url": "", "post_date": "2018-04-17T16:51:18.977000", "content": "I agree. I think. I really see both sides to all of these questions. 😜
", "votes": 0, "replies": [] } ] }, { "id": 315799, "author_name": "fergusoci", "author_url": "", "post_date": "2018-04-17T16:02:56.933000", "content": "I'm in favour of having the image identities released, purely from a learning perspective: we have two drastically different models and it would be nice to understand why one was favoured over the other. I think the reason must be down to the subset of data used for scoring.
\n\nAs mentioned in another comment, if you don't release the list and use the data for further competition you will be, in part, reducing it to a LB probing contest... Not in anyone's interest!
\n\nI don't think you need to release the annotations, if you want to keep options open in that regard in the future.
", "votes": 0, "replies": [ { "id": 315844, "author_name": "Allen Goodman", "author_url": "", "post_date": "2018-04-17T16:53:13.120000", "content": "I would expect that we’d use it for benchmarking rather than a competition. I don’t think is common outside of Kaggle-like competitions.
", "votes": 0, "replies": [] } ] }, { "id": 315911, "author_name": "", "author_url": "", "post_date": "2018-04-17T19:36:07.970000", "content": "", "votes": 1, "replies": [] } ], "raw_markdown_by_id": { "315715": "What do you think, should we release the list of stage 2 test images used for scoring?\n\nPro: It seems like it would be useful for competitors to digest how/why their model did well (e.g., https://www.kaggle.com/c/data-science-bowl-2018/discussion/54739). And also make people feel better about their model performing badly on poison images that we would not intend a winning model to work nicely on. We'd include metadata so you know which images come from the same lab/experiment.\n\nCon: It might be nice to keep it secret if we did want to rerun this competition or make it an ongoing thing (can Kaggle make it an ongoing leaderboard, for example?)\n\nSo the choices are:\n\n 1. release image identities (i.e. removing the poison images) and\n annotations\n 2. release image identities (i.e. removing the poison\n images) but not annotations \n 3. keep them all secret\n\n", "315927": "My initial inclination is towards 1 in order to maximize knowledge gain by everyone right now, and who knows where that might lead. Would you be willing to share any thoughts on how you see things playing out from here both short and long-term? What applications appear most important, relevant, and promising with newfound deep learning results? We all would like to learn where state-of-the-art lies now with these kinds of images and it seems the stage 2 set is diverse enough to provide some nice \"watershed\" cases. In what kinds of images are the best methods really improving things and exactly where are they breaking down and making errors? For example, our unet-based model is currently doing poorly in the speckled / granular cases in which there can be a global scale resolution decision; that is, is the picture of three nuclei or several hundred. By retraining with updated data we could very likely do much better on them, but how important are such gains? At this time does it make sense to perhaps move towards building better models for specific staining and microscope modalities or do we press on further towards a one-model-fits-all global nuclei finder?", "315721": "What about releasing the images (i.e. removing the poison images) but not releasing the annotations?", "405978": "Hi everyone! \nThe data is now released here: https://data.broadinstitute.org/bbbc/BBBC038/\nWe do still plan to release another version, with many corrections to the data. Keep an eye on that page!", "319379": "I would vote for 1. There should be totally new images/annotations for another competition down the road.", "316274": "@Anne Carpenter Thank you for setting up the discussion. I would opt for 2. Actually, kaggler can always probe the data and know the \"estimated ground truth\" for all the options 2 and 3.\n\nIf there is an intention of \"rerun this competition or make it an ongoing thing\", I suggest additional of some new black box data at the next round. Once \"black box data\" is seen, it become \"not so black\" anymore.", "315766": "There is a pro in leaving poison images secret as well. Some folks may want to perform unsupervised learning and try to identify which are the poisoned images and which are not. Albeit not the original intent of this competition, it is also an interesting vision machine learning topic. ", "315764": "Support Option 2 with an ongoing contest if possible. Note that if image identities are not released, but this is a made an ongoing contest, everyone is going to LB probe to find out which images change the score.", "315733": "Both Pro and Con are true. It is frequent that Kaggle allows for some time submissions on past comps, what is both cool and really useful to learn and test future algorithms/frameworks. \n\n I guess for the ones that took part in the competition it would be interesting to know. \n\nAnyway, post competition submissions are usually in the context of courses/research, so no much point on taking shorcuts, or using aditional information like true images used. I mean, releasing that info doesn't harm and allows anyone seriously interested on using the data using it in same conditions we had. Or custom modify thos conditions, if their research justifies it. \n\nSo... my opinion, yes, I think it would be interesting to know... Just my opinion, will be good to get some other points of view about it :-) ", "316066": "What really matters IMHO is to have another competition on that topic in the future, say in a year or so. If keeping stage 2 annotations private helps, then I'd keep them private if I were you.", "351547": "At this point, the team is inclined towards 1 (release everything) but given there is not a strong consensus we have fallen into option 3 (keep all secret) out of inertia. ", "319460": "I want to make it as my thesis. So I vote 1.", "319315": "Thanks for sharing in the community. I am with 2. More than that, I would like to see kind of ongoing leaderboard!", "316122": "I would vote for option 2. Eliminating poison images would enable people to improve their models on what really matters. Also the time generating predictions would be shorter. However I would not release annotations as that would introduce another factor which is not present in new unseen data. Grouping/clustering images should be better done based on the images alone and not the external annotations..", "315976": "I think it should be keep all secret and open a workshop at MICCAI where many researchers can try his own method. Hope the challenge will open a workshop at MICCAI 2018 and we can meet many kagglers :)", "315921": "Put raw image data on your server.\npublish url at kaggle\nupdate image data as it comes in.\n\"poison\" images are good. real world.\n\nopen software resulted in great sw. Excel was last to go for me.\nopen data could result in great solutions.\nimagine snapping a picture of blood smear with your iPhone right at bed-side and getting a pretty good analysis.", "315825": "Would be interesting to see the metadata. Is getting or labeling new images difficult? It seems like the con would be small if more data can be acquired for the next time. ", "315799": "I'm in favour of having the image identities released, purely from a learning perspective: we have two drastically different models and it would be nice to understand why one was favoured over the other. I think the reason must be down to the subset of data used for scoring.\n\nAs mentioned in another comment, if you don't release the list and use the data for further competition you will be, in part, reducing it to a LB probing contest... Not in anyone's interest!\n\nI don't think you need to release the annotations, if you want to keep options open in that regard in the future.", "315911": "" } }