\"The second stage dataset will contain images from unseen experimental conditions\" -- Does unseen mean different modalities or just different resolution/ noise levels?
", "messages": [ { "id": "269967", "postDate": "01/17/2018 15:41:15", "content": "\"The second stage dataset will contain images from unseen experimental conditions\" -- Does unseen mean different modalities or just different resolution/ noise levels?
", "rawMarkdown": "\"The second stage dataset will contain images from unseen experimental conditions\" -- Does unseen mean different modalities or just different resolution/ noise levels?", "votes": null }, { "id": "270080", "postDate": "01/17/2018 18:27:14", "content": "I’m an organizer, but I (purposefully) don’t have any knowledge of the test data. Nevertheless, I believe this information is intentionally withheld.
", "rawMarkdown": "I’m an organizer, but I (purposefully) don’t have any knowledge of the test data. Nevertheless, I believe this information is intentionally withheld.", "votes": null }, { "id": "270112", "postDate": "01/17/2018 19:01:21", "content": "You can google search \"nucleus histology\" and switch to images. You will see so many unseen experimental conditions and more.
", "rawMarkdown": "You can google search \"nucleus histology\" and switch to images. You will see so many unseen experimental conditions and more.", "votes": null }, { "id": "270671", "postDate": "01/18/2018 17:55:56", "content": "\n\n\nAllen Goodman wrote
\n \n\n\nI’m an organizer, but I (purposefully) don’t have any knowledge of the test data. Nevertheless, I believe this information is intentionally withheld.
\n
So do you want us to work 3 months on the challenge just to find that images on the 2 stage are totally different and our work is worthless?
\n\nI don't think that is reasonable. I think having a 2 stage is acceptable, but the requirements for the 2 stage should be clearly defined. For example:
\n\nOtherwise 2 stage will be a lottery. If you want a good solution you have to specify it correctly.
", "rawMarkdown": "> **Allen Goodman wrote**\n> \n> > I’m an organizer, but I (purposefully) don’t have any knowledge of the test data. Nevertheless, I believe this information is intentionally withheld. \n> \n\nSo do you want us to work 3 months on the challenge just to find that images on the 2 stage are totally different and our work is worthless?\n\nI don't think that is reasonable. I think having a 2 stage is acceptable, but the requirements for the 2 stage should be clearly defined. For example:\n\n* Enumerating the different experimental setups and having at least one example of each\n* Diferences in scale between images\n* Color/gray images\n\nOtherwise 2 stage will be a lottery. If you want a good solution you have to specify it correctly.", "votes": null }, { "id": "270678", "postDate": "01/18/2018 18:07:11", "content": "\n\n\nI’m an organizer, but I (purposefully) don’t have any knowledge of the test data. Nevertheless, I believe this information is intentionally withheld.
\n
This is excellent:) While not knowing what the stage two data will be does make it difficult to ensure your model will generalize, I think this set up is very good.
\n\nHaving models that generalize to a wide variety of unseen conditions seems to be a theme recently. And it is a good sign to see that we now can start addressing such problems. This should be a fun competition to watch:)
", "rawMarkdown": "> I’m an organizer, but I (purposefully) don’t have any knowledge of the test data. Nevertheless, I believe this information is intentionally withheld. \n\nThis is excellent:) While not knowing what the stage two data will be does make it difficult to ensure your model will generalize, I think this set up is very good. \n\nHaving models that generalize to a wide variety of unseen conditions seems to be a theme recently. And it is a good sign to see that we now can start addressing such problems. This should be a fun competition to watch:)", "votes": null }, { "id": "270679", "postDate": "01/18/2018 18:10:09", "content": "I also like this setup, it keeps intrigue up to the last second.
", "rawMarkdown": "I also like this setup, it keeps intrigue up to the last second.", "votes": null }, { "id": "270684", "postDate": "01/18/2018 18:20:59", "content": "Would you like dropping from first position to last only because all 2 stage images have a different setup that you didn't think of?
\n\nThe conditions should be specified, even more in a problem like this where most of people don't have expertise at it. I don't know anything about microscopic setups, do I have to make a great search to cover all possible setups? I think it's a waste of time that could be used on building a better model.
", "rawMarkdown": "Would you like dropping from first position to last only because all 2 stage images have a different setup that you didn't think of?\n\nThe conditions should be specified, even more in a problem like this where most of people don't have expertise at it. I don't know anything about microscopic setups, do I have to make a great search to cover all possible setups? I think it's a waste of time that could be used on building a better model.", "votes": null }, { "id": "270691", "postDate": "01/18/2018 18:32:26", "content": "No I would not like it. I would be angry, and feel cheated, but that is how things are. This is data science bowl, not ML bowl, so actually is very good that we can go dredge up whatever data is out there. If you want to be able to tell how you will perform based on validation, then this is probably not the best competition to compete in.
", "rawMarkdown": "No I would not like it. I would be angry, and feel cheated, but that is how things are. This is data science bowl, not ML bowl, so actually is very good that we can go dredge up whatever data is out there. If you want to be able to tell how you will perform based on validation, then this is probably not the best competition to compete in.", "votes": null }, { "id": "270704", "postDate": "01/18/2018 18:55:29", "content": "I would say it's not ML bow either data science bowl, it's lottery bowl unless 2 stage is minimum specified ;)
", "rawMarkdown": "I would say it's not ML bow either data science bowl, it's lottery bowl unless 2 stage is minimum specified ;)", "votes": null }, { "id": "270709", "postDate": "01/18/2018 18:58:38", "content": "At least it should be possible to say that the second stage will cover a balanced set of new setups. If the new cases are balanced a model that generalizes well could win the competition.
\n\nIf all the cases on the 2 stage belong to a new strange setup the winning model does not necesarily need to be the model that best generalizes.
", "rawMarkdown": "At least it should be possible to say that the second stage will cover a balanced set of new setups. If the new cases are balanced a model that generalizes well could win the competition. \n\nIf all the cases on the 2 stage belong to a new strange setup the winning model does not necesarily need to be the model that best generalizes.", "votes": null }, { "id": "270713", "postDate": "01/18/2018 19:06:27", "content": "I do not like LB overfitting. Second stage with unknown images give real test on all little \"optimization\" tweaks that were done to get top public LB position.
", "rawMarkdown": "I do not like LB overfitting. Second stage with unknown images give real test on all little \"optimization\" tweaks that were done to get top public LB position.", "votes": null }, { "id": "270721", "postDate": "01/18/2018 19:21:34", "content": "You can assume stage two will be reasonable. Neither Kaggle nor the competition hosts are out to trick you or present impossibly hard problems.
", "rawMarkdown": "You can assume stage two will be reasonable. Neither Kaggle nor the competition hosts are out to trick you or present impossibly hard problems.", "votes": null }, { "id": "270733", "postDate": "01/18/2018 19:47:35", "content": "\n\n\nWilliam Cukierski wrote
\n \n\n\nYou can assume stage two will be reasonable. Neither Kaggle nor the competition hosts are out to trick you or present impossibly hard problems.
\n
I still remember DSTL competition when only a small fraction of the test set was used without any warning.
\nAnd most recent Tensorflow Speech Recognition with a very different test and train set.
Yes, googled it a little and asked a doctor. There are different dying methods / imaging modalities for different purposes, and images would look a lot different. It is reasonable and important to develop a method that can be well generalized to various experimental conditions, otherwise the a lot of effort needs to be done to acquire all kinds of dataset.
", "rawMarkdown": "Yes, googled it a little and asked a doctor. There are different dying methods / imaging modalities for different purposes, and images would look a lot different. It is reasonable and important to develop a method that can be well generalized to various experimental conditions, otherwise the a lot of effort needs to be done to acquire all kinds of dataset.", "votes": null }, { "id": "271560", "postDate": "01/20/2018 19:31:33", "content": "An organizer here. You could take two views:\n(1) Annoyance that this challenge is hard because we are intentionally not giving you the full range of all possible image types. \n(2) Excitement that this challenge is about generalizability and robustness.
\n\nThe advantage of perspective (2) is it produces meaningful solutions that help in the real world. I also think this problem is a solvable one: biologists can detect nuclei accurately no matter the size, species, or microscopy type (so long as they agree on some ground rules, like whether to count micronuclei). I'm not saying that I expect >0.90 in this challenge, perhaps we need much more training data to get to that point, but I just mean that I think it will be solved sooner or later.
\n\nIf we gave you a defined set of image types as training data, the problem would be, well, perhaps not trivial but at least a lot easier, where differences between competitors come down to not-so-interesting choices in models. As we have planned the challenge, we expect a lot of variety in how teams choose to handle it.
\n\nBonus: This one will be a nail-biter at the end, as we see how generalizable the models end up being!
", "rawMarkdown": "An organizer here. You could take two views:\n(1) Annoyance that this challenge is hard because we are intentionally not giving you the full range of all possible image types. \n(2) Excitement that this challenge is about generalizability and robustness. \n\nThe advantage of perspective (2) is it produces meaningful solutions that help in the real world. I also think this problem is a solvable one: biologists can detect nuclei accurately no matter the size, species, or microscopy type (so long as they agree on some ground rules, like whether to count micronuclei). I'm not saying that I expect >0.90 in this challenge, perhaps we need much more training data to get to that point, but I just mean that I think it will be solved sooner or later. \n\nIf we gave you a defined set of image types as training data, the problem would be, well, perhaps not trivial but at least a lot easier, where differences between competitors come down to not-so-interesting choices in models. As we have planned the challenge, we expect a lot of variety in how teams choose to handle it. \n\nBonus: This one will be a nail-biter at the end, as we see how generalizable the models end up being!", "votes": null }, { "id": "275392", "postDate": "01/29/2018 02:26:40", "content": "I searched \"nucleus histology\" images and it seems most of the images are more complex than the ones in our training set. It seems like they show multiple dyes, maybe including dyes that aren't DNA specific. I don't know how frightened I should be. Any source for images more like the ones in the competition?
", "rawMarkdown": "I searched \"nucleus histology\" images and it seems most of the images are more complex than the ones in our training set. It seems like they show multiple dyes, maybe including dyes that aren't DNA specific. I don't know how frightened I should be. Any source for images more like the ones in the competition?", "votes": null } ], "comments": [ { "id": 270080, "author_name": "agoodman", "author_url": "", "post_date": "01/17/2018 18:27:14", "content": "I’m an organizer, but I (purposefully) don’t have any knowledge of the test data. Nevertheless, I believe this information is intentionally withheld.
", "votes": null, "replies": [ { "id": 270671, "author_name": "ironbar", "author_url": "", "post_date": "01/18/2018 17:55:56", "content": "\n\n\nAllen Goodman wrote
\n \n\n\nI’m an organizer, but I (purposefully) don’t have any knowledge of the test data. Nevertheless, I believe this information is intentionally withheld.
\n
So do you want us to work 3 months on the challenge just to find that images on the 2 stage are totally different and our work is worthless?
\n\nI don't think that is reasonable. I think having a 2 stage is acceptable, but the requirements for the 2 stage should be clearly defined. For example:
\n\nOtherwise 2 stage will be a lottery. If you want a good solution you have to specify it correctly.
", "votes": null, "replies": [] }, { "id": 270678, "author_name": "devinanzelmo", "author_url": "", "post_date": "01/18/2018 18:07:11", "content": "\n\n\nI’m an organizer, but I (purposefully) don’t have any knowledge of the test data. Nevertheless, I believe this information is intentionally withheld.
\n
This is excellent:) While not knowing what the stage two data will be does make it difficult to ensure your model will generalize, I think this set up is very good.
\n\nHaving models that generalize to a wide variety of unseen conditions seems to be a theme recently. And it is a good sign to see that we now can start addressing such problems. This should be a fun competition to watch:)
", "votes": null, "replies": [] }, { "id": 270679, "author_name": "serhiy", "author_url": "", "post_date": "01/18/2018 18:10:09", "content": "I also like this setup, it keeps intrigue up to the last second.
", "votes": null, "replies": [] }, { "id": 270684, "author_name": "ironbar", "author_url": "", "post_date": "01/18/2018 18:20:59", "content": "Would you like dropping from first position to last only because all 2 stage images have a different setup that you didn't think of?
\n\nThe conditions should be specified, even more in a problem like this where most of people don't have expertise at it. I don't know anything about microscopic setups, do I have to make a great search to cover all possible setups? I think it's a waste of time that could be used on building a better model.
", "votes": null, "replies": [] }, { "id": 270691, "author_name": "devinanzelmo", "author_url": "", "post_date": "01/18/2018 18:32:26", "content": "No I would not like it. I would be angry, and feel cheated, but that is how things are. This is data science bowl, not ML bowl, so actually is very good that we can go dredge up whatever data is out there. If you want to be able to tell how you will perform based on validation, then this is probably not the best competition to compete in.
", "votes": null, "replies": [] }, { "id": 270704, "author_name": "ironbar", "author_url": "", "post_date": "01/18/2018 18:55:29", "content": "I would say it's not ML bow either data science bowl, it's lottery bowl unless 2 stage is minimum specified ;)
", "votes": null, "replies": [] }, { "id": 270709, "author_name": "ironbar", "author_url": "", "post_date": "01/18/2018 18:58:38", "content": "At least it should be possible to say that the second stage will cover a balanced set of new setups. If the new cases are balanced a model that generalizes well could win the competition.
\n\nIf all the cases on the 2 stage belong to a new strange setup the winning model does not necesarily need to be the model that best generalizes.
", "votes": null, "replies": [] }, { "id": 270713, "author_name": "serhiy", "author_url": "", "post_date": "01/18/2018 19:06:27", "content": "I do not like LB overfitting. Second stage with unknown images give real test on all little \"optimization\" tweaks that were done to get top public LB position.
", "votes": null, "replies": [] }, { "id": 270721, "author_name": "wcukierski", "author_url": "", "post_date": "01/18/2018 19:21:34", "content": "You can assume stage two will be reasonable. Neither Kaggle nor the competition hosts are out to trick you or present impossibly hard problems.
", "votes": null, "replies": [] }, { "id": 270733, "author_name": "ironbar", "author_url": "", "post_date": "01/18/2018 19:47:35", "content": "\n\n\nWilliam Cukierski wrote
\n \n\n\nYou can assume stage two will be reasonable. Neither Kaggle nor the competition hosts are out to trick you or present impossibly hard problems.
\n
I still remember DSTL competition when only a small fraction of the test set was used without any warning.
\nAnd most recent Tensorflow Speech Recognition with a very different test and train set.
An organizer here. You could take two views:\n(1) Annoyance that this challenge is hard because we are intentionally not giving you the full range of all possible image types. \n(2) Excitement that this challenge is about generalizability and robustness.
\n\nThe advantage of perspective (2) is it produces meaningful solutions that help in the real world. I also think this problem is a solvable one: biologists can detect nuclei accurately no matter the size, species, or microscopy type (so long as they agree on some ground rules, like whether to count micronuclei). I'm not saying that I expect >0.90 in this challenge, perhaps we need much more training data to get to that point, but I just mean that I think it will be solved sooner or later.
\n\nIf we gave you a defined set of image types as training data, the problem would be, well, perhaps not trivial but at least a lot easier, where differences between competitors come down to not-so-interesting choices in models. As we have planned the challenge, we expect a lot of variety in how teams choose to handle it.
\n\nBonus: This one will be a nail-biter at the end, as we see how generalizable the models end up being!
", "votes": null, "replies": [] } ] }, { "id": 270112, "author_name": "serhiy", "author_url": "", "post_date": "01/17/2018 19:01:21", "content": "You can google search \"nucleus histology\" and switch to images. You will see so many unseen experimental conditions and more.
", "votes": null, "replies": [ { "id": 271060, "author_name": "isalin", "author_url": "", "post_date": "01/19/2018 16:01:07", "content": "Yes, googled it a little and asked a doctor. There are different dying methods / imaging modalities for different purposes, and images would look a lot different. It is reasonable and important to develop a method that can be well generalized to various experimental conditions, otherwise the a lot of effort needs to be done to acquire all kinds of dataset.
", "votes": null, "replies": [] }, { "id": 275392, "author_name": "nathanrm", "author_url": "", "post_date": "01/29/2018 02:26:40", "content": "I searched \"nucleus histology\" images and it seems most of the images are more complex than the ones in our training set. It seems like they show multiple dyes, maybe including dyes that aren't DNA specific. I don't know how frightened I should be. Any source for images more like the ones in the competition?
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "269967": "\"The second stage dataset will contain images from unseen experimental conditions\" -- Does unseen mean different modalities or just different resolution/ noise levels?", "270080": "I’m an organizer, but I (purposefully) don’t have any knowledge of the test data. Nevertheless, I believe this information is intentionally withheld.", "270112": "You can google search \"nucleus histology\" and switch to images. You will see so many unseen experimental conditions and more.", "270671": "> **Allen Goodman wrote**\n> \n> > I’m an organizer, but I (purposefully) don’t have any knowledge of the test data. Nevertheless, I believe this information is intentionally withheld. \n> \n\nSo do you want us to work 3 months on the challenge just to find that images on the 2 stage are totally different and our work is worthless?\n\nI don't think that is reasonable. I think having a 2 stage is acceptable, but the requirements for the 2 stage should be clearly defined. For example:\n\n* Enumerating the different experimental setups and having at least one example of each\n* Diferences in scale between images\n* Color/gray images\n\nOtherwise 2 stage will be a lottery. If you want a good solution you have to specify it correctly.", "270678": "> I’m an organizer, but I (purposefully) don’t have any knowledge of the test data. Nevertheless, I believe this information is intentionally withheld. \n\nThis is excellent:) While not knowing what the stage two data will be does make it difficult to ensure your model will generalize, I think this set up is very good. \n\nHaving models that generalize to a wide variety of unseen conditions seems to be a theme recently. And it is a good sign to see that we now can start addressing such problems. This should be a fun competition to watch:)", "270679": "I also like this setup, it keeps intrigue up to the last second.", "270684": "Would you like dropping from first position to last only because all 2 stage images have a different setup that you didn't think of?\n\nThe conditions should be specified, even more in a problem like this where most of people don't have expertise at it. I don't know anything about microscopic setups, do I have to make a great search to cover all possible setups? I think it's a waste of time that could be used on building a better model.", "270691": "No I would not like it. I would be angry, and feel cheated, but that is how things are. This is data science bowl, not ML bowl, so actually is very good that we can go dredge up whatever data is out there. If you want to be able to tell how you will perform based on validation, then this is probably not the best competition to compete in.", "270704": "I would say it's not ML bow either data science bowl, it's lottery bowl unless 2 stage is minimum specified ;)", "270709": "At least it should be possible to say that the second stage will cover a balanced set of new setups. If the new cases are balanced a model that generalizes well could win the competition. \n\nIf all the cases on the 2 stage belong to a new strange setup the winning model does not necesarily need to be the model that best generalizes.", "270713": "I do not like LB overfitting. Second stage with unknown images give real test on all little \"optimization\" tweaks that were done to get top public LB position.", "270721": "You can assume stage two will be reasonable. Neither Kaggle nor the competition hosts are out to trick you or present impossibly hard problems.", "270733": "> **William Cukierski wrote**\n> \n> > You can assume stage two will be reasonable. Neither Kaggle nor the competition hosts are out to trick you or present impossibly hard problems.\n\nI still remember DSTL competition when only a small fraction of the test set was used without any warning. \nAnd most recent Tensorflow Speech Recognition with a very different test and train set.", "271060": "Yes, googled it a little and asked a doctor. There are different dying methods / imaging modalities for different purposes, and images would look a lot different. It is reasonable and important to develop a method that can be well generalized to various experimental conditions, otherwise the a lot of effort needs to be done to acquire all kinds of dataset.", "271560": "An organizer here. You could take two views:\n(1) Annoyance that this challenge is hard because we are intentionally not giving you the full range of all possible image types. \n(2) Excitement that this challenge is about generalizability and robustness. \n\nThe advantage of perspective (2) is it produces meaningful solutions that help in the real world. I also think this problem is a solvable one: biologists can detect nuclei accurately no matter the size, species, or microscopy type (so long as they agree on some ground rules, like whether to count micronuclei). I'm not saying that I expect >0.90 in this challenge, perhaps we need much more training data to get to that point, but I just mean that I think it will be solved sooner or later. \n\nIf we gave you a defined set of image types as training data, the problem would be, well, perhaps not trivial but at least a lot easier, where differences between competitors come down to not-so-interesting choices in models. As we have planned the challenge, we expect a lot of variety in how teams choose to handle it. \n\nBonus: This one will be a nail-biter at the end, as we see how generalizable the models end up being!", "275392": "I searched \"nucleus histology\" images and it seems most of the images are more complex than the ones in our training set. It seems like they show multiple dyes, maybe including dyes that aren't DNA specific. I don't know how frightened I should be. Any source for images more like the ones in the competition?" }, "source": "meta" }