This competition had way too little data to train neural nets reliably. I have absolutely no clue about how my stuff will fare, but I'm not surprised by stage2 images. They are different from stage1, as announced.
\n\nI don't think the competition setting is right to find most robust models. I would rather have looked for robust model training, i.e. developping fully automated training pipeline, and provide stage2 training data representative of unforeseen data.
\n\nExpecting us to find models robust to new types of stain/microscope without representative trianing data is just a misconception about what machine learning can do.
\n\nThis said, I'll retrain my models on as much data as I can ;)
", "messages": [ { "id": "313170", "postDate": "04/12/2018 21:22:57", "content": "This competition had way too little data to train neural nets reliably. I have absolutely no clue about how my stuff will fare, but I'm not surprised by stage2 images. They are different from stage1, as announced.
\n\nI don't think the competition setting is right to find most robust models. I would rather have looked for robust model training, i.e. developping fully automated training pipeline, and provide stage2 training data representative of unforeseen data.
\n\nExpecting us to find models robust to new types of stain/microscope without representative trianing data is just a misconception about what machine learning can do.
\n\nThis said, I'll retrain my models on as much data as I can ;)
", "rawMarkdown": "This competition had way too little data to train neural nets reliably. I have absolutely no clue about how my stuff will fare, but I'm not surprised by stage2 images. They are different from stage1, as announced.\n\nI don't think the competition setting is right to find most robust models. I would rather have looked for robust model training, i.e. developping fully automated training pipeline, and provide stage2 training data representative of unforeseen data. \n\nExpecting us to find models robust to new types of stain/microscope without representative trianing data is just a misconception about what machine learning can do.\n\nThis said, I'll retrain my models on as much data as I can ;)", "votes": null }, { "id": "313173", "postDate": "04/12/2018 21:25:10", "content": "I may sound negative about this competition, but one thing I really enjoy is thtat no LB probing is possible in stage2. This is such a relief that I am almost speechless. I don't know who had the idea but having all 0 scores is brillant! I love it.
", "rawMarkdown": "I may sound negative about this competition, but one thing I really enjoy is thtat no LB probing is possible in stage2. This is such a relief that I am almost speechless. I don't know who had the idea but having all 0 scores is brillant! I love it.", "votes": null }, { "id": "313180", "postDate": "04/12/2018 21:42:26", "content": "Agreed. I was concerned about this too. If the stage2 LB was visible now we'd be seeing people's scores slowly creeping up...
", "rawMarkdown": "Agreed. I was concerned about this too. If the stage2 LB was visible now we'd be seeing people's scores slowly creeping up...", "votes": null }, { "id": "313348", "postDate": "04/13/2018 06:14:15", "content": "@CPMP I too love having 0 LB scores, that means we only need to submit our result once and done and enjoy our weekend.
", "rawMarkdown": "CPMP I too love having 0 LB scores, that means we only need to submit our result once and done and enjoy our weekend.", "votes": null }, { "id": "313352", "postDate": "04/13/2018 06:23:58", "content": "@Nicole Finnie, the only drawback is what if there is something wrong with format or something like that. I wish they could use a tiny data like 1% to make us sure there is no such thing.
", "rawMarkdown": "Nicole Finnie, the only drawback is what if there is something wrong with format or something like that. I wish they could use a tiny data like 1% to make us sure there is no such thing.", "votes": null }, { "id": "313354", "postDate": "04/13/2018 06:34:02", "content": "Agree, but you still can compare results visually. Some new types of images looks totally incorrect for my MaskRCNN.
", "rawMarkdown": "Agree, but you still can compare results visually. Some new types of images looks totally incorrect for my MaskRCNN.", "votes": null }, { "id": "313366", "postDate": "04/13/2018 07:04:39", "content": "I belive it is unfair and against the rules to add new data and this stage. It's ok to tune/retrain your models with stage1 answers. But it's clearly cheating to find new data similiar to stage2 (or even take from stage2 and label it). It's much worse than changing \"hyperparameters\".
", "rawMarkdown": "I belive it is unfair and against the rules to add new data and this stage. It's ok to tune/retrain your models with stage1 answers. But it's clearly cheating to find new data similiar to stage2 (or even take from stage2 and label it). It's much worse than changing \"hyperparameters\".", "votes": null }, { "id": "313381", "postDate": "04/13/2018 07:23:32", "content": "In practice I'll retrain with train and test from stage1. It is against the rules to use data not published in the external data thread before deadline. This includes any of the stage2 test data.
\n\nBut it does not seem to be against the rule to retrain models on data shared before deadline, provided you don't modify your code (eg adding a blurring preprocessing if you did not have one in stage1).
", "rawMarkdown": "In practice I'll retrain with train and test from stage1. It is against the rules to use data not published in the external data thread before deadline. This includes any of the stage2 test data.\n\nBut it does not seem to be against the rule to retrain models on data shared before deadline, provided you don't modify your code (eg adding a blurring preprocessing if you did not have one in stage1).", "votes": null }, { "id": "313384", "postDate": "04/13/2018 07:31:58", "content": "You are changing your decision about training data after looking at stage2 images. It's like tweaking hyperparameters to fit in stage2 data better. It's even worse. Your initial data and code should be frozen and ready just to take stage1 answers and train with them.
", "rawMarkdown": "You are changing your decision about training data after looking at stage2 images. It's like tweaking hyperparameters to fit in stage2 data better. It's even worse. Your initial data and code should be frozen and ready just to take stage1 answers and train with them.", "votes": null }, { "id": "313457", "postDate": "04/13/2018 09:42:27", "content": "Is there a confirmation from the host saying retraining the model with the released stage 1 test data is allowed? I don't think we're supposed to add more new data other than the released stage 1 images after the stage 2 test images have been released - of course we wouldn't need to change our code, but I think we'd be in violation of rules this way.
", "rawMarkdown": "Is there a confirmation from the host saying retraining the model with the released stage 1 test data is allowed? I don't think we're supposed to add more new data other than the released stage 1 images after the stage 2 test images have been released - of course we wouldn't need to change our code, but I think we'd be in violation of rules this way.", "votes": null }, { "id": "313476", "postDate": "04/13/2018 10:10:42", "content": "Here is the direct answer: https://www.kaggle.com/c/data-science-bowl-2018/discussion/52960#303802
", "rawMarkdown": "Here is the direct answer: https://www.kaggle.com/c/data-science-bowl-2018/discussion/52960#303802", "votes": null }, { "id": "313477", "postDate": "04/13/2018 10:13:54", "content": "I am reading the rules of this competition (https://www.kaggle.com/c/data-science-bowl-2018/rules) and it states: \" If you are using a source of external data, you must post the source to the official external data forum thread no later than one week prior to the deadline of the first stage.\"
\n\nSo \"stage 2\" images + new masks would be a NEW external data set, I think CPMP was also making it clear with \"In practice I'll retrain with train and test from stage1. It is against the rules to use data not published in the external data thread before deadline. This includes any of the stage2 test data.\"!
\n\nThe question is rather is it allowed to use existing external data from https://www.kaggle.com/c/data-science-bowl-2018/discussion/47572 that was submitted before the deadline and is within the rules to use when they have not been used in stage1?
\n\nA statement from the Kaggle team would be helpful at this point.
", "rawMarkdown": "I am reading the rules of this competition (https://www.kaggle.com/c/data-science-bowl-2018/rules) and it states: \" If you are using a source of external data, you must post the source to the official external data forum thread no later than one week prior to the deadline of the first stage.\"\n\nSo \"stage 2\" images + new masks would be a NEW external data set, I think CPMP was also making it clear with \"In practice I'll retrain with train and test from stage1. It is against the rules to use data not published in the external data thread before deadline. This includes any of the stage2 test data.\"!\n\nThe question is rather is it allowed to use existing external data from https://www.kaggle.com/c/data-science-bowl-2018/discussion/47572 that was submitted before the deadline and is within the rules to use when they have not been used in stage1?\n\nA statement from the Kaggle team would be helpful at this point.", "votes": null }, { "id": "313478", "postDate": "04/13/2018 10:13:58", "content": "Yes, we not allowed to add new data after the stage 2 test images have been released. But as long as we already prepared our code to add stage1 test answers and retrain/tune with it before stage2 released - it is ok.
", "rawMarkdown": "Yes, we not allowed to add new data after the stage 2 test images have been released. But as long as we already prepared our code to add stage1 test answers and retrain/tune with it before stage2 released - it is ok.", "votes": null }, { "id": "313483", "postDate": "04/13/2018 10:23:31", "content": "\n\n\nYou are changing your decision about training data after looking at stage2 images.
\n
No I'm not. I have not yet downloaded the test data.
\n\nRe my intention, I've been asking if training with stage1 test was allowed, before end of stage1, just check the forum. And I also wrote in the forum before deadline that I will not use more external data as I don't know how to label it.
\n\nI won't lose time looking for the exact links, but they exist. I know that a trolling discussion like that can only get worse if I provide facts, as you already made your mind about me.
\n\nWhat is a bit weird is that I agree with you, upvoted you, and only added some precision to what you said. Using stage 2 is clearly against the rule as I wrote: It is against the rules to use data not published in the external data thread before deadline. This includes any of the stage2 test data.
\n\nMaybe you read my message wrongly?
", "rawMarkdown": "> You are changing your decision about training data after looking at stage2 images.\n\nNo I'm not. I have not yet downloaded the test data. \n\nRe my intention, I've been asking if training with stage1 test was allowed, before end of stage1, just check the forum. And I also wrote in the forum before deadline that I will not use more external data as I don't know how to label it. \n\nI won't lose time looking for the exact links, but they exist. I know that a trolling discussion like that can only get worse if I provide facts, as you already made your mind about me.\n\nWhat is a bit weird is that I agree with you, upvoted you, and only added some precision to what you said. Using stage 2 is clearly against the rule as I wrote: *It is against the rules to use data not published in the external data thread before deadline. This includes any of the stage2 test data.*\n\nMaybe you read my message wrongly?", "votes": null }, { "id": "313490", "postDate": "04/13/2018 10:39:06", "content": "\n\n\nI'm not surprised by stage2 images
\n \nI'll retrain my models on as much data as I can
\n
Hm.. maybe my english is really not so good..
", "rawMarkdown": "> I'm not surprised by stage2 images\n\n> I'll retrain my models on as much data as I can\n\nHm.. maybe my english is really not so good..", "votes": null }, { "id": "313524", "postDate": "04/13/2018 12:08:53", "content": "btw. the \"Stage 2 Data\" does not fall under the \"External Data\" category (imo)
\n\nas I understand it there are (always) two types of data:\n- Competition Data - provided by the organizer for the purpose of the competition; this includes both Stage 1 and Stage 2 data imo\n- External Data - data not provided by the organizer
\n\nAnd anyway training on pseudo-labelled data was always considered OK as far as I can tell
\n\nimo as long as you have made and uploaded your code to directly work with pseudo labelling (i.e. you have on epipeline which predicts th etets images and saves the pseudo data; 2nd pipeline which trains on stage 1 + stage 2 pseudo labelled + any external data) and a 3rd pipeline for predicting based on weights from step 2 you should be fine
", "rawMarkdown": "btw. the \"Stage 2 Data\" does not fall under the \"External Data\" category (imo)\n\nas I understand it there are (always) two types of data:\n- Competition Data - provided by the organizer for the purpose of the competition; this includes both Stage 1 and Stage 2 data imo\n- External Data - data not provided by the organizer\n\nAnd anyway training on pseudo-labelled data was always considered OK as far as I can tell\n\nimo as long as you have made and uploaded your code to directly work with pseudo labelling (i.e. you have on epipeline which predicts th etets images and saves the pseudo data; 2nd pipeline which trains on stage 1 + stage 2 pseudo labelled + any external data) and a 3rd pipeline for predicting based on weights from step 2 you should be fine", "votes": null }, { "id": "313533", "postDate": "04/13/2018 12:26:34", "content": "Interesting. I think it's obvious we should not re-select training data after test2 is released.
", "rawMarkdown": "Interesting. I think it's obvious we should not re-select training data after test2 is released.", "votes": null }, { "id": "313541", "postDate": "04/13/2018 12:39:09", "content": "An official statement from Kaggle staff would be useful here.
", "rawMarkdown": "An official statement from Kaggle staff would be useful here.", "votes": null }, { "id": "313542", "postDate": "04/13/2018 12:42:00", "content": "@YanWang retraining is allowed provided code is frozen before deadline, see answers from Wendy:
\n\n\n\n", "rawMarkdown": "YanWang retraining is allowed provided code is frozen before deadline, see answers from Wendy:\n\nhttps://www.kaggle.com/c/data-science-bowl-2018/forums/t/52385/confusion-about-contest-timeline-april-11-deadline?forumMessageId=309194#post309194\n\nhttps://www.kaggle.com/c/data-science-bowl-2018/forums/t/52960/question-about-stage-2?forumMessageId=303802#post303802", "votes": null }, { "id": "313546", "postDate": "04/13/2018 12:57:52", "content": "This is funny, I'm downvoted for citing official answers from Kaggle staff. If you disagree with their answers then don't kill the messenger. React to these answers instead.
", "rawMarkdown": "This is funny, I'm downvoted for citing official answers from Kaggle staff. If you disagree with their answers then don't kill the messenger. React to these answers instead.", "votes": null }, { "id": "313547", "postDate": "04/13/2018 12:59:43", "content": "@YanWang yes, it's obvious! Each single training sample could be treated as 'hyperparameter'. And whole training set for model is one big most important 'hyperparameter'. Of course everyone is able find the most similiar to stage2 images from hundreds of links in \"External Data\" thread. I belive administration is able to catch this cheating for 'in money' winners. But it will be unfair to to deprive someone a deserved medal.
", "rawMarkdown": "YanWang yes, it's obvious! Each single training sample could be treated as 'hyperparameter'. And whole training set for model is one big most important 'hyperparameter'. Of course everyone is able find the most similiar to stage2 images from hundreds of links in \"External Data\" thread. I belive administration is able to catch this cheating for 'in money' winners. But it will be unfair to to deprive someone a deserved medal.", "votes": null }, { "id": "313564", "postDate": "04/13/2018 13:41:13", "content": "To clarify, I never use downvote :)\n（edit: at least never in this thread）
", "rawMarkdown": "To clarify, I never use downvote :)\n（edit: at least never in this thread）", "votes": null }, { "id": "313566", "postDate": "04/13/2018 13:52:20", "content": "\n\n\nTo clarify, I never use downvote :)
\n
+1.
\n\nNeither do I.
\n\nTo clarify as well, it would have been way better to not allow retraining in stage2, and not share stage1 labels before end of stage2.
\n\nBut the rules are what they are, and anything allowed by them should be accepted, even if it may seem unfair. To your point, retraining should have been planned for before stage1 deadline. Selecting new data to train now is against the spirit of the competition IMHO.
\n\nIn my case, retraining on as much data as I can means using stage1 test labels, as I don't have any other labelled data at hand. And I planned for it before deadline.
", "rawMarkdown": "> To clarify, I never use downvote :)\n\n+1.\n\nNeither do I.\n\nTo clarify as well, it would have been way better to not allow retraining in stage2, and not share stage1 labels before end of stage2. \n\nBut the rules are what they are, and anything allowed by them should be accepted, even if it may seem unfair. To your point, retraining should have been planned for before stage1 deadline. Selecting new data to train now is against the spirit of the competition IMHO. \n\nIn my case, retraining on as much data as I can means using stage1 test labels, as I don't have any other labelled data at hand. And I planned for it before deadline.", "votes": null }, { "id": "313586", "postDate": "04/13/2018 14:13:21", "content": "@CPMP I agree, I don't think the stage 1 answers should be released before the end of the competition if retraining with the stage 1 test data is not allowed. And the Kaggle admin or the host should make it very clear if we're allowed to retrain our models with the stage 1 test data or not, if not, they should not release the answers at all. And that way, I don't see any point to retrain the models at the 2nd stage either. I'm still waiting for the answer from the admin/host.
", "rawMarkdown": "CPMP I agree, I don't think the stage 1 answers should be released before the end of the competition if retraining with the stage 1 test data is not allowed. And the Kaggle admin or the host should make it very clear if we're allowed to retrain our models with the stage 1 test data or not, if not, they should not release the answers at all. And that way, I don't see any point to retrain the models at the 2nd stage either. I'm still waiting for the answer from the admin/host.", "votes": null }, { "id": "313632", "postDate": "04/13/2018 15:47:45", "content": "A side effect is also that you cannot select your final 2 submissions based on LB score. You'll have to trust your local validation. I guess this is a whole new territory for some of the participants ;)
", "rawMarkdown": "A side effect is also that you cannot select your final 2 submissions based on LB score. You'll have to trust your local validation. I guess this is a whole new territory for some of the participants ;)", "votes": null }, { "id": "313671", "postDate": "04/13/2018 17:18:54", "content": "I'm so glad someone has finally said this. This stage 1 test release will sink many hours of our collective time. There has been much confusion already and models will likely need to be rerun. Its only likely to encourage bending of the rules too making things less locked in.
\n\nIt is only 10% extra data and I'd really rather just run on the original stage 1 train but will no doubt feel compelled to add the stage 1 test unless hear otherwise (code is ready to cope with it).
\n\nWell done to the organisers for a great competition with incredible engagement but this has been a headache and needs clear and detailed clarification asap.
", "rawMarkdown": "I'm so glad someone has finally said this. This stage 1 test release will sink many hours of our collective time. There has been much confusion already and models will likely need to be rerun. Its only likely to encourage bending of the rules too making things less locked in. \n\n\nIt is only 10% extra data and I'd really rather just run on the original stage 1 train but will no doubt feel compelled to add the stage 1 test unless hear otherwise (code is ready to cope with it). \n\n\nWell done to the organisers for a great competition with incredible engagement but this has been a headache and needs clear and detailed clarification asap.", "votes": null }, { "id": "313739", "postDate": "04/13/2018 19:18:09", "content": "What do you guys think about the current LB? Is it totally random, or It shows the real descending order but with 0.000-s?
", "rawMarkdown": "What do you guys think about the current LB? Is it totally random, or It shows the real descending order but with 0.000-s?", "votes": null }, { "id": "313741", "postDate": "04/13/2018 19:21:22", "content": "It's chronological. You can download the time stamp for everybody's submission and see.
", "rawMarkdown": "It's chronological. You can download the time stamp for everybody's submission and see.", "votes": null }, { "id": "313925", "postDate": "04/14/2018 05:00:59", "content": "There is no hidden public score, no image is used for public score, see https://www.kaggle.com/c/data-science-bowl-2018/discussion/54432#313734 or https://www.kaggle.com/c/data-science-bowl-2018/discussion/54315#313729
", "rawMarkdown": "There is no hidden public score, no image is used for public score, see https://www.kaggle.com/c/data-science-bowl-2018/discussion/54432#313734 or https://www.kaggle.com/c/data-science-bowl-2018/discussion/54315#313729", "votes": null } ], "comments": [ { "id": 313173, "author_name": "cpmpml", "author_url": "", "post_date": "04/12/2018 21:25:10", "content": "I may sound negative about this competition, but one thing I really enjoy is thtat no LB probing is possible in stage2. This is such a relief that I am almost speechless. I don't know who had the idea but having all 0 scores is brillant! I love it.
", "votes": null, "replies": [ { "id": 313180, "author_name": "robertkag", "author_url": "", "post_date": "04/12/2018 21:42:26", "content": "Agreed. I was concerned about this too. If the stage2 LB was visible now we'd be seeing people's scores slowly creeping up...
", "votes": null, "replies": [] }, { "id": 313632, "author_name": "cpmpml", "author_url": "", "post_date": "04/13/2018 15:47:45", "content": "A side effect is also that you cannot select your final 2 submissions based on LB score. You'll have to trust your local validation. I guess this is a whole new territory for some of the participants ;)
", "votes": null, "replies": [] } ] }, { "id": 313348, "author_name": "nicolefinnie", "author_url": "", "post_date": "04/13/2018 06:14:15", "content": "@CPMP I too love having 0 LB scores, that means we only need to submit our result once and done and enjoy our weekend.
", "votes": null, "replies": [ { "id": 313352, "author_name": "ybwu01", "author_url": "", "post_date": "04/13/2018 06:23:58", "content": "@Nicole Finnie, the only drawback is what if there is something wrong with format or something like that. I wish they could use a tiny data like 1% to make us sure there is no such thing.
", "votes": null, "replies": [] }, { "id": 313354, "author_name": "nhargan", "author_url": "", "post_date": "04/13/2018 06:34:02", "content": "Agree, but you still can compare results visually. Some new types of images looks totally incorrect for my MaskRCNN.
", "votes": null, "replies": [] } ] }, { "id": 313366, "author_name": "victorsd", "author_url": "", "post_date": "04/13/2018 07:04:39", "content": "I belive it is unfair and against the rules to add new data and this stage. It's ok to tune/retrain your models with stage1 answers. But it's clearly cheating to find new data similiar to stage2 (or even take from stage2 and label it). It's much worse than changing \"hyperparameters\".
", "votes": null, "replies": [ { "id": 313381, "author_name": "cpmpml", "author_url": "", "post_date": "04/13/2018 07:23:32", "content": "In practice I'll retrain with train and test from stage1. It is against the rules to use data not published in the external data thread before deadline. This includes any of the stage2 test data.
\n\nBut it does not seem to be against the rule to retrain models on data shared before deadline, provided you don't modify your code (eg adding a blurring preprocessing if you did not have one in stage1).
", "votes": null, "replies": [] }, { "id": 313384, "author_name": "victorsd", "author_url": "", "post_date": "04/13/2018 07:31:58", "content": "You are changing your decision about training data after looking at stage2 images. It's like tweaking hyperparameters to fit in stage2 data better. It's even worse. Your initial data and code should be frozen and ready just to take stage1 answers and train with them.
", "votes": null, "replies": [] }, { "id": 313457, "author_name": "nicolefinnie", "author_url": "", "post_date": "04/13/2018 09:42:27", "content": "Is there a confirmation from the host saying retraining the model with the released stage 1 test data is allowed? I don't think we're supposed to add more new data other than the released stage 1 images after the stage 2 test images have been released - of course we wouldn't need to change our code, but I think we'd be in violation of rules this way.
", "votes": null, "replies": [] }, { "id": 313476, "author_name": "victorsd", "author_url": "", "post_date": "04/13/2018 10:10:42", "content": "Here is the direct answer: https://www.kaggle.com/c/data-science-bowl-2018/discussion/52960#303802
", "votes": null, "replies": [] }, { "id": 313477, "author_name": "hafeneger", "author_url": "", "post_date": "04/13/2018 10:13:54", "content": "I am reading the rules of this competition (https://www.kaggle.com/c/data-science-bowl-2018/rules) and it states: \" If you are using a source of external data, you must post the source to the official external data forum thread no later than one week prior to the deadline of the first stage.\"
\n\nSo \"stage 2\" images + new masks would be a NEW external data set, I think CPMP was also making it clear with \"In practice I'll retrain with train and test from stage1. It is against the rules to use data not published in the external data thread before deadline. This includes any of the stage2 test data.\"!
\n\nThe question is rather is it allowed to use existing external data from https://www.kaggle.com/c/data-science-bowl-2018/discussion/47572 that was submitted before the deadline and is within the rules to use when they have not been used in stage1?
\n\nA statement from the Kaggle team would be helpful at this point.
", "votes": null, "replies": [] }, { "id": 313478, "author_name": "victorsd", "author_url": "", "post_date": "04/13/2018 10:13:58", "content": "Yes, we not allowed to add new data after the stage 2 test images have been released. But as long as we already prepared our code to add stage1 test answers and retrain/tune with it before stage2 released - it is ok.
", "votes": null, "replies": [] }, { "id": 313483, "author_name": "cpmpml", "author_url": "", "post_date": "04/13/2018 10:23:31", "content": "\n\n\nYou are changing your decision about training data after looking at stage2 images.
\n
No I'm not. I have not yet downloaded the test data.
\n\nRe my intention, I've been asking if training with stage1 test was allowed, before end of stage1, just check the forum. And I also wrote in the forum before deadline that I will not use more external data as I don't know how to label it.
\n\nI won't lose time looking for the exact links, but they exist. I know that a trolling discussion like that can only get worse if I provide facts, as you already made your mind about me.
\n\nWhat is a bit weird is that I agree with you, upvoted you, and only added some precision to what you said. Using stage 2 is clearly against the rule as I wrote: It is against the rules to use data not published in the external data thread before deadline. This includes any of the stage2 test data.
\n\nMaybe you read my message wrongly?
", "votes": null, "replies": [] }, { "id": 313490, "author_name": "victorsd", "author_url": "", "post_date": "04/13/2018 10:39:06", "content": "\n\n\nI'm not surprised by stage2 images
\n \nI'll retrain my models on as much data as I can
\n
Hm.. maybe my english is really not so good..
", "votes": null, "replies": [] }, { "id": 313533, "author_name": "wayne999", "author_url": "", "post_date": "04/13/2018 12:26:34", "content": "Interesting. I think it's obvious we should not re-select training data after test2 is released.
", "votes": null, "replies": [] }, { "id": 313542, "author_name": "cpmpml", "author_url": "", "post_date": "04/13/2018 12:42:00", "content": "@YanWang retraining is allowed provided code is frozen before deadline, see answers from Wendy:
\n\n\n\n", "votes": null, "replies": [] }, { "id": 313546, "author_name": "cpmpml", "author_url": "", "post_date": "04/13/2018 12:57:52", "content": "This is funny, I'm downvoted for citing official answers from Kaggle staff. If you disagree with their answers then don't kill the messenger. React to these answers instead.
", "votes": null, "replies": [] }, { "id": 313547, "author_name": "victorsd", "author_url": "", "post_date": "04/13/2018 12:59:43", "content": "@YanWang yes, it's obvious! Each single training sample could be treated as 'hyperparameter'. And whole training set for model is one big most important 'hyperparameter'. Of course everyone is able find the most similiar to stage2 images from hundreds of links in \"External Data\" thread. I belive administration is able to catch this cheating for 'in money' winners. But it will be unfair to to deprive someone a deserved medal.
", "votes": null, "replies": [] }, { "id": 313564, "author_name": "wayne999", "author_url": "", "post_date": "04/13/2018 13:41:13", "content": "To clarify, I never use downvote :)\n（edit: at least never in this thread）
", "votes": null, "replies": [] }, { "id": 313566, "author_name": "cpmpml", "author_url": "", "post_date": "04/13/2018 13:52:20", "content": "\n\n\nTo clarify, I never use downvote :)
\n
+1.
\n\nNeither do I.
\n\nTo clarify as well, it would have been way better to not allow retraining in stage2, and not share stage1 labels before end of stage2.
\n\nBut the rules are what they are, and anything allowed by them should be accepted, even if it may seem unfair. To your point, retraining should have been planned for before stage1 deadline. Selecting new data to train now is against the spirit of the competition IMHO.
\n\nIn my case, retraining on as much data as I can means using stage1 test labels, as I don't have any other labelled data at hand. And I planned for it before deadline.
", "votes": null, "replies": [] }, { "id": 313586, "author_name": "nicolefinnie", "author_url": "", "post_date": "04/13/2018 14:13:21", "content": "@CPMP I agree, I don't think the stage 1 answers should be released before the end of the competition if retraining with the stage 1 test data is not allowed. And the Kaggle admin or the host should make it very clear if we're allowed to retrain our models with the stage 1 test data or not, if not, they should not release the answers at all. And that way, I don't see any point to retrain the models at the 2nd stage either. I'm still waiting for the answer from the admin/host.
", "votes": null, "replies": [] }, { "id": 313671, "author_name": "stanwar", "author_url": "", "post_date": "04/13/2018 17:18:54", "content": "I'm so glad someone has finally said this. This stage 1 test release will sink many hours of our collective time. There has been much confusion already and models will likely need to be rerun. Its only likely to encourage bending of the rules too making things less locked in.
\n\nIt is only 10% extra data and I'd really rather just run on the original stage 1 train but will no doubt feel compelled to add the stage 1 test unless hear otherwise (code is ready to cope with it).
\n\nWell done to the organisers for a great competition with incredible engagement but this has been a headache and needs clear and detailed clarification asap.
", "votes": null, "replies": [] } ] }, { "id": 313524, "author_name": "steelrose", "author_url": "", "post_date": "04/13/2018 12:08:53", "content": "btw. the \"Stage 2 Data\" does not fall under the \"External Data\" category (imo)
\n\nas I understand it there are (always) two types of data:\n- Competition Data - provided by the organizer for the purpose of the competition; this includes both Stage 1 and Stage 2 data imo\n- External Data - data not provided by the organizer
\n\nAnd anyway training on pseudo-labelled data was always considered OK as far as I can tell
\n\nimo as long as you have made and uploaded your code to directly work with pseudo labelling (i.e. you have on epipeline which predicts th etets images and saves the pseudo data; 2nd pipeline which trains on stage 1 + stage 2 pseudo labelled + any external data) and a 3rd pipeline for predicting based on weights from step 2 you should be fine
", "votes": null, "replies": [ { "id": 313541, "author_name": "cpmpml", "author_url": "", "post_date": "04/13/2018 12:39:09", "content": "An official statement from Kaggle staff would be useful here.
", "votes": null, "replies": [] } ] }, { "id": 313739, "author_name": "fkovacs", "author_url": "", "post_date": "04/13/2018 19:18:09", "content": "What do you guys think about the current LB? Is it totally random, or It shows the real descending order but with 0.000-s?
", "votes": null, "replies": [ { "id": 313741, "author_name": "pliptor", "author_url": "", "post_date": "04/13/2018 19:21:22", "content": "It's chronological. You can download the time stamp for everybody's submission and see.
", "votes": null, "replies": [] }, { "id": 313925, "author_name": "cpmpml", "author_url": "", "post_date": "04/14/2018 05:00:59", "content": "There is no hidden public score, no image is used for public score, see https://www.kaggle.com/c/data-science-bowl-2018/discussion/54432#313734 or https://www.kaggle.com/c/data-science-bowl-2018/discussion/54315#313729
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "313170": "This competition had way too little data to train neural nets reliably. I have absolutely no clue about how my stuff will fare, but I'm not surprised by stage2 images. They are different from stage1, as announced.\n\nI don't think the competition setting is right to find most robust models. I would rather have looked for robust model training, i.e. developping fully automated training pipeline, and provide stage2 training data representative of unforeseen data. \n\nExpecting us to find models robust to new types of stain/microscope without representative trianing data is just a misconception about what machine learning can do.\n\nThis said, I'll retrain my models on as much data as I can ;)", "313173": "I may sound negative about this competition, but one thing I really enjoy is thtat no LB probing is possible in stage2. This is such a relief that I am almost speechless. I don't know who had the idea but having all 0 scores is brillant! I love it.", "313180": "Agreed. I was concerned about this too. If the stage2 LB was visible now we'd be seeing people's scores slowly creeping up...", "313348": "CPMP I too love having 0 LB scores, that means we only need to submit our result once and done and enjoy our weekend.", "313352": "Nicole Finnie, the only drawback is what if there is something wrong with format or something like that. I wish they could use a tiny data like 1% to make us sure there is no such thing.", "313354": "Agree, but you still can compare results visually. Some new types of images looks totally incorrect for my MaskRCNN.", "313366": "I belive it is unfair and against the rules to add new data and this stage. It's ok to tune/retrain your models with stage1 answers. But it's clearly cheating to find new data similiar to stage2 (or even take from stage2 and label it). It's much worse than changing \"hyperparameters\".", "313381": "In practice I'll retrain with train and test from stage1. It is against the rules to use data not published in the external data thread before deadline. This includes any of the stage2 test data.\n\nBut it does not seem to be against the rule to retrain models on data shared before deadline, provided you don't modify your code (eg adding a blurring preprocessing if you did not have one in stage1).", "313384": "You are changing your decision about training data after looking at stage2 images. It's like tweaking hyperparameters to fit in stage2 data better. It's even worse. Your initial data and code should be frozen and ready just to take stage1 answers and train with them.", "313457": "Is there a confirmation from the host saying retraining the model with the released stage 1 test data is allowed? I don't think we're supposed to add more new data other than the released stage 1 images after the stage 2 test images have been released - of course we wouldn't need to change our code, but I think we'd be in violation of rules this way.", "313476": "Here is the direct answer: https://www.kaggle.com/c/data-science-bowl-2018/discussion/52960#303802", "313477": "I am reading the rules of this competition (https://www.kaggle.com/c/data-science-bowl-2018/rules) and it states: \" If you are using a source of external data, you must post the source to the official external data forum thread no later than one week prior to the deadline of the first stage.\"\n\nSo \"stage 2\" images + new masks would be a NEW external data set, I think CPMP was also making it clear with \"In practice I'll retrain with train and test from stage1. It is against the rules to use data not published in the external data thread before deadline. This includes any of the stage2 test data.\"!\n\nThe question is rather is it allowed to use existing external data from https://www.kaggle.com/c/data-science-bowl-2018/discussion/47572 that was submitted before the deadline and is within the rules to use when they have not been used in stage1?\n\nA statement from the Kaggle team would be helpful at this point.", "313478": "Yes, we not allowed to add new data after the stage 2 test images have been released. But as long as we already prepared our code to add stage1 test answers and retrain/tune with it before stage2 released - it is ok.", "313483": "> You are changing your decision about training data after looking at stage2 images.\n\nNo I'm not. I have not yet downloaded the test data. \n\nRe my intention, I've been asking if training with stage1 test was allowed, before end of stage1, just check the forum. And I also wrote in the forum before deadline that I will not use more external data as I don't know how to label it. \n\nI won't lose time looking for the exact links, but they exist. I know that a trolling discussion like that can only get worse if I provide facts, as you already made your mind about me.\n\nWhat is a bit weird is that I agree with you, upvoted you, and only added some precision to what you said. Using stage 2 is clearly against the rule as I wrote: *It is against the rules to use data not published in the external data thread before deadline. This includes any of the stage2 test data.*\n\nMaybe you read my message wrongly?", "313490": "> I'm not surprised by stage2 images\n\n> I'll retrain my models on as much data as I can\n\nHm.. maybe my english is really not so good..", "313524": "btw. the \"Stage 2 Data\" does not fall under the \"External Data\" category (imo)\n\nas I understand it there are (always) two types of data:\n- Competition Data - provided by the organizer for the purpose of the competition; this includes both Stage 1 and Stage 2 data imo\n- External Data - data not provided by the organizer\n\nAnd anyway training on pseudo-labelled data was always considered OK as far as I can tell\n\nimo as long as you have made and uploaded your code to directly work with pseudo labelling (i.e. you have on epipeline which predicts th etets images and saves the pseudo data; 2nd pipeline which trains on stage 1 + stage 2 pseudo labelled + any external data) and a 3rd pipeline for predicting based on weights from step 2 you should be fine", "313533": "Interesting. I think it's obvious we should not re-select training data after test2 is released.", "313541": "An official statement from Kaggle staff would be useful here.", "313542": "YanWang retraining is allowed provided code is frozen before deadline, see answers from Wendy:\n\nhttps://www.kaggle.com/c/data-science-bowl-2018/forums/t/52385/confusion-about-contest-timeline-april-11-deadline?forumMessageId=309194#post309194\n\nhttps://www.kaggle.com/c/data-science-bowl-2018/forums/t/52960/question-about-stage-2?forumMessageId=303802#post303802", "313546": "This is funny, I'm downvoted for citing official answers from Kaggle staff. If you disagree with their answers then don't kill the messenger. React to these answers instead.", "313547": "YanWang yes, it's obvious! Each single training sample could be treated as 'hyperparameter'. And whole training set for model is one big most important 'hyperparameter'. Of course everyone is able find the most similiar to stage2 images from hundreds of links in \"External Data\" thread. I belive administration is able to catch this cheating for 'in money' winners. But it will be unfair to to deprive someone a deserved medal.", "313564": "To clarify, I never use downvote :)\n（edit: at least never in this thread）", "313566": "> To clarify, I never use downvote :)\n\n+1.\n\nNeither do I.\n\nTo clarify as well, it would have been way better to not allow retraining in stage2, and not share stage1 labels before end of stage2. \n\nBut the rules are what they are, and anything allowed by them should be accepted, even if it may seem unfair. To your point, retraining should have been planned for before stage1 deadline. Selecting new data to train now is against the spirit of the competition IMHO. \n\nIn my case, retraining on as much data as I can means using stage1 test labels, as I don't have any other labelled data at hand. And I planned for it before deadline.", "313586": "CPMP I agree, I don't think the stage 1 answers should be released before the end of the competition if retraining with the stage 1 test data is not allowed. And the Kaggle admin or the host should make it very clear if we're allowed to retrain our models with the stage 1 test data or not, if not, they should not release the answers at all. And that way, I don't see any point to retrain the models at the 2nd stage either. I'm still waiting for the answer from the admin/host.", "313632": "A side effect is also that you cannot select your final 2 submissions based on LB score. You'll have to trust your local validation. I guess this is a whole new territory for some of the participants ;)", "313671": "I'm so glad someone has finally said this. This stage 1 test release will sink many hours of our collective time. There has been much confusion already and models will likely need to be rerun. Its only likely to encourage bending of the rules too making things less locked in. \n\n\nIt is only 10% extra data and I'd really rather just run on the original stage 1 train but will no doubt feel compelled to add the stage 1 test unless hear otherwise (code is ready to cope with it). \n\n\nWell done to the organisers for a great competition with incredible engagement but this has been a headache and needs clear and detailed clarification asap.", "313739": "What do you guys think about the current LB? Is it totally random, or It shows the real descending order but with 0.000-s?", "313741": "It's chronological. You can download the time stamp for everybody's submission and see.", "313925": "There is no hidden public score, no image is used for public score, see https://www.kaggle.com/c/data-science-bowl-2018/discussion/54432#313734 or https://www.kaggle.com/c/data-science-bowl-2018/discussion/54315#313729" }, "source": "meta" }