![\"enter](\"https://storage.googleapis.com/kaggle-forum-message-attachments/316295/9213/limit.png\")
I was the person that belabored the point that after some AP threshold (say 0.5) performance was substantially less important than bias. Nevertheless, I’m curious.
\n\nWhat is the best possible AP (without substantial architectural changes, supplementary data, or infinite training time, etc.)? 0.7? Higher? 🤔
", "messages": [ { "id": "315842", "postDate": "04/17/2018 16:48:59", "content": "I was the person that belabored the point that after some AP threshold (say 0.5) performance was substantially less important than bias. Nevertheless, I’m curious.
\n\nWhat is the best possible AP (without substantial architectural changes, supplementary data, or infinite training time, etc.)? 0.7? Higher? 🤔
", "rawMarkdown": "I was the person that belabored the point that after some AP threshold (say 0.5) performance was substantially less important than bias. Nevertheless, I’m curious.\n\nWhat is the best possible AP (without substantial architectural changes, supplementary data, or infinite training time, etc.)? 0.7? Higher? 🤔", "votes": null }, { "id": "315868", "postDate": "04/17/2018 17:39:48", "content": "This contest had a massive data leak, imho. Congratulations to 1st place for exploiting this. I learned this lesson my last year as an undergraduate, there will be additional information (bias?) Introduced in data that is generated from raw data. I forgot this lesson, until i read the 1st place solution.
\n\nagain Congratulations to 1st place winners. very clever.
\n\nS.B. Allen, When you do this
\n\nI also supplemented the training set in an attempt to standardize annotation disagreements:
\n\nI computed a distance transform on each of the masks (not the flattened label matrix) and found the maximum peak. I computed a watershed segmentation for each masked image (that was dilated with a (1, 1) structuring element) using the peaks as markers. I added this new set of masks to the existing set.
\n\nI took the centroid of each mask and used the random walker algorithm to compute a third set of masks using the aforementioned masked images. I then used the same watershed process from before.
\n\nFinally, I created a fourth and fifth set of annotations by using a geodesic active contour with two different edge preserved responses (from a median filter and an inverse Gaussian filter). I then used the same watershed process from before.
\n\nyou are getting rid of most (maybe) of this bias in the \"ground truth\"
\n\nI will continue to work on this problem, at first probably retracing your first steps\n- vanilla U-net (done) \n- vanilla matterhorn f-MSK (doing)\n- vanilla broadhead f-msk (tote done)\n- vanilla slide cam (blackbox)\n- research results in the future
", "rawMarkdown": "This contest had a massive data leak, imho. Congratulations to 1st place for exploiting this. I learned this lesson my last year as an undergraduate, there will be additional information (bias?) Introduced in data that is generated from raw data. I forgot this lesson, until i read the 1st place solution. \n\nagain Congratulations to 1st place winners. very clever.\n\nS.B. Allen, When you do this\n\n I also supplemented the training set in an attempt to standardize annotation disagreements:\n\n I computed a distance transform on each of the masks (not the flattened label matrix) and found the maximum peak. I computed a watershed segmentation for each masked image (that was dilated with a (1, 1) structuring element) using the peaks as markers. I added this new set of masks to the existing set.\n\nI took the centroid of each mask and used the random walker algorithm to compute a third set of masks using the aforementioned masked images. I then used the same watershed process from before. \n\nFinally, I created a fourth and fifth set of annotations by using a geodesic active contour with two different edge preserved responses (from a median filter and an inverse Gaussian filter). I then used the same watershed process from before.\n\nyou are getting rid of most (maybe) of this bias in the \"ground truth\"\n\nI will continue to work on this problem, at first probably retracing your first steps\n- vanilla U-net (done) \n- vanilla matterhorn f-MSK (doing)\n- vanilla broadhead f-msk (tote done)\n- vanilla slide cam (blackbox)\n- research results in the future", "votes": null }, { "id": "315876", "postDate": "04/17/2018 18:05:32", "content": "could you please be more specific about the leak? what do you mean?
", "rawMarkdown": "could you please be more specific about the leak? what do you mean?", "votes": null }, { "id": "315885", "postDate": "04/17/2018 18:23:02", "content": "Did you at least read the solution of the top-1 command? https://www.kaggle.com/c/data-science-bowl-2018/discussion/54741
", "rawMarkdown": "Did you at least read the solution of the top-1 command? https://www.kaggle.com/c/data-science-bowl-2018/discussion/54741", "votes": null }, { "id": "315891", "postDate": "04/17/2018 18:33:14", "content": "Seems that it's not necessary to use mass of specific techniques and approaches in order to be in top, isn't it? Moreover it's so, if by using them you still can not achieve appropriate results. \nAnd well, before blaming someone who solved the problem which you didn't(read as \"smarter\"), it's better to find conclusive proofs
", "rawMarkdown": "Seems that it's not necessary to use mass of specific techniques and approaches in order to be in top, isn't it? Moreover it's so, if by using them you still can not achieve appropriate results. \nAnd well, before blaming someone who solved the problem which you didn't(read as \"smarter\"), it's better to find conclusive proofs", "votes": null }, { "id": "315895", "postDate": "04/17/2018 18:47:20", "content": "yes. sorry. perhaps data leak is too strong or wrong a term. the goal was too maximize score of IOU of masks generated by humans/tool not to generate masks of cells. While you might do better at generating cell masks with your tool/model, the LB score would not necessarily indicate so.
\n\nto be clear, ( i am up too -5 votes and counting), i think this has been one of best kaggle contests ever, in terms of learning. This is SOA data from Broadhead.!!!!
\n\nand i thought what 1st place did was very cool. to imply they were exploiting a data leak was wrong as it was an assumption on my part. it would be simpler to assume they trying to get a maximal score on some internal metric they might have derived.
\n\nI was just making a comment, that was not discussed, except by Heng, who labeled it gray/grey box, that the goal to maximizing IOU to a tool not the raw data. to go further, perhaps an ensemble generated of masks generated using Allen's methods would have generated masks without so much bias to the tool that generated the masks.
", "rawMarkdown": "yes. sorry. perhaps data leak is too strong or wrong a term. the goal was too maximize score of IOU of **masks** generated by humans/tool not to generate masks of cells. While you might do better at generating cell masks with your tool/model, the LB score would not necessarily indicate so. \n\nto be clear, ( i am up too -5 votes and counting), i think this has been one of best kaggle contests ever, in terms of learning. This is SOA data from Broadhead.!!!! \n\nand i thought what 1st place did was very cool. to imply they were exploiting a data leak was wrong as it was an assumption on my part. it would be simpler to assume they trying to get a maximal score on some internal metric they might have derived.\n\nI was just making a comment, that was not discussed, except by Heng, who labeled it gray/grey box, that the goal to maximizing IOU to a tool not the raw data. to go further, perhaps an ensemble generated of masks generated using Allen's methods would have generated masks without so much bias to the tool that generated the masks.", "votes": null }, { "id": "315897", "postDate": "04/17/2018 18:50:43", "content": "yes.
", "rawMarkdown": "yes.", "votes": null }, { "id": "315898", "postDate": "04/17/2018 18:53:12", "content": "rereading my post. i now realize how insulting it was. i actually do think that 1st place solution was very clever and i learned a lot from it.
", "rawMarkdown": "rereading my post. i now realize how insulting it was. i actually do think that 1st place solution was very clever and i learned a lot from it.", "votes": null }, { "id": "315901", "postDate": "04/17/2018 18:59:49", "content": "i agree with sentence #1, disagree with #2 and agree with #3 (in that i agree that ones should state facts not assumptions)
", "rawMarkdown": "i agree with sentence #1, disagree with #2 and agree with #3 (in that i agree that ones should state facts not assumptions)", "votes": null }, { "id": "315904", "postDate": "04/17/2018 19:08:40", "content": "Well, a week before stage 1 end we had an experiment on public LB, same masks.
\n\nwithout FP removal 576
with FP removal 577
Not a huge gain :) But as you see even without this \"data leak\" it was top-1 on public as well (even one week before stage1 end).
\n\nDo you think this postprocessing gave us a jump, let's say from 0.500 to 0.631 on stage 2?
", "rawMarkdown": "Well, a week before stage 1 end we had an experiment on public LB, same masks.\n\n1. without FP removal 576\n\n2. with FP removal 577\n\nNot a huge gain :) But as you see even without this \"data leak\" it was top-1 on public as well (even one week before stage1 end). \n\nDo you think this postprocessing gave us a jump, let's say from 0.500 to 0.631 on stage 2?", "votes": null }, { "id": "315909", "postDate": "04/17/2018 19:30:33", "content": "good experiment. i don't know. probably not. i am middle coding to try score on pre-lupkin, to try something like this.
\n\nnote on AM NOT stating that g.t. is not ok. i am stating that you can not get a better score than gt masks. if you had generated these masks 2 years they would have been ok , but not as good as now. i am way off the goal of this competition. i am trying to improve SOA, not get close to it. sorry, if this sounds arrogant, but i think this is a very interesting set of raw data, especially the stage 2 test data.
\n\n(this should generate some negative votes.. glad i am not running for president :))
", "rawMarkdown": "good experiment. i don't know. probably not. i am middle coding to try score on pre-lupkin, to try something like this.\n\nnote on AM NOT stating that g.t. is not ok. i am stating that you can not get a better score than gt masks. if you had generated these masks 2 years they would have been *ok* , but not as good as now. i am way off the goal of this competition. i am trying to improve SOA, not get close to it. sorry, if this sounds arrogant, but i think this is a very interesting set of raw data, especially the stage 2 test data.\n\n(this should generate some negative votes.. glad i am not running for president :))", "votes": null }, { "id": "316295", "postDate": "04/18/2018 16:44:27", "content": "@ Bcottman
\n\nI think what @topcoder did is correct. Understanding the metric is important and i think it is ok to post process your results to increase the metric score. The objective of kaggle is to find the upper limit.
\n\nIf there is problem with the rule or metric , we can fix it later. Before the rule and metric are fixed, we should stick to it (even if there may be problem). The metric of this competition is mAP over IOU 0.5 to 1.0 and we should try all methods to \"exploit it\" to find the upper limit, and then understand why we had the score. After which, we just need to be aware of the metric problem,etc and fix it.
\n\nJust to name a few example:
\n\nin caltech pedestrian detection in computer vision, the pedestrian box were re-annotated to accommodate detection of occluded pedestrian
pascal VOC 2007 mAP metric computation is improved in VOC 2012
Evaluation itself is a difficult process. We can only improve it as we get along. Metrics/labels can not be 100% perfect and of course, one should always be careful when interpreting ML results.
\n\nFurther, the fact that @topcoder solution has good results on both stage1 test and stage2 test show that their solution works! They have very implementation techniques and very high observation skills.
\n\nML solution is not only about algorithm design, it is also about \"dealing with data\" (e.g. pre, post processing, metric understanding). @topcoder certainly \"deal with data very well\"
\n\n
ok... yes, ..
\n\nas i have done before and all do again i wrote an email in awe/insight/shock before realizing my \"sharing\"might be perceived as insulting to others.
\n\nyou see i have been fooling around with this problem (identifying cells) since 1980. i started out working with an assoc. professor wiring together the backplanes of modified pdp-11s we got them unto 64kb . that is kilobits, when the std high end was 32 kb). any way i dropped out because i had this pesky thing called class load. i heard they were able to distinguish between 4 types of white cells. anyway i (erroronosly) thought computers were to slow and went onto other things.... enough about the good ole days.
\n\nStarting around January, i got someone i know, to send me 40 or so florescent cell images. being an old-timer, it did not occur to me to just search the web.... as i was delighted (as well as the person whose arm i twisted) to find out i could download off the web. from reading discussion threads here.
\n\nyou see i was really really delighted to see 700 some images made available by Broadhead and more importantly the current home of Cellfinder. When Dr. Carpenter refers to hard work to provide the images, i estimate 2 hours for mask(s) for an image with 20 cells with tools. also the blurry images you saw in stage 2 test set are typical not exceptional. (for me anyway.)
\n\ni imaged the Nobel prize (ok maybe a little exaggerated) when i saw Allen's idea of augmented/ensemble/averaging masks, because it is such a great way to remove tool bias.
\n\nAs i was reading up in this area, imagine my excitement when i saw a kaggle completion with a stated goal..\n Teams will create a computer model that can identify a range of nuclei across varied conditions. By observing patterns, asking questions, and building a model, participants will have a chance to push state-of-the-art technology farther.
\n\nso all you state above i think it is correct (well i don't agree with your figure at the end, and before anybody asks no i won't expand). i will also add that i missed the goal of this completion. which having said that, i must also state, i am not missing my goal. (yet)
\n\nAnd i will also add the top 100 or so in this contest did this in 3-months (or less) of work (on free-ware), which a group of 5 of us could not even get close to 38 years ago with a $200,000 grant. The tools today are sci-if. i look forward to seeing advances in the next 20 years.
", "rawMarkdown": "ok... yes, ..\n\n as i have done before and all do again i wrote an email in awe/insight/shock before realizing my \"sharing\"might be perceived as insulting to others. \n\nyou see i have been fooling around with this problem (identifying cells) since 1980. i started out working with an assoc. professor wiring together the backplanes of modified pdp-11s we got them unto 64kb . that is kilobits, when the std high end was 32 kb). any way i dropped out because i had this pesky thing called class load. i heard they were able to distinguish between 4 types of white cells. anyway i (erroronosly) thought computers were to slow and went onto other things.... enough about the good ole days.\n\nStarting around January, i got someone i know, to send me 40 or so florescent cell images. being an old-timer, it did not occur to me to just search the web.... as i was delighted (as well as the person whose arm i twisted) to find out i could download off the web. from reading discussion threads here.\n\nyou see i was really really delighted to see 700 some images made available by Broadhead and more importantly the current home of Cellfinder. When Dr. Carpenter refers to hard work to provide the images, i estimate 2 hours for mask(s) for an image with 20 cells with tools. also the blurry images you saw in stage 2 test set are typical not exceptional. (for me anyway.)\n\ni imaged the Nobel prize (ok maybe a little exaggerated) when i saw Allen's idea of augmented/ensemble/averaging masks, because it is such a great way to remove tool bias.\n\nAs i was reading up in this area, imagine my excitement when i saw a kaggle completion with a stated goal..\n *Teams will create a computer model that can identify a range of nuclei across varied conditions. By observing patterns, asking questions, and building a model, participants will have a chance to push state-of-the-art technology farther.*\n\nso all you state above i think it is correct (well i don't agree with your figure at the end, and before anybody asks no i won't expand). i will also add that i missed the goal of this completion. which having said that, i must also state, i am not missing my goal. (yet)\n\nAnd i will also add the top 100 or so in this contest did this in 3-months (or less) of work (on free-ware), which a group of 5 of us could not even get close to 38 years ago with a $200,000 grant. The tools today are sci-if. i look forward to seeing advances in the next 20 years.", "votes": null } ], "comments": [ { "id": 315868, "author_name": "bcottman", "author_url": "", "post_date": "04/17/2018 17:39:48", "content": "This contest had a massive data leak, imho. Congratulations to 1st place for exploiting this. I learned this lesson my last year as an undergraduate, there will be additional information (bias?) Introduced in data that is generated from raw data. I forgot this lesson, until i read the 1st place solution.
\n\nagain Congratulations to 1st place winners. very clever.
\n\nS.B. Allen, When you do this
\n\nI also supplemented the training set in an attempt to standardize annotation disagreements:
\n\nI computed a distance transform on each of the masks (not the flattened label matrix) and found the maximum peak. I computed a watershed segmentation for each masked image (that was dilated with a (1, 1) structuring element) using the peaks as markers. I added this new set of masks to the existing set.
\n\nI took the centroid of each mask and used the random walker algorithm to compute a third set of masks using the aforementioned masked images. I then used the same watershed process from before.
\n\nFinally, I created a fourth and fifth set of annotations by using a geodesic active contour with two different edge preserved responses (from a median filter and an inverse Gaussian filter). I then used the same watershed process from before.
\n\nyou are getting rid of most (maybe) of this bias in the \"ground truth\"
\n\nI will continue to work on this problem, at first probably retracing your first steps\n- vanilla U-net (done) \n- vanilla matterhorn f-MSK (doing)\n- vanilla broadhead f-msk (tote done)\n- vanilla slide cam (blackbox)\n- research results in the future
", "votes": null, "replies": [ { "id": 315876, "author_name": "", "author_url": "", "post_date": "04/17/2018 18:05:32", "content": "could you please be more specific about the leak? what do you mean?
", "votes": null, "replies": [] }, { "id": 315885, "author_name": "artgor", "author_url": "", "post_date": "04/17/2018 18:23:02", "content": "Did you at least read the solution of the top-1 command? https://www.kaggle.com/c/data-science-bowl-2018/discussion/54741
", "votes": null, "replies": [] }, { "id": 315891, "author_name": "vladimirgroza", "author_url": "", "post_date": "04/17/2018 18:33:14", "content": "Seems that it's not necessary to use mass of specific techniques and approaches in order to be in top, isn't it? Moreover it's so, if by using them you still can not achieve appropriate results. \nAnd well, before blaming someone who solved the problem which you didn't(read as \"smarter\"), it's better to find conclusive proofs
", "votes": null, "replies": [] }, { "id": 315895, "author_name": "bcottman", "author_url": "", "post_date": "04/17/2018 18:47:20", "content": "yes. sorry. perhaps data leak is too strong or wrong a term. the goal was too maximize score of IOU of masks generated by humans/tool not to generate masks of cells. While you might do better at generating cell masks with your tool/model, the LB score would not necessarily indicate so.
\n\nto be clear, ( i am up too -5 votes and counting), i think this has been one of best kaggle contests ever, in terms of learning. This is SOA data from Broadhead.!!!!
\n\nand i thought what 1st place did was very cool. to imply they were exploiting a data leak was wrong as it was an assumption on my part. it would be simpler to assume they trying to get a maximal score on some internal metric they might have derived.
\n\nI was just making a comment, that was not discussed, except by Heng, who labeled it gray/grey box, that the goal to maximizing IOU to a tool not the raw data. to go further, perhaps an ensemble generated of masks generated using Allen's methods would have generated masks without so much bias to the tool that generated the masks.
", "votes": null, "replies": [] }, { "id": 315897, "author_name": "bcottman", "author_url": "", "post_date": "04/17/2018 18:50:43", "content": "yes.
", "votes": null, "replies": [] }, { "id": 315898, "author_name": "bcottman", "author_url": "", "post_date": "04/17/2018 18:53:12", "content": "rereading my post. i now realize how insulting it was. i actually do think that 1st place solution was very clever and i learned a lot from it.
", "votes": null, "replies": [] }, { "id": 315901, "author_name": "bcottman", "author_url": "", "post_date": "04/17/2018 18:59:49", "content": "i agree with sentence #1, disagree with #2 and agree with #3 (in that i agree that ones should state facts not assumptions)
", "votes": null, "replies": [] }, { "id": 315904, "author_name": "selimsef", "author_url": "", "post_date": "04/17/2018 19:08:40", "content": "Well, a week before stage 1 end we had an experiment on public LB, same masks.
\n\nwithout FP removal 576
with FP removal 577
Not a huge gain :) But as you see even without this \"data leak\" it was top-1 on public as well (even one week before stage1 end).
\n\nDo you think this postprocessing gave us a jump, let's say from 0.500 to 0.631 on stage 2?
", "votes": null, "replies": [] }, { "id": 315909, "author_name": "bcottman", "author_url": "", "post_date": "04/17/2018 19:30:33", "content": "good experiment. i don't know. probably not. i am middle coding to try score on pre-lupkin, to try something like this.
\n\nnote on AM NOT stating that g.t. is not ok. i am stating that you can not get a better score than gt masks. if you had generated these masks 2 years they would have been ok , but not as good as now. i am way off the goal of this competition. i am trying to improve SOA, not get close to it. sorry, if this sounds arrogant, but i think this is a very interesting set of raw data, especially the stage 2 test data.
\n\n(this should generate some negative votes.. glad i am not running for president :))
", "votes": null, "replies": [] } ] }, { "id": 316295, "author_name": "hengck23", "author_url": "", "post_date": "04/18/2018 16:44:27", "content": "@ Bcottman
\n\nI think what @topcoder did is correct. Understanding the metric is important and i think it is ok to post process your results to increase the metric score. The objective of kaggle is to find the upper limit.
\n\nIf there is problem with the rule or metric , we can fix it later. Before the rule and metric are fixed, we should stick to it (even if there may be problem). The metric of this competition is mAP over IOU 0.5 to 1.0 and we should try all methods to \"exploit it\" to find the upper limit, and then understand why we had the score. After which, we just need to be aware of the metric problem,etc and fix it.
\n\nJust to name a few example:
\n\nin caltech pedestrian detection in computer vision, the pedestrian box were re-annotated to accommodate detection of occluded pedestrian
pascal VOC 2007 mAP metric computation is improved in VOC 2012
Evaluation itself is a difficult process. We can only improve it as we get along. Metrics/labels can not be 100% perfect and of course, one should always be careful when interpreting ML results.
\n\nFurther, the fact that @topcoder solution has good results on both stage1 test and stage2 test show that their solution works! They have very implementation techniques and very high observation skills.
\n\nML solution is not only about algorithm design, it is also about \"dealing with data\" (e.g. pre, post processing, metric understanding). @topcoder certainly \"deal with data very well\"
\n\n
ok... yes, ..
\n\nas i have done before and all do again i wrote an email in awe/insight/shock before realizing my \"sharing\"might be perceived as insulting to others.
\n\nyou see i have been fooling around with this problem (identifying cells) since 1980. i started out working with an assoc. professor wiring together the backplanes of modified pdp-11s we got them unto 64kb . that is kilobits, when the std high end was 32 kb). any way i dropped out because i had this pesky thing called class load. i heard they were able to distinguish between 4 types of white cells. anyway i (erroronosly) thought computers were to slow and went onto other things.... enough about the good ole days.
\n\nStarting around January, i got someone i know, to send me 40 or so florescent cell images. being an old-timer, it did not occur to me to just search the web.... as i was delighted (as well as the person whose arm i twisted) to find out i could download off the web. from reading discussion threads here.
\n\nyou see i was really really delighted to see 700 some images made available by Broadhead and more importantly the current home of Cellfinder. When Dr. Carpenter refers to hard work to provide the images, i estimate 2 hours for mask(s) for an image with 20 cells with tools. also the blurry images you saw in stage 2 test set are typical not exceptional. (for me anyway.)
\n\ni imaged the Nobel prize (ok maybe a little exaggerated) when i saw Allen's idea of augmented/ensemble/averaging masks, because it is such a great way to remove tool bias.
\n\nAs i was reading up in this area, imagine my excitement when i saw a kaggle completion with a stated goal..\n Teams will create a computer model that can identify a range of nuclei across varied conditions. By observing patterns, asking questions, and building a model, participants will have a chance to push state-of-the-art technology farther.
\n\nso all you state above i think it is correct (well i don't agree with your figure at the end, and before anybody asks no i won't expand). i will also add that i missed the goal of this completion. which having said that, i must also state, i am not missing my goal. (yet)
\n\nAnd i will also add the top 100 or so in this contest did this in 3-months (or less) of work (on free-ware), which a group of 5 of us could not even get close to 38 years ago with a $200,000 grant. The tools today are sci-if. i look forward to seeing advances in the next 20 years.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "315842": "I was the person that belabored the point that after some AP threshold (say 0.5) performance was substantially less important than bias. Nevertheless, I’m curious.\n\nWhat is the best possible AP (without substantial architectural changes, supplementary data, or infinite training time, etc.)? 0.7? Higher? 🤔", "315868": "This contest had a massive data leak, imho. Congratulations to 1st place for exploiting this. I learned this lesson my last year as an undergraduate, there will be additional information (bias?) Introduced in data that is generated from raw data. I forgot this lesson, until i read the 1st place solution. \n\nagain Congratulations to 1st place winners. very clever.\n\nS.B. Allen, When you do this\n\n I also supplemented the training set in an attempt to standardize annotation disagreements:\n\n I computed a distance transform on each of the masks (not the flattened label matrix) and found the maximum peak. I computed a watershed segmentation for each masked image (that was dilated with a (1, 1) structuring element) using the peaks as markers. I added this new set of masks to the existing set.\n\nI took the centroid of each mask and used the random walker algorithm to compute a third set of masks using the aforementioned masked images. I then used the same watershed process from before. \n\nFinally, I created a fourth and fifth set of annotations by using a geodesic active contour with two different edge preserved responses (from a median filter and an inverse Gaussian filter). I then used the same watershed process from before.\n\nyou are getting rid of most (maybe) of this bias in the \"ground truth\"\n\nI will continue to work on this problem, at first probably retracing your first steps\n- vanilla U-net (done) \n- vanilla matterhorn f-MSK (doing)\n- vanilla broadhead f-msk (tote done)\n- vanilla slide cam (blackbox)\n- research results in the future", "315876": "could you please be more specific about the leak? what do you mean?", "315885": "Did you at least read the solution of the top-1 command? https://www.kaggle.com/c/data-science-bowl-2018/discussion/54741", "315891": "Seems that it's not necessary to use mass of specific techniques and approaches in order to be in top, isn't it? Moreover it's so, if by using them you still can not achieve appropriate results. \nAnd well, before blaming someone who solved the problem which you didn't(read as \"smarter\"), it's better to find conclusive proofs", "315895": "yes. sorry. perhaps data leak is too strong or wrong a term. the goal was too maximize score of IOU of **masks** generated by humans/tool not to generate masks of cells. While you might do better at generating cell masks with your tool/model, the LB score would not necessarily indicate so. \n\nto be clear, ( i am up too -5 votes and counting), i think this has been one of best kaggle contests ever, in terms of learning. This is SOA data from Broadhead.!!!! \n\nand i thought what 1st place did was very cool. to imply they were exploiting a data leak was wrong as it was an assumption on my part. it would be simpler to assume they trying to get a maximal score on some internal metric they might have derived.\n\nI was just making a comment, that was not discussed, except by Heng, who labeled it gray/grey box, that the goal to maximizing IOU to a tool not the raw data. to go further, perhaps an ensemble generated of masks generated using Allen's methods would have generated masks without so much bias to the tool that generated the masks.", "315897": "yes.", "315898": "rereading my post. i now realize how insulting it was. i actually do think that 1st place solution was very clever and i learned a lot from it.", "315901": "i agree with sentence #1, disagree with #2 and agree with #3 (in that i agree that ones should state facts not assumptions)", "315904": "Well, a week before stage 1 end we had an experiment on public LB, same masks.\n\n1. without FP removal 576\n\n2. with FP removal 577\n\nNot a huge gain :) But as you see even without this \"data leak\" it was top-1 on public as well (even one week before stage1 end). \n\nDo you think this postprocessing gave us a jump, let's say from 0.500 to 0.631 on stage 2?", "315909": "good experiment. i don't know. probably not. i am middle coding to try score on pre-lupkin, to try something like this.\n\nnote on AM NOT stating that g.t. is not ok. i am stating that you can not get a better score than gt masks. if you had generated these masks 2 years they would have been *ok* , but not as good as now. i am way off the goal of this competition. i am trying to improve SOA, not get close to it. sorry, if this sounds arrogant, but i think this is a very interesting set of raw data, especially the stage 2 test data.\n\n(this should generate some negative votes.. glad i am not running for president :))", "316295": "Bcottman\n\nI think what @topcoder did is correct. Understanding the metric is important and i think it is ok to post process your results to increase the metric score. The objective of kaggle is to find the upper limit.\n\nIf there is problem with the rule or metric , we can fix it later. Before the rule and metric are fixed, we should stick to it (even if there may be problem). The metric of this competition is mAP over IOU 0.5 to 1.0 and we should try all methods to \"exploit it\" to find the upper limit, and then understand why we had the score. After which, we just need to be aware of the metric problem,etc and fix it.\n\nJust to name a few example:\n\n - in caltech pedestrian detection in computer vision, the pedestrian box were re-annotated to accommodate detection of occluded pedestrian \n\n - pascal VOC 2007 mAP metric computation is improved in VOC 2012\n\n\nEvaluation itself is a difficult process. We can only improve it as we get along. Metrics/labels can not be 100% perfect and of course, one should always be careful when interpreting ML results.\n\n----\n\nFurther, the fact that @topcoder solution has good results on both stage1 test and stage2 test show that their solution works! They have very implementation techniques and very high observation skills.\n\nML solution is not only about algorithm design, it is also about \"dealing with data\" (e.g. pre, post processing, metric understanding). @topcoder certainly \"deal with data very well\"\n\n\n\n\n ![enter image description here][1]\n\n\n [1]: https://kaggle2.blob.core.windows.net/forum-message-attachments/316295/9213/limit.png", "316344": "ok... yes, ..\n\n as i have done before and all do again i wrote an email in awe/insight/shock before realizing my \"sharing\"might be perceived as insulting to others. \n\nyou see i have been fooling around with this problem (identifying cells) since 1980. i started out working with an assoc. professor wiring together the backplanes of modified pdp-11s we got them unto 64kb . that is kilobits, when the std high end was 32 kb). any way i dropped out because i had this pesky thing called class load. i heard they were able to distinguish between 4 types of white cells. anyway i (erroronosly) thought computers were to slow and went onto other things.... enough about the good ole days.\n\nStarting around January, i got someone i know, to send me 40 or so florescent cell images. being an old-timer, it did not occur to me to just search the web.... as i was delighted (as well as the person whose arm i twisted) to find out i could download off the web. from reading discussion threads here.\n\nyou see i was really really delighted to see 700 some images made available by Broadhead and more importantly the current home of Cellfinder. When Dr. Carpenter refers to hard work to provide the images, i estimate 2 hours for mask(s) for an image with 20 cells with tools. also the blurry images you saw in stage 2 test set are typical not exceptional. (for me anyway.)\n\ni imaged the Nobel prize (ok maybe a little exaggerated) when i saw Allen's idea of augmented/ensemble/averaging masks, because it is such a great way to remove tool bias.\n\nAs i was reading up in this area, imagine my excitement when i saw a kaggle completion with a stated goal..\n *Teams will create a computer model that can identify a range of nuclei across varied conditions. By observing patterns, asking questions, and building a model, participants will have a chance to push state-of-the-art technology farther.*\n\nso all you state above i think it is correct (well i don't agree with your figure at the end, and before anybody asks no i won't expand). i will also add that i missed the goal of this completion. which having said that, i must also state, i am not missing my goal. (yet)\n\nAnd i will also add the top 100 or so in this contest did this in 3-months (or less) of work (on free-ware), which a group of 5 of us could not even get close to 38 years ago with a $200,000 grant. The tools today are sci-if. i look forward to seeing advances in the next 20 years." }, "source": "meta" }