I have looked at two kernels:\nOne from William Cukierski: https://www.kaggle.com/wcukierski/example-metric-implementation\nOne from Glen Slade : https://www.kaggle.com/glenslade/alternative-metrics-kernel/comments
\n\nThey both have some issue, they give different results, and I am not sure which one is correct, while I'm inclined for the former.
\n\nWilliam Cukierski's code seems closer to the description of the evaluation page. The only issue is that the number of true objects is wrong as computed, as it includes the background. Same for the number of predicted objects.
\n\nGlen's code gives different results in my CV setting, I have not investigated precisely why, but I find suspicious that it computes IoU for a single match between each predicted object and the true objects. The test example is such that there is indeed a single possible match for each prediction, hence the issue does not show. Last, Glen's code is way slower.
", "messages": [ { "id": "287065", "postDate": "02/23/2018 12:18:38", "content": "I have looked at two kernels:\nOne from William Cukierski: https://www.kaggle.com/wcukierski/example-metric-implementation\nOne from Glen Slade : https://www.kaggle.com/glenslade/alternative-metrics-kernel/comments
\n\nThey both have some issue, they give different results, and I am not sure which one is correct, while I'm inclined for the former.
\n\nWilliam Cukierski's code seems closer to the description of the evaluation page. The only issue is that the number of true objects is wrong as computed, as it includes the background. Same for the number of predicted objects.
\n\nGlen's code gives different results in my CV setting, I have not investigated precisely why, but I find suspicious that it computes IoU for a single match between each predicted object and the true objects. The test example is such that there is indeed a single possible match for each prediction, hence the issue does not show. Last, Glen's code is way slower.
", "rawMarkdown": "I have looked at two kernels:\nOne from William Cukierski: https://www.kaggle.com/wcukierski/example-metric-implementation\nOne from Glen Slade : https://www.kaggle.com/glenslade/alternative-metrics-kernel/comments\n\nThey both have some issue, they give different results, and I am not sure which one is correct, while I'm inclined for the former.\n\nWilliam Cukierski's code seems closer to the description of the evaluation page. The only issue is that the number of true objects is wrong as computed, as it includes the background. Same for the number of predicted objects.\n\nGlen's code gives different results in my CV setting, I have not investigated precisely why, but I find suspicious that it computes IoU for a single match between each predicted object and the true objects. The test example is such that there is indeed a single possible match for each prediction, hence the issue does not show. Last, Glen's code is way slower.", "votes": null }, { "id": "287142", "postDate": "02/23/2018 17:00:53", "content": "The metric thresholds for IoU start at 0.5, so I believe that indeed only a single match is possible between each predicted mask and any of ground truth masks. I mean, you can have several matches with IoU < 0.5 but they would not make any contribution to precision.
", "rawMarkdown": "The metric thresholds for IoU start at 0.5, so I believe that indeed only a single match is possible between each predicted mask and any of ground truth masks. I mean, you can have several matches with IoU < 0.5 but they would not make any contribution to precision.", "votes": null }, { "id": "287144", "postDate": "02/23/2018 17:21:15", "content": "Hi CPMP\nThanks - you have drawn my attention to an error in the example table that arose after I edited a formula without checking all the consequences. I have corrected the print statement and it now matches Cukierski again (notwithstanding the background item in the count).\nPart of the reason I wrote this version was that I found the histogram method opaque for boundary cases, including what happens with multiple overlaps. The evaluation text from the organizers says the following:\n\"A true positive is counted when a single predicted object matches a ground truth object with an IoU above the threshold. A false positive indicates a predicted object had no associated ground truth object. A false negative indicates a ground truth object had no associated predicted object. The average precision of a single image is then calculated as the mean of the above precision values at each IoU threshold.\"\nI take this to mean that the driver of the algorithm is the predictions, not the ground truths. I am unclear whether the histogram approach will always work in this sense. \nCheers\nGlen
", "rawMarkdown": "Hi CPMP\nThanks - you have drawn my attention to an error in the example table that arose after I edited a formula without checking all the consequences. I have corrected the print statement and it now matches Cukierski again (notwithstanding the background item in the count).\nPart of the reason I wrote this version was that I found the histogram method opaque for boundary cases, including what happens with multiple overlaps. The evaluation text from the organizers says the following:\n*\"A true positive is counted when a single predicted object matches a ground truth object with an IoU above the threshold. A false positive indicates a predicted object had no associated ground truth object. A false negative indicates a ground truth object had no associated predicted object. The average precision of a single image is then calculated as the mean of the above precision values at each IoU threshold.\"*\nI take this to mean that the driver of the algorithm is the predictions, not the ground truths. I am unclear whether the histogram approach will always work in this sense. \nCheers\nGlen", "votes": null }, { "id": "287172", "postDate": "02/23/2018 19:18:12", "content": "I started like you, and thought Glen's code was the correct one. Now I'm confused.
", "rawMarkdown": "I started like you, and thought Glen's code was the correct one. Now I'm confused.", "votes": null }, { "id": "287175", "postDate": "02/23/2018 19:20:48", "content": "As said, I am not sure William code is correct, and yours may be the correct one. I have a doubt however, because of the following The metric is symmetric, i.e. it should be the same if you swap predictions and ground truth. Your code is not symmetric that way.
", "rawMarkdown": "As said, I am not sure William code is correct, and yours may be the correct one. I have a doubt however, because of the following The metric is symmetric, i.e. it should be the same if you swap predictions and ground truth. Your code is not symmetric that way.", "votes": null }, { "id": "287188", "postDate": "02/23/2018 20:10:10", "content": "It seems to me that the only place where Glen's code is not symmetric is the nested loops in IoU calculation. I also believe that if we swap the loops over predicted and true masks, we'll still get the same result, as the TP pairs will be the same. The only possible exception I see would be when a predicted mask exactly covers two true masks of the same size, yielding two IoU of exactly 0.5. But even in this case the precision will be the same.\nDo I miss something? Do you have an example of masks that gives different results?
", "rawMarkdown": "It seems to me that the only place where Glen's code is not symmetric is the nested loops in IoU calculation. I also believe that if we swap the loops over predicted and true masks, we'll still get the same result, as the TP pairs will be the same. The only possible exception I see would be when a predicted mask exactly covers two true masks of the same size, yielding two IoU of exactly 0.5. But even in this case the precision will be the same.\nDo I miss something? Do you have an example of masks that gives different results?", "votes": null }, { "id": "287201", "postDate": "02/23/2018 21:14:32", "content": "they give different results in my CV setting. I'll dig into it tomorrow and share what I found here.
", "rawMarkdown": "they give different results in my CV setting. I'll dig into it tomorrow and share what I found here.", "votes": null }, { "id": "287979", "postDate": "02/26/2018 11:35:28", "content": "Well, I think I'll focus on improving my submission quality (it really needs improvement ;) ) rather than debugging this.
\nFor now I use William's code for IoU computation.
The metric thresholds for IoU start at 0.5, so I believe that indeed only a single match is possible between each predicted mask and any of ground truth masks. I mean, you can have several matches with IoU < 0.5 but they would not make any contribution to precision.
", "votes": null, "replies": [ { "id": 287172, "author_name": "cpmpml", "author_url": "", "post_date": "02/23/2018 19:18:12", "content": "I started like you, and thought Glen's code was the correct one. Now I'm confused.
", "votes": null, "replies": [] } ] }, { "id": 287144, "author_name": "glenslade", "author_url": "", "post_date": "02/23/2018 17:21:15", "content": "Hi CPMP\nThanks - you have drawn my attention to an error in the example table that arose after I edited a formula without checking all the consequences. I have corrected the print statement and it now matches Cukierski again (notwithstanding the background item in the count).\nPart of the reason I wrote this version was that I found the histogram method opaque for boundary cases, including what happens with multiple overlaps. The evaluation text from the organizers says the following:\n\"A true positive is counted when a single predicted object matches a ground truth object with an IoU above the threshold. A false positive indicates a predicted object had no associated ground truth object. A false negative indicates a ground truth object had no associated predicted object. The average precision of a single image is then calculated as the mean of the above precision values at each IoU threshold.\"\nI take this to mean that the driver of the algorithm is the predictions, not the ground truths. I am unclear whether the histogram approach will always work in this sense. \nCheers\nGlen
", "votes": null, "replies": [ { "id": 287175, "author_name": "cpmpml", "author_url": "", "post_date": "02/23/2018 19:20:48", "content": "As said, I am not sure William code is correct, and yours may be the correct one. I have a doubt however, because of the following The metric is symmetric, i.e. it should be the same if you swap predictions and ground truth. Your code is not symmetric that way.
", "votes": null, "replies": [] }, { "id": 287188, "author_name": "alexz0", "author_url": "", "post_date": "02/23/2018 20:10:10", "content": "It seems to me that the only place where Glen's code is not symmetric is the nested loops in IoU calculation. I also believe that if we swap the loops over predicted and true masks, we'll still get the same result, as the TP pairs will be the same. The only possible exception I see would be when a predicted mask exactly covers two true masks of the same size, yielding two IoU of exactly 0.5. But even in this case the precision will be the same.\nDo I miss something? Do you have an example of masks that gives different results?
", "votes": null, "replies": [] }, { "id": 287201, "author_name": "cpmpml", "author_url": "", "post_date": "02/23/2018 21:14:32", "content": "they give different results in my CV setting. I'll dig into it tomorrow and share what I found here.
", "votes": null, "replies": [] }, { "id": 287979, "author_name": "cpmpml", "author_url": "", "post_date": "02/26/2018 11:35:28", "content": "Well, I think I'll focus on improving my submission quality (it really needs improvement ;) ) rather than debugging this.
\nFor now I use William's code for IoU computation.