I'm new to object detection, still trying to get my head around how various detectors work in general. I was reading the metric calculation kernel by Peter.
\n\n1) I'm confused about the find_best_match function, if I understood correctly, it finds the prediction that matches the ground truth bbox the most and use that as the prediction for mAP calculation, but in test time, we don't have the ground truth, how can we output the \"best_matched\" as our prediction? If we do the post-processing NMS, it's not guaranteed to produce the same bbox as used in that metric calculation function.
2) A general question regarding NMS, is NMS guaranteed to give at most one bbox prediction per ground truth?
\n\n3) let's say we don't apply NMS whatsoever, so the output predicted bboxes are not filtered at all, how does the metric calculation look like in that case? There are likely multiple bbox predicted for a same ground truth bbox, I imagine if one of the predicted bboxes fall below the iou threshold, it then would get counted as false negative (which makes sense), but if two or more similar/overlapping predictions are made, then are they all getting counted as true positive? (it cannot be as it would just naively inflate the metric), or does it work like following, e.g. if you have multiple overlapping predicted bbox, after your highest scored prediction is counted as true positive then that ground truth is removed so the rest two redundant predictions are then counted as false negative? I'm not sure if this makes much sense, again I don't have a good understanding on this, if someone could help that would be great.
", "messages": [ { "id": "922277", "postDate": "07/10/2020 01:43:57", "content": "I'm new to object detection, still trying to get my head around how various detectors work in general. I was reading the metric calculation kernel by Peter.
\n\n1) I'm confused about the find_best_match function, if I understood correctly, it finds the prediction that matches the ground truth bbox the most and use that as the prediction for mAP calculation, but in test time, we don't have the ground truth, how can we output the \"best_matched\" as our prediction? If we do the post-processing NMS, it's not guaranteed to produce the same bbox as used in that metric calculation function.
2) A general question regarding NMS, is NMS guaranteed to give at most one bbox prediction per ground truth?
\n\n3) let's say we don't apply NMS whatsoever, so the output predicted bboxes are not filtered at all, how does the metric calculation look like in that case? There are likely multiple bbox predicted for a same ground truth bbox, I imagine if one of the predicted bboxes fall below the iou threshold, it then would get counted as false negative (which makes sense), but if two or more similar/overlapping predictions are made, then are they all getting counted as true positive? (it cannot be as it would just naively inflate the metric), or does it work like following, e.g. if you have multiple overlapping predicted bbox, after your highest scored prediction is counted as true positive then that ground truth is removed so the rest two redundant predictions are then counted as false negative? I'm not sure if this makes much sense, again I don't have a good understanding on this, if someone could help that would be great.
", "rawMarkdown": "I'm new to object detection, still trying to get my head around how various detectors work in general. I was reading the metric calculation [kernel](https://www.kaggle.com/pestipeti/competition-metric-details-script/notebook) by Peter. \n\n1) I'm confused about the `find_best_match` function, if I understood correctly, it finds the prediction that matches the ground truth bbox the most and use that as the prediction for mAP calculation, but in test time, we don't have the ground truth, how can we output the \"best\\_matched\" as our prediction? If we do the post-processing NMS, it's not guaranteed to produce the same bbox as used in that metric calculation function.\n\n2) A general question regarding NMS, is NMS guaranteed to give at most one bbox prediction per ground truth?\n\n3) let's say we don't apply NMS whatsoever, so the output predicted bboxes are not filtered at all, how does the metric calculation look like in that case? There are likely multiple bbox predicted for a same ground truth bbox, I imagine if one of the predicted bboxes fall below the iou threshold, it then would get counted as false negative (which makes sense), but if two or more similar/overlapping predictions are made, then are they all getting counted as true positive? (it cannot be as it would just naively inflate the metric), or does it work like following, e.g. if you have multiple overlapping predicted bbox, after your highest scored prediction is counted as true positive then that ground truth is removed so the rest two redundant predictions are then counted as false negative? I'm not sure if this makes much sense, again I don't have a good understanding on this, if someone could help that would be great.", "votes": null }, { "id": "922585", "postDate": "07/10/2020 08:07:51", "content": "Let me try and help you with some of your questions \nAbout your question about the test data - This is always the case in Kaggle. We as a competitor don't have the ground truth, but in Kaggle, they have it, and they take your model and compute the metric based on your model and the hidden ground truth. Your task is to build the best model based on train data. You need to use the metric.\nRegrading not using an algorithm to reduce the number of bbox . In general, any optimization algorithm like NMS or WBF aims to optimize the result.\nThe target is that it will only have the areas that match the best, which will eventually increase the number of True positive and reduce the number of false's which eventually is used to calculate the precision. \nI don't think it how many bbox the algorithm find, what mater is the overlap or the mean maP
", "rawMarkdown": "Let me try and help you with some of your questions \nAbout your question about the test data - This is always the case in Kaggle. We as a competitor don't have the ground truth, but in Kaggle, they have it, and they take your model and compute the metric based on your model and the hidden ground truth. Your task is to build the best model based on train data. You need to use the metric.\nRegrading not using an algorithm to reduce the number of bbox . In general, any optimization algorithm like NMS or WBF aims to optimize the result.\nThe target is that it will only have the areas that match the best, which will eventually increase the number of True positive and reduce the number of false's which eventually is used to calculate the precision. \nI don't think it how many bbox the algorithm find, what mater is the overlap or the mean maP", "votes": null }, { "id": "923187", "postDate": "07/10/2020 15:41:24", "content": "Thanks for the reply. \n1) I understood that we do not have true label in test time surely, and yes we need to compute the metric to evaluate our model. I'm not sure maybe the way I described it is not that clear, my question was, the metric calculation in Peter's kernel, (again, if I understood correctly) it relies on ground truth to find the best matching and then use that best matched to make mAP calculation, but in test time (either apply NMS or not) I think the predictions are not guaranteed to be evaluated same as in this kernel calculation, e.g. the best matched one could have lower confidence, and then in test time evaluation starts with highest confidence, and a TP might get created without reaching the \"best matched\" one, in other words, the kernels metric calculation is inflated, or better than what it shows\n2) I understood NMS or its variant and WBF aims to optimize the result, but my question was \" is NMS guaranteed to give at most one bbox prediction per ground truth?\"\n3) I think however many bbox the algo finds actually matters, as intuitively, it is only useful when you can give a number of bbox in a task, it doesnt help to just provide bunch bbox and let the user or final objective to decide which one is useful. So I think intuitively, once a TP is made, the ground truth is probably removed and all possible subsequent overlapping predictions provided will be counted as FP.
\n\nFor now I will try to dig into that kernel metric calculation more, maybe Im missing something obvious as I don't see anyone asking these questions lol
", "rawMarkdown": "Thanks for the reply. \n1) I understood that we do not have true label in test time surely, and yes we need to compute the metric to evaluate our model. I'm not sure maybe the way I described it is not that clear, my question was, the metric calculation in Peter's kernel, (again, if I understood correctly) it relies on ground truth to find the best matching and then use that best matched to make mAP calculation, but in test time (either apply NMS or not) I think the predictions are not guaranteed to be evaluated same as in this kernel calculation, e.g. the best matched one could have lower confidence, and then in test time evaluation starts with highest confidence, and a TP might get created without reaching the \"best matched\" one, in other words, the kernels metric calculation is inflated, or better than what it shows\n2) I understood NMS or its variant and WBF aims to optimize the result, but my question was \" is NMS guaranteed to give at most one bbox prediction per ground truth?\"\n3) I think however many bbox the algo finds actually matters, as intuitively, it is only useful when you can give a number of bbox in a task, it doesnt help to just provide bunch bbox and let the user or final objective to decide which one is useful. So I think intuitively, once a TP is made, the ground truth is probably removed and all possible subsequent overlapping predictions provided will be counted as FP.\n\nFor now I will try to dig into that kernel metric calculation more, maybe Im missing something obvious as I don't see anyone asking these questions lol", "votes": null } ], "comments": [ { "id": 922585, "author_name": "omershect", "author_url": "", "post_date": "07/10/2020 08:07:51", "content": "Let me try and help you with some of your questions \nAbout your question about the test data - This is always the case in Kaggle. We as a competitor don't have the ground truth, but in Kaggle, they have it, and they take your model and compute the metric based on your model and the hidden ground truth. Your task is to build the best model based on train data. You need to use the metric.\nRegrading not using an algorithm to reduce the number of bbox . In general, any optimization algorithm like NMS or WBF aims to optimize the result.\nThe target is that it will only have the areas that match the best, which will eventually increase the number of True positive and reduce the number of false's which eventually is used to calculate the precision. \nI don't think it how many bbox the algorithm find, what mater is the overlap or the mean maP
", "votes": null, "replies": [ { "id": 923187, "author_name": "samshipengs", "author_url": "", "post_date": "07/10/2020 15:41:24", "content": "Thanks for the reply. \n1) I understood that we do not have true label in test time surely, and yes we need to compute the metric to evaluate our model. I'm not sure maybe the way I described it is not that clear, my question was, the metric calculation in Peter's kernel, (again, if I understood correctly) it relies on ground truth to find the best matching and then use that best matched to make mAP calculation, but in test time (either apply NMS or not) I think the predictions are not guaranteed to be evaluated same as in this kernel calculation, e.g. the best matched one could have lower confidence, and then in test time evaluation starts with highest confidence, and a TP might get created without reaching the \"best matched\" one, in other words, the kernels metric calculation is inflated, or better than what it shows\n2) I understood NMS or its variant and WBF aims to optimize the result, but my question was \" is NMS guaranteed to give at most one bbox prediction per ground truth?\"\n3) I think however many bbox the algo finds actually matters, as intuitively, it is only useful when you can give a number of bbox in a task, it doesnt help to just provide bunch bbox and let the user or final objective to decide which one is useful. So I think intuitively, once a TP is made, the ground truth is probably removed and all possible subsequent overlapping predictions provided will be counted as FP.
\n\nFor now I will try to dig into that kernel metric calculation more, maybe Im missing something obvious as I don't see anyone asking these questions lol
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "922277": "I'm new to object detection, still trying to get my head around how various detectors work in general. I was reading the metric calculation [kernel](https://www.kaggle.com/pestipeti/competition-metric-details-script/notebook) by Peter. \n\n1) I'm confused about the `find_best_match` function, if I understood correctly, it finds the prediction that matches the ground truth bbox the most and use that as the prediction for mAP calculation, but in test time, we don't have the ground truth, how can we output the \"best\\_matched\" as our prediction? If we do the post-processing NMS, it's not guaranteed to produce the same bbox as used in that metric calculation function.\n\n2) A general question regarding NMS, is NMS guaranteed to give at most one bbox prediction per ground truth?\n\n3) let's say we don't apply NMS whatsoever, so the output predicted bboxes are not filtered at all, how does the metric calculation look like in that case? There are likely multiple bbox predicted for a same ground truth bbox, I imagine if one of the predicted bboxes fall below the iou threshold, it then would get counted as false negative (which makes sense), but if two or more similar/overlapping predictions are made, then are they all getting counted as true positive? (it cannot be as it would just naively inflate the metric), or does it work like following, e.g. if you have multiple overlapping predicted bbox, after your highest scored prediction is counted as true positive then that ground truth is removed so the rest two redundant predictions are then counted as false negative? I'm not sure if this makes much sense, again I don't have a good understanding on this, if someone could help that would be great.", "922585": "Let me try and help you with some of your questions \nAbout your question about the test data - This is always the case in Kaggle. We as a competitor don't have the ground truth, but in Kaggle, they have it, and they take your model and compute the metric based on your model and the hidden ground truth. Your task is to build the best model based on train data. You need to use the metric.\nRegrading not using an algorithm to reduce the number of bbox . In general, any optimization algorithm like NMS or WBF aims to optimize the result.\nThe target is that it will only have the areas that match the best, which will eventually increase the number of True positive and reduce the number of false's which eventually is used to calculate the precision. \nI don't think it how many bbox the algorithm find, what mater is the overlap or the mean maP", "923187": "Thanks for the reply. \n1) I understood that we do not have true label in test time surely, and yes we need to compute the metric to evaluate our model. I'm not sure maybe the way I described it is not that clear, my question was, the metric calculation in Peter's kernel, (again, if I understood correctly) it relies on ground truth to find the best matching and then use that best matched to make mAP calculation, but in test time (either apply NMS or not) I think the predictions are not guaranteed to be evaluated same as in this kernel calculation, e.g. the best matched one could have lower confidence, and then in test time evaluation starts with highest confidence, and a TP might get created without reaching the \"best matched\" one, in other words, the kernels metric calculation is inflated, or better than what it shows\n2) I understood NMS or its variant and WBF aims to optimize the result, but my question was \" is NMS guaranteed to give at most one bbox prediction per ground truth?\"\n3) I think however many bbox the algo finds actually matters, as intuitively, it is only useful when you can give a number of bbox in a task, it doesnt help to just provide bunch bbox and let the user or final objective to decide which one is useful. So I think intuitively, once a TP is made, the ground truth is probably removed and all possible subsequent overlapping predictions provided will be counted as FP.\n\nFor now I will try to dig into that kernel metric calculation more, maybe Im missing something obvious as I don't see anyone asking these questions lol" }, "source": "meta" }