After inference we are left with a set of bounding box predictions and associated confidence scores, ie confidence that the bounding box contains a wheat head and not background.
\n\nWe should filter the bounding boxes such that the confidence score is about a threshold. I've seen most notebooks use 0.5 as that threshold. I'm wondering if anyone else has done some testing to see what the impact of varying that threshold is. It might matter, depending on the number of available background vs foregrounds samples in your setup.
", "messages": [ { "id": "842115", "postDate": "05/11/2020 07:48:12", "content": "After inference we are left with a set of bounding box predictions and associated confidence scores, ie confidence that the bounding box contains a wheat head and not background.
\n\nWe should filter the bounding boxes such that the confidence score is about a threshold. I've seen most notebooks use 0.5 as that threshold. I'm wondering if anyone else has done some testing to see what the impact of varying that threshold is. It might matter, depending on the number of available background vs foregrounds samples in your setup.
", "rawMarkdown": "After inference we are left with a set of bounding box predictions and associated confidence scores, ie confidence that the bounding box contains a wheat head and not background.\n\nWe should filter the bounding boxes such that the confidence score is about a threshold. I've seen most notebooks use 0.5 as that threshold. I'm wondering if anyone else has done some testing to see what the impact of varying that threshold is. It might matter, depending on the number of available background vs foregrounds samples in your setup.", "votes": null }, { "id": "842395", "postDate": "05/11/2020 11:20:14", "content": "@alexandersoare Yes, I've tried that. Initially, my threshold was 0.50 and I got 0.6420 on public LB (I've trained a simplest FasterRCNN without any augmentations). Then for the same model, I changed the threshold to 0.40 and I got 0.6426 (this implies that being less strict increases the performance in my case). I also tried with threshold 0.30 on the same model and got 0.6407 (so very loose threshold decreases the performance). So in my case, the optimum threshold lies between somewhere 0.4 and 0.5 (my assumption).
\n\nNow I've also tried with threshold 0.60 and got 0.6366.
", "rawMarkdown": "alexandersoare Yes, I've tried that. Initially, my threshold was **0.50** and I got **0.6420** on public LB (I've trained a simplest FasterRCNN without any augmentations). Then for the same model, I changed the threshold to **0.40** and I got **0.6426** (this implies that being less strict increases the performance in my case). I also tried with threshold **0.30** on the same model and got **0.6407** (so very loose threshold decreases the performance). So in my case, the optimum threshold lies between somewhere 0.4 and 0.5 (my assumption).\n\nNow I've also tried with threshold 0.60 and got 0.6366.", "votes": null }, { "id": "842399", "postDate": "05/11/2020 11:23:49", "content": "Awesome, thank you for sharing!
", "rawMarkdown": "Awesome, thank you for sharing!", "votes": null }, { "id": "842864", "postDate": "05/11/2020 17:07:39", "content": "I played couple of times with those and never got any decent result past the .5 .. however I believe it can be explored via ML-ish approach (an extra output of your network can be that threshold and you simply train the network as you would (the image properties come inference will determine the threshold based on training data). It's on my list of things to try while I wait for my GPU quota to reset 🤕
", "rawMarkdown": "I played couple of times with those and never got any decent result past the .5 .. however I believe it can be explored via ML-ish approach (an extra output of your network can be that threshold and you simply train the network as you would (the image properties come inference will determine the threshold based on training data). It's on my list of things to try while I wait for my GPU quota to reset 🤕", "votes": null } ], "comments": [ { "id": 842395, "author_name": "kaushal2896", "author_url": "", "post_date": "05/11/2020 11:20:14", "content": "@alexandersoare Yes, I've tried that. Initially, my threshold was 0.50 and I got 0.6420 on public LB (I've trained a simplest FasterRCNN without any augmentations). Then for the same model, I changed the threshold to 0.40 and I got 0.6426 (this implies that being less strict increases the performance in my case). I also tried with threshold 0.30 on the same model and got 0.6407 (so very loose threshold decreases the performance). So in my case, the optimum threshold lies between somewhere 0.4 and 0.5 (my assumption).
\n\nNow I've also tried with threshold 0.60 and got 0.6366.
", "votes": null, "replies": [ { "id": 842399, "author_name": "alexandersoare", "author_url": "", "post_date": "05/11/2020 11:23:49", "content": "Awesome, thank you for sharing!
", "votes": null, "replies": [] } ] }, { "id": 842864, "author_name": "ma7moud", "author_url": "", "post_date": "05/11/2020 17:07:39", "content": "I played couple of times with those and never got any decent result past the .5 .. however I believe it can be explored via ML-ish approach (an extra output of your network can be that threshold and you simply train the network as you would (the image properties come inference will determine the threshold based on training data). It's on my list of things to try while I wait for my GPU quota to reset 🤕
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "842115": "After inference we are left with a set of bounding box predictions and associated confidence scores, ie confidence that the bounding box contains a wheat head and not background.\n\nWe should filter the bounding boxes such that the confidence score is about a threshold. I've seen most notebooks use 0.5 as that threshold. I'm wondering if anyone else has done some testing to see what the impact of varying that threshold is. It might matter, depending on the number of available background vs foregrounds samples in your setup.", "842395": "alexandersoare Yes, I've tried that. Initially, my threshold was **0.50** and I got **0.6420** on public LB (I've trained a simplest FasterRCNN without any augmentations). Then for the same model, I changed the threshold to **0.40** and I got **0.6426** (this implies that being less strict increases the performance in my case). I also tried with threshold **0.30** on the same model and got **0.6407** (so very loose threshold decreases the performance). So in my case, the optimum threshold lies between somewhere 0.4 and 0.5 (my assumption).\n\nNow I've also tried with threshold 0.60 and got 0.6366.", "842399": "Awesome, thank you for sharing!", "842864": "I played couple of times with those and never got any decent result past the .5 .. however I believe it can be explored via ML-ish approach (an extra output of your network can be that threshold and you simply train the network as you would (the image properties come inference will determine the threshold based on training data). It's on my list of things to try while I wait for my GPU quota to reset 🤕" }, "source": "meta" }