![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F864684%2Fcc16bbad8416c8992aecb2e012d7f2ea%2Fresults_1.jpg?generation=1589995713183962&alt=media\")
I finally overcame Submission CSV Not Found error, now I'm very happy to report my cv and lb score relationship! (Even though it's not so good😹)
\n\nMy single fold cv score (out of 5 folds) is 0.5882 and LB score is 0.5805. Looks not so bad, doesn't it?
\n\nHow about you?
", "messages": [ { "id": 838550, "postDate": "2020-05-08T16:32:33.883Z", "content": "I finally overcame Submission CSV Not Found error, now I'm very happy to report my cv and lb score relationship! (Even though it's not so good😹)
\n\nMy single fold cv score (out of 5 folds) is 0.5882 and LB score is 0.5805. Looks not so bad, doesn't it?
\n\nHow about you?
", "rawMarkdown": "I finally overcame Submission CSV Not Found error, now I'm very happy to report my cv and lb score relationship! (Even though it's not so good😹)\n\nMy single fold cv score (out of 5 folds) is 0.5882 and LB score is 0.5805. Looks not so bad, doesn't it?\n\nHow about you?", "votes": 22 }, { "id": 838598, "postDate": "2020-05-08T17:06:06.107Z", "content": "single fold, CV: 0.723, LB: 0.7037
", "rawMarkdown": "single fold, CV: 0.723, LB: 0.7037", "votes": 6, "replies": [ { "id": 839822, "postDate": "2020-05-09T16:02:44.013Z", "content": "are you using the MAP from https://www.kaggle.com/pestipeti/optimized-map-calculator ?
", "rawMarkdown": "are you using the MAP from https://www.kaggle.com/pestipeti/optimized-map-calculator ?" }, { "id": 839850, "postDate": "2020-05-09T16:15:29.040Z", "content": "No, I use original evaluation optimizer.
", "rawMarkdown": "No, I use original evaluation optimizer.", "votes": 1 }, { "id": 839855, "postDate": "2020-05-09T16:18:41.350Z", "content": "@zero00 I'm not 100% sure that my MAP is correct.\nmy result: single fold, CV: 0.6018; LB: 0.6826\nrandom 80:20 split
", "rawMarkdown": "@zero00 I'm not 100% sure that my MAP is correct.\nmy result: single fold, CV: 0.6018; LB: 0.6826\nrandom 80:20 split", "votes": 1 }, { "id": 855804, "postDate": "2020-05-21T08:27:24.837Z", "content": "Hey Peter! I'm using your metrics to measure my model, it's great. But sometimes the results aren't stable. Sometimes CV increase but LB decrease, sometimes CV and LB both increase. I'm using same seed to ramdom 80:20 split. Should I use StratifiedKFold to do split? It bothering me a long time.
", "rawMarkdown": "Hey Peter! I'm using your metrics to measure my model, it's great. But sometimes the results aren't stable. Sometimes CV increase but LB decrease, sometimes CV and LB both increase. I'm using same seed to ramdom 80:20 split. Should I use StratifiedKFold to do split? It bothering me a long time." }, { "id": 855849, "postDate": "2020-05-21T09:22:43.473Z", "content": "I think this is normal. There is a difference in \"detection difficulty\" between the sources. See my comment above (the one with the chart).
", "rawMarkdown": "I think this is normal. There is a difference in \"detection difficulty\" between the sources. See my comment above (the one with the chart). \n" }, { "id": 856158, "postDate": "2020-05-21T14:34:48.440Z", "content": "Got it! So it's depend on the data domain.
", "rawMarkdown": "Got it! So it's depend on the data domain." } ] }, { "id": 854427, "postDate": "2020-05-20T02:46:03.433Z", "content": "I use Peter's metrics, CV: 65.4, LB:68.02, randomly 8:2 split.
", "rawMarkdown": "I use Peter's metrics, CV: 65.4, LB:68.02, randomly 8:2 split.", "votes": 1 }, { "id": 848347, "postDate": "2020-05-14T22:21:26.583Z", "content": "So how to split data to measure cv？Just randomly split? Or take one source as valid, others as train?
", "rawMarkdown": "So how to split data to measure cv？Just randomly split? Or take one source as valid, others as train?", "votes": 1 }, { "id": 841720, "postDate": "2020-05-11T01:28:50.957Z", "content": "single fold, CV: 0.6519, LB: 0.6485\nValidation strategies - kRegionsOut
", "rawMarkdown": "single fold, CV: 0.6519, LB: 0.6485\nValidation strategies - [kRegionsOut](https://www.kaggle.com/c/global-wheat-detection/discussion/148696)", "votes": 1 }, { "id": 859638, "postDate": "2020-05-24T15:53:01.657Z", "content": "Single Model : 5 checkpoints CV: 0.7160 LB:0.7245
", "rawMarkdown": "Single Model : 5 checkpoints CV: 0.7160 LB:0.7245", "votes": 2 }, { "id": 851858, "postDate": "2020-05-18T01:08:28.120Z", "content": "What eval scripts are you guys using for metrics?
\n\nhttps://github.com/cocodataset/cocoapi/blob/master/PythonAPI/pycocotools/cocoeval.py\nI think a good standard is cocoeval, and we can teak lines 498- as \npython\nclass Params:\n '''\n Params for coco evaluation api\n '''\n def setDetParams(self):\n self.imgIds = []\n self.catIds = []\n # np.arange causes trouble. the data point on arange is slightly larger than the true value\n self.iouThrs = np.linspace(.5, 0.75, int(np.round((0.75 - .5) / .05)) + 1, endpoint=True)\n self.recThrs = np.linspace(.0, 1.00, int(np.round((1.00 - .0) / .01)) + 1, endpoint=True)\n self.maxDets = [1, 10, 100]\n self.areaRng = [[0 ** 2, 1e5 ** 2], [0 ** 2, 32 ** 2], [32 ** 2, 96 ** 2], [96 ** 2, 1e5 ** 2]]\n self.areaRngLbl = ['all', 'small', 'medium', 'large']\n self.useCats = 1\n
to realize the metrics as provided in this competition.
\n\nI also tried out Peter's eval scripts, but there seems to be divergence against the cocoeval results.
\n\nBTW with centernet single fold single model, I get cocoCV 0.69 LB 0.65.
", "rawMarkdown": "What eval scripts are you guys using for metrics?\n\nhttps://github.com/cocodataset/cocoapi/blob/master/PythonAPI/pycocotools/cocoeval.py\nI think a good standard is cocoeval, and we can teak lines 498- as \n```python\nclass Params:\n '''\n Params for coco evaluation api\n '''\n def setDetParams(self):\n self.imgIds = []\n self.catIds = []\n # np.arange causes trouble. the data point on arange is slightly larger than the true value\n self.iouThrs = np.linspace(.5, 0.75, int(np.round((0.75 - .5) / .05)) + 1, endpoint=True)\n self.recThrs = np.linspace(.0, 1.00, int(np.round((1.00 - .0) / .01)) + 1, endpoint=True)\n self.maxDets = [1, 10, 100]\n self.areaRng = [[0 ** 2, 1e5 ** 2], [0 ** 2, 32 ** 2], [32 ** 2, 96 ** 2], [96 ** 2, 1e5 ** 2]]\n self.areaRngLbl = ['all', 'small', 'medium', 'large']\n self.useCats = 1\n```\n\nto realize the metrics as provided in this competition.\n\nI also tried out Peter's eval scripts, but there seems to be divergence against the cocoeval results.\n\nBTW with centernet single fold single model, I get cocoCV 0.69 LB 0.65.", "votes": 2, "replies": [ { "id": 851992, "postDate": "2020-05-18T05:12:30.743Z", "content": "I can confirm your result, it seems there is a bug in my script. I'll try to debug and fix it as soon as I'll have a bit of time.
", "rawMarkdown": "I can confirm your result, it seems there is a bug in my script. I'll try to debug and fix it as soon as I'll have a bit of time.", "votes": 2 }, { "id": 852965, "postDate": "2020-05-18T20:58:07.707Z", "content": "@pestipeti I just did some experiments, for each epoch, I just printed out the valuation score using your script, vs using the cocoeval script, and here are the results:
\n\nyour_score, coco_score\n0.430711174576225, 0.6523691508406063\n0.4785398646805657, 0.6892521575561043\n0.48073431913247816, 0.6962216598700071\n0.49023349687333, 0.7288417268693561\n0.47705885089740174, 0.7230724922949511\n0.4660380495242629, 0.7376911721208542\n0.5314250379854468, 0.7414370430705847
\n\nSo it seems your script gives consistently much lower score than what coco eval gives, but good thing is both are roughly consistent when ups or downs, which means people can still use your script to select the best model / checkpoint, and submit to LB for verification.
\n\nFYI for people who are interested in how to use coco eval - you will need to \n1) create ground truth json as in coco format. \n2) dump your predictions into json format in local, \n3) run eval using coco APIs, following this example - https://github.com/cocodataset/cocoapi/blob/master/PythonAPI/pycocoEvalDemo.ipynb. Most importantly, set this param iou threshods: cocoEval.params.iouThrs = np.linspace(.5, 0.75, np.round((0.75 - .5) / .05) + 1, endpoint=True)
Thanks, @weimin!
", "rawMarkdown": "Thanks, @weimin! ", "votes": 1 }, { "id": 853930, "postDate": "2020-05-19T16:05:12.113Z", "content": "May I ask do we still need to change how average precision is calculated in cocoapi? Because in this competition, ap=tp/(tp+fp+fn), but in cocoapi, it seems ap = tp/(tp+fp). Correct me if I am wrong.
", "rawMarkdown": "May I ask do we still need to change how average precision is calculated in cocoapi? Because in this competition, ap=tp/(tp+fp+fn), but in cocoapi, it seems ap = tp/(tp+fp). Correct me if I am wrong.", "votes": 3 }, { "id": 854038, "postDate": "2020-05-19T17:37:55.937Z", "content": "It is interesting; @weimin, can you confirm that without FN, the result of my script equals COCOEval?
", "rawMarkdown": "It is interesting; @weimin, can you confirm that without FN, the result of my script equals COCOEval?" }, { "id": 854340, "postDate": "2020-05-20T01:09:52.973Z", "content": "This is what I got - \nScore using coco eval - 0.6741089713448498\nNew yours (tp / (tp + fp)) - 0.4924787830466831 - old yours (tp / (tp + fp + fn))- 0.43898056126435475\nAll I did for new is to replace - \n #return tp / (tp + fp + fn)\n return tp / (tp + fp)
\n\nfor your calculate_precision function.
", "rawMarkdown": "This is what I got - \nScore using coco eval - 0.6741089713448498\nNew yours (tp / (tp + fp)) - 0.4924787830466831 - old yours (tp / (tp + fp + fn))- 0.43898056126435475\nAll I did for new is to replace - \n #return tp / (tp + fp + fn)\n return tp / (tp + fp)\n\nfor your calculate_precision function. ", "votes": 3 }, { "id": 854412, "postDate": "2020-05-20T02:32:47.140Z", "content": "A basic question: it seems mAP in coco is related to precision-recall curve, which is not simple averge precision in different iou thresholds in this case. What do you think?
", "rawMarkdown": "A basic question: it seems mAP in coco is related to precision-recall curve, which is not simple averge precision in different iou thresholds in this case. What do you think? ", "votes": 2 }, { "id": 855156, "postDate": "2020-05-20T16:32:26.210Z", "content": "yeah, I agreed with what you said, it seems to me that peter's calculation should be more related to this competition metric than COCO's... it could be helpful if the host team can release some sort of code for the metric.
", "rawMarkdown": "yeah, I agreed with what you said, it seems to me that peter's calculation should be more related to this competition metric than COCO's... it could be helpful if the host team can release some sort of code for the metric. " }, { "id": 855233, "postDate": "2020-05-20T17:30:40.553Z", "content": "I agree. As @mrxuehb said, COCO is related to the precision-recall curve. Kaggle probably is not using COCOeval; they would have mentioned in the evaluation description.
\n\nI think the reason for the high cv-lb gap is not because of a bug. See the image below; I calculated the mAP per source. \n(The final mAP is 0.602 (The LB was 0.7102) for this experiment.)
\n\n
As you can see, one of the sources (rres_1) had a 0.7266 mAP and a more difficult one had 0.4588 (arvalis_2). I think this simply means that the samples in the public LB have a similar difficulty level that rres_1.
IMO, the private LB could be very different.
", "rawMarkdown": "I agree. As @mrxuehb said, COCO is related to the precision-recall curve. Kaggle probably is not using COCOeval; they would have mentioned in the evaluation description.\n\nI think the reason for the high cv-lb gap is not because of a bug. See the image below; I calculated the mAP per source. \n(The final mAP is 0.602 (The LB was 0.7102) for this experiment.)\n\n\n\n\nAs you can see, one of the sources (`rres_1`) had a 0.7266 mAP and a more difficult one had 0.4588 (`arvalis_2`). I think this simply means that the samples in the public LB have a similar difficulty level that `rres_1`.\n\nIMO, the private LB could be very different.\n\n\n", "votes": 12 }, { "id": 855254, "postDate": "2020-05-20T18:01:05.753Z", "content": "Good findings!
", "rawMarkdown": "Good findings! " }, { "id": 855388, "postDate": "2020-05-20T21:52:15.010Z", "content": "Thanks for this great discussion guys, really helpful!
", "rawMarkdown": "Thanks for this great discussion guys, really helpful!" }, { "id": 855890, "postDate": "2020-05-21T10:14:43.460Z", "content": "@pestipeti \nGreat findings! \nSounds like shake-down coming.. fingers crossed.
", "rawMarkdown": "@pestipeti \nGreat findings! \nSounds like shake-down coming.. fingers crossed." } ] }, { "id": 839357, "postDate": "2020-05-09T10:43:41.140Z", "content": "CV:?(no check....) LB: 0.6959\nrandom split 9 : 1, single fold only
", "rawMarkdown": "CV:?(no check....) LB: 0.6959\nrandom split 9 : 1, single fold only", "votes": 2 }, { "id": 877282, "postDate": "2020-06-07T13:11:03.413Z", "content": "Another data point: my single fold v score is 0.6884 and LB score is 0.7178 using Peter's metric.
", "rawMarkdown": "Another data point: my single fold v score is 0.6884 and LB score is 0.7178 using Peter's metric. " }, { "id": 851131, "postDate": "2020-05-17T11:23:26.603Z", "rawMarkdown": "", "votes": 2, "isDeleted": true } ], "comments": [ { "id": 838598, "author_name": "phalanx", "author_url": "", "post_date": "2020-05-08T17:06:06.107000", "content": "single fold, CV: 0.723, LB: 0.7037
", "votes": 6, "replies": [ { "id": 839822, "author_name": "Zero", "author_url": "", "post_date": "2020-05-09T16:02:44.013000", "content": "are you using the MAP from https://www.kaggle.com/pestipeti/optimized-map-calculator ?
", "votes": 0, "replies": [] }, { "id": 839850, "author_name": "phalanx", "author_url": "", "post_date": "2020-05-09T16:15:29.040000", "content": "No, I use original evaluation optimizer.
", "votes": 1, "replies": [] }, { "id": 839855, "author_name": "Peter", "author_url": "", "post_date": "2020-05-09T16:18:41.350000", "content": "@zero00 I'm not 100% sure that my MAP is correct.\nmy result: single fold, CV: 0.6018; LB: 0.6826\nrandom 80:20 split
", "votes": 1, "replies": [] }, { "id": 855804, "author_name": "27Wilson", "author_url": "", "post_date": "2020-05-21T08:27:24.837000", "content": "Hey Peter! I'm using your metrics to measure my model, it's great. But sometimes the results aren't stable. Sometimes CV increase but LB decrease, sometimes CV and LB both increase. I'm using same seed to ramdom 80:20 split. Should I use StratifiedKFold to do split? It bothering me a long time.
", "votes": 0, "replies": [] }, { "id": 855849, "author_name": "Peter", "author_url": "", "post_date": "2020-05-21T09:22:43.473000", "content": "I think this is normal. There is a difference in \"detection difficulty\" between the sources. See my comment above (the one with the chart).
", "votes": 0, "replies": [] }, { "id": 856158, "author_name": "27Wilson", "author_url": "", "post_date": "2020-05-21T14:34:48.440000", "content": "Got it! So it's depend on the data domain.
", "votes": 0, "replies": [] } ] }, { "id": 854427, "author_name": "27Wilson", "author_url": "", "post_date": "2020-05-20T02:46:03.433000", "content": "I use Peter's metrics, CV: 65.4, LB:68.02, randomly 8:2 split.
", "votes": 1, "replies": [] }, { "id": 848347, "author_name": "27Wilson", "author_url": "", "post_date": "2020-05-14T22:21:26.583000", "content": "So how to split data to measure cv？Just randomly split? Or take one source as valid, others as train?
", "votes": 1, "replies": [] }, { "id": 841720, "author_name": "Fayzur", "author_url": "", "post_date": "2020-05-11T01:28:50.957000", "content": "single fold, CV: 0.6519, LB: 0.6485\nValidation strategies - kRegionsOut
", "votes": 1, "replies": [] }, { "id": 859638, "author_name": "Testing", "author_url": "", "post_date": "2020-05-24T15:53:01.657000", "content": "Single Model : 5 checkpoints CV: 0.7160 LB:0.7245
", "votes": 2, "replies": [] }, { "id": 851858, "author_name": "arutema47", "author_url": "", "post_date": "2020-05-18T01:08:28.120000", "content": "What eval scripts are you guys using for metrics?
\n\nhttps://github.com/cocodataset/cocoapi/blob/master/PythonAPI/pycocotools/cocoeval.py\nI think a good standard is cocoeval, and we can teak lines 498- as \npython\nclass Params:\n '''\n Params for coco evaluation api\n '''\n def setDetParams(self):\n self.imgIds = []\n self.catIds = []\n # np.arange causes trouble. the data point on arange is slightly larger than the true value\n self.iouThrs = np.linspace(.5, 0.75, int(np.round((0.75 - .5) / .05)) + 1, endpoint=True)\n self.recThrs = np.linspace(.0, 1.00, int(np.round((1.00 - .0) / .01)) + 1, endpoint=True)\n self.maxDets = [1, 10, 100]\n self.areaRng = [[0 ** 2, 1e5 ** 2], [0 ** 2, 32 ** 2], [32 ** 2, 96 ** 2], [96 ** 2, 1e5 ** 2]]\n self.areaRngLbl = ['all', 'small', 'medium', 'large']\n self.useCats = 1\n
to realize the metrics as provided in this competition.
\n\nI also tried out Peter's eval scripts, but there seems to be divergence against the cocoeval results.
\n\nBTW with centernet single fold single model, I get cocoCV 0.69 LB 0.65.
", "votes": 2, "replies": [ { "id": 851992, "author_name": "Peter", "author_url": "", "post_date": "2020-05-18T05:12:30.743000", "content": "I can confirm your result, it seems there is a bug in my script. I'll try to debug and fix it as soon as I'll have a bit of time.
", "votes": 2, "replies": [] }, { "id": 852965, "author_name": "Weimin Wang", "author_url": "", "post_date": "2020-05-18T20:58:07.707000", "content": "@pestipeti I just did some experiments, for each epoch, I just printed out the valuation score using your script, vs using the cocoeval script, and here are the results:
\n\nyour_score, coco_score\n0.430711174576225, 0.6523691508406063\n0.4785398646805657, 0.6892521575561043\n0.48073431913247816, 0.6962216598700071\n0.49023349687333, 0.7288417268693561\n0.47705885089740174, 0.7230724922949511\n0.4660380495242629, 0.7376911721208542\n0.5314250379854468, 0.7414370430705847
\n\nSo it seems your script gives consistently much lower score than what coco eval gives, but good thing is both are roughly consistent when ups or downs, which means people can still use your script to select the best model / checkpoint, and submit to LB for verification.
\n\nFYI for people who are interested in how to use coco eval - you will need to \n1) create ground truth json as in coco format. \n2) dump your predictions into json format in local, \n3) run eval using coco APIs, following this example - https://github.com/cocodataset/cocoapi/blob/master/PythonAPI/pycocoEvalDemo.ipynb. Most importantly, set this param iou threshods: cocoEval.params.iouThrs = np.linspace(.5, 0.75, np.round((0.75 - .5) / .05) + 1, endpoint=True)
Thanks, @weimin!
", "votes": 1, "replies": [] }, { "id": 853930, "author_name": "mrxuehb", "author_url": "", "post_date": "2020-05-19T16:05:12.113000", "content": "May I ask do we still need to change how average precision is calculated in cocoapi? Because in this competition, ap=tp/(tp+fp+fn), but in cocoapi, it seems ap = tp/(tp+fp). Correct me if I am wrong.
", "votes": 3, "replies": [] }, { "id": 854038, "author_name": "Peter", "author_url": "", "post_date": "2020-05-19T17:37:55.937000", "content": "It is interesting; @weimin, can you confirm that without FN, the result of my script equals COCOEval?
", "votes": 0, "replies": [] }, { "id": 854340, "author_name": "Weimin Wang", "author_url": "", "post_date": "2020-05-20T01:09:52.973000", "content": "This is what I got - \nScore using coco eval - 0.6741089713448498\nNew yours (tp / (tp + fp)) - 0.4924787830466831 - old yours (tp / (tp + fp + fn))- 0.43898056126435475\nAll I did for new is to replace - \n #return tp / (tp + fp + fn)\n return tp / (tp + fp)
\n\nfor your calculate_precision function.
", "votes": 3, "replies": [] }, { "id": 854412, "author_name": "mrxuehb", "author_url": "", "post_date": "2020-05-20T02:32:47.140000", "content": "A basic question: it seems mAP in coco is related to precision-recall curve, which is not simple averge precision in different iou thresholds in this case. What do you think?
", "votes": 2, "replies": [] }, { "id": 855156, "author_name": "Weimin Wang", "author_url": "", "post_date": "2020-05-20T16:32:26.210000", "content": "yeah, I agreed with what you said, it seems to me that peter's calculation should be more related to this competition metric than COCO's... it could be helpful if the host team can release some sort of code for the metric.
", "votes": 0, "replies": [] }, { "id": 855233, "author_name": "Peter", "author_url": "", "post_date": "2020-05-20T17:30:40.553000", "content": "I agree. As @mrxuehb said, COCO is related to the precision-recall curve. Kaggle probably is not using COCOeval; they would have mentioned in the evaluation description.
\n\nI think the reason for the high cv-lb gap is not because of a bug. See the image below; I calculated the mAP per source. \n(The final mAP is 0.602 (The LB was 0.7102) for this experiment.)
\n\n
As you can see, one of the sources (rres_1) had a 0.7266 mAP and a more difficult one had 0.4588 (arvalis_2). I think this simply means that the samples in the public LB have a similar difficulty level that rres_1.
IMO, the private LB could be very different.
", "votes": 12, "replies": [] }, { "id": 855254, "author_name": "Weimin Wang", "author_url": "", "post_date": "2020-05-20T18:01:05.753000", "content": "Good findings!
", "votes": 0, "replies": [] }, { "id": 855388, "author_name": "Matt", "author_url": "", "post_date": "2020-05-20T21:52:15.010000", "content": "Thanks for this great discussion guys, really helpful!
", "votes": 0, "replies": [] }, { "id": 855890, "author_name": "arutema47", "author_url": "", "post_date": "2020-05-21T10:14:43.460000", "content": "@pestipeti \nGreat findings! \nSounds like shake-down coming.. fingers crossed.
", "votes": 0, "replies": [] } ] }, { "id": 839357, "author_name": "tereka", "author_url": "", "post_date": "2020-05-09T10:43:41.140000", "content": "CV:?(no check....) LB: 0.6959\nrandom split 9 : 1, single fold only
", "votes": 2, "replies": [] }, { "id": 877282, "author_name": "XY", "author_url": "", "post_date": "2020-06-07T13:11:03.413000", "content": "Another data point: my single fold v score is 0.6884 and LB score is 0.7178 using Peter's metric.
", "votes": 0, "replies": [] }, { "id": 851131, "author_name": "", "author_url": "", "post_date": "2020-05-17T11:23:26.603000", "content": "", "votes": 2, "replies": [] } ], "raw_markdown_by_id": { "838550": "I finally overcame Submission CSV Not Found error, now I'm very happy to report my cv and lb score relationship! (Even though it's not so good😹)\n\nMy single fold cv score (out of 5 folds) is 0.5882 and LB score is 0.5805. Looks not so bad, doesn't it?\n\nHow about you?", "838598": "single fold, CV: 0.723, LB: 0.7037", "854427": "I use Peter's metrics, CV: 65.4, LB:68.02, randomly 8:2 split.", "848347": "So how to split data to measure cv？Just randomly split? Or take one source as valid, others as train?", "841720": "single fold, CV: 0.6519, LB: 0.6485\nValidation strategies - [kRegionsOut](https://www.kaggle.com/c/global-wheat-detection/discussion/148696)", "859638": "Single Model : 5 checkpoints CV: 0.7160 LB:0.7245", "851858": "What eval scripts are you guys using for metrics?\n\nhttps://github.com/cocodataset/cocoapi/blob/master/PythonAPI/pycocotools/cocoeval.py\nI think a good standard is cocoeval, and we can teak lines 498- as \n```python\nclass Params:\n '''\n Params for coco evaluation api\n '''\n def setDetParams(self):\n self.imgIds = []\n self.catIds = []\n # np.arange causes trouble. the data point on arange is slightly larger than the true value\n self.iouThrs = np.linspace(.5, 0.75, int(np.round((0.75 - .5) / .05)) + 1, endpoint=True)\n self.recThrs = np.linspace(.0, 1.00, int(np.round((1.00 - .0) / .01)) + 1, endpoint=True)\n self.maxDets = [1, 10, 100]\n self.areaRng = [[0 ** 2, 1e5 ** 2], [0 ** 2, 32 ** 2], [32 ** 2, 96 ** 2], [96 ** 2, 1e5 ** 2]]\n self.areaRngLbl = ['all', 'small', 'medium', 'large']\n self.useCats = 1\n```\n\nto realize the metrics as provided in this competition.\n\nI also tried out Peter's eval scripts, but there seems to be divergence against the cocoeval results.\n\nBTW with centernet single fold single model, I get cocoCV 0.69 LB 0.65.", "839357": "CV:?(no check....) LB: 0.6959\nrandom split 9 : 1, single fold only", "877282": "Another data point: my single fold v score is 0.6884 and LB score is 0.7178 using Peter's metric. ", "851131": "" } }