I encountered a situation where the mAP I got in my validation set is ~6% higher than that I got in public LB. Just wanna see if anyone also encounter this?
", "messages": [ { "id": "868825", "postDate": "05/31/2020 14:16:51", "content": "I encountered a situation where the mAP I got in my validation set is ~6% higher than that I got in public LB. Just wanna see if anyone also encounter this?
", "rawMarkdown": "I encountered a situation where the mAP I got in my validation set is ~6% higher than that I got in public LB. Just wanna see if anyone also encounter this?", "votes": null }, { "id": "869299", "postDate": "05/31/2020 23:11:04", "content": "We also encountered that issue. The CV score was around 0.83 while the public LB score was 0.67.
", "rawMarkdown": "We also encountered that issue. The CV score was around 0.83 while the public LB score was 0.67.", "votes": null }, { "id": "869300", "postDate": "05/31/2020 23:15:58", "content": "Hi @minomonter \nThanks for confirming this! \nHow did you calculate your mAP metrics function? Did you borrow the one published in kernel notebooks? (e.g. this notebook)
", "rawMarkdown": "Hi @minomonter \nThanks for confirming this! \nHow did you calculate your mAP metrics function? Did you borrow the one published in kernel notebooks? (e.g. [this notebook](https://www.kaggle.com/pestipeti/competition-metric-details-script))", "votes": null }, { "id": "869317", "postDate": "06/01/2020 00:00:26", "content": "Interesting - I have the exact opposite problem: cv 0.5967, lb 0.7299
", "rawMarkdown": "Interesting - I have the exact opposite problem: cv 0.5967, lb 0.7299", "votes": null }, { "id": "869345", "postDate": "06/01/2020 00:39:16", "content": "Yeah, I'm having the same pattern as you. CV I average around 0.53 but obviously my submissions have been much higher! Not sure how much I can trust this evaluation script
", "rawMarkdown": "Yeah, I'm having the same pattern as you. CV I average around 0.53 but obviously my submissions have been much higher! Not sure how much I can trust this evaluation script", "votes": null }, { "id": "869388", "postDate": "06/01/2020 01:37:45", "content": "did you get such inconsistency with different folds of validation set?
", "rawMarkdown": "did you get such inconsistency with different folds of validation set?", "votes": null }, { "id": "869408", "postDate": "06/01/2020 02:02:35", "content": "No, fold were relatively consistent ranging from 0.51 to 0.55.
", "rawMarkdown": "No, fold were relatively consistent ranging from 0.51 to 0.55.", "votes": null }, { "id": "869493", "postDate": "06/01/2020 03:49:05", "content": "How did you split the data?
\n\nHere is one of my comment on another topic:
\n\n\n", "rawMarkdown": "How did you split the data?\n\nHere is one of my comment on another topic:\n\n> I think the reason for the high cv-lb gap is not because of a bug. See the image below; I calculated the mAP per source. \n> (The final mAP is 0.602 (The LB was 0.7102) for this experiment.)\n> \n> \n> \n> \n> As you can see, one of the sources (`rres_1`) had a 0.7266 mAP and a more difficult one had 0.4588 (`arvalis_2`). I think this simply means that the samples in the **public LB (!)** have a similar difficulty level that `rres_1`.", "votes": null }, { "id": "869503", "postDate": "06/01/2020 03:59:05", "content": "I think the reason for the high cv-lb gap is not because of a bug. See the image below; I calculated the mAP per source. \n (The final mAP is 0.602 (The LB was 0.7102) for this experiment.)
\n \n\n \n
As you can see, one of the sources (
\nrres_1) had a 0.7266 mAP and a more difficult one had 0.4588 (arvalis_2). I think this simply means that the samples in the public LB (!) have a similar difficulty level thatrres_1.
This is insightful! Didn't think it may have sth to do with the data source. \nI simply do random split in my training set, so I think my validation set is a mix of different sources, hence causing such discrepancy. \nDid you close the cv-lb gap by taking image only from rres_1 as validation set?
", "rawMarkdown": "This is insightful! Didn't think it may have sth to do with the data source. \nI simply do random split in my training set, so I think my validation set is a mix of different sources, hence causing such discrepancy. \nDid you close the cv-lb gap by taking image only from rres_1 as validation set?", "votes": null }, { "id": "869654", "postDate": "06/01/2020 07:23:50", "content": "Yes, we took that function and modified a bit for our pipeline.
", "rawMarkdown": "Yes, we took that function and modified a bit for our pipeline.", "votes": null }, { "id": "871311", "postDate": "06/02/2020 09:02:57", "content": "Konrad, how do you split data?\nIf using folds from Alex's kernels, local validaiton is ~0.67-0.68 for one fold.
", "rawMarkdown": "Konrad, how do you split data?\nIf using folds from Alex's kernels, local validaiton is ~0.67-0.68 for one fold.", "votes": null }, { "id": "871343", "postDate": "06/02/2020 09:28:25", "content": "0.83 ? IOU@0.5 or IOU@[0.5:0.75:0.05]
", "rawMarkdown": "0.83 ? IOU@0.5 or IOU@[0.5:0.75:0.05]", "votes": null }, { "id": "871556", "postDate": "06/02/2020 13:09:56", "content": "I am using leave-one-region-out as a basis for my cv split - I believe it mimics the difference between training and test set more accurately.
", "rawMarkdown": "I am using leave-one-region-out as a basis for my cv split - I believe it mimics the difference between training and test set more accurately.", "votes": null }, { "id": "948177", "postDate": "07/27/2020 17:56:22", "content": "@pestipeti thanks for sharing that chart. Would you happen to know what the sources are for each trace?
", "rawMarkdown": "pestipeti thanks for sharing that chart. Would you happen to know what the sources are for each trace?", "votes": null }, { "id": "948201", "postDate": "07/27/2020 18:13:04", "content": "@alexandersoare \nThe last (right-most) values of the traces (from top to bottom):\n- rres (0.72659)\n- inrae (0.706732)\n- ethz (0.664106)\n- arvalis3 (pink; 0.583327)\n- usask (orange; 0.577871)\n- arvalis1 (0.545167)\n- arvalis2 (0.458834)
", "rawMarkdown": "alexandersoare \nThe last (right-most) values of the traces (from top to bottom):\n- rres (0.72659)\n- inrae (0.706732)\n- ethz (0.664106)\n- arvalis3 (pink; 0.583327)\n- usask (orange; 0.577871)\n- arvalis1 (0.545167)\n- arvalis2 (0.458834)", "votes": null }, { "id": "948253", "postDate": "07/27/2020 19:02:27", "content": "Many thanks Peter!
", "rawMarkdown": "Many thanks Peter!", "votes": null } ], "comments": [ { "id": 869299, "author_name": "minomonter", "author_url": "", "post_date": "05/31/2020 23:11:04", "content": "We also encountered that issue. The CV score was around 0.83 while the public LB score was 0.67.
", "votes": null, "replies": [ { "id": 871343, "author_name": "fengyun0720", "author_url": "", "post_date": "06/02/2020 09:28:25", "content": "0.83 ? IOU@0.5 or IOU@[0.5:0.75:0.05]
", "votes": null, "replies": [] } ] }, { "id": 869300, "author_name": "alexlwh", "author_url": "", "post_date": "05/31/2020 23:15:58", "content": "Hi @minomonter \nThanks for confirming this! \nHow did you calculate your mAP metrics function? Did you borrow the one published in kernel notebooks? (e.g. this notebook)
", "votes": null, "replies": [ { "id": 869654, "author_name": "minomonter", "author_url": "", "post_date": "06/01/2020 07:23:50", "content": "Yes, we took that function and modified a bit for our pipeline.
", "votes": null, "replies": [] } ] }, { "id": 869317, "author_name": "konradb", "author_url": "", "post_date": "06/01/2020 00:00:26", "content": "Interesting - I have the exact opposite problem: cv 0.5967, lb 0.7299
", "votes": null, "replies": [ { "id": 869345, "author_name": "matthewmasters", "author_url": "", "post_date": "06/01/2020 00:39:16", "content": "Yeah, I'm having the same pattern as you. CV I average around 0.53 but obviously my submissions have been much higher! Not sure how much I can trust this evaluation script
", "votes": null, "replies": [] }, { "id": 869388, "author_name": "alexlwh", "author_url": "", "post_date": "06/01/2020 01:37:45", "content": "did you get such inconsistency with different folds of validation set?
", "votes": null, "replies": [] }, { "id": 869408, "author_name": "matthewmasters", "author_url": "", "post_date": "06/01/2020 02:02:35", "content": "No, fold were relatively consistent ranging from 0.51 to 0.55.
", "votes": null, "replies": [] }, { "id": 871311, "author_name": "sergeyzlobin", "author_url": "", "post_date": "06/02/2020 09:02:57", "content": "Konrad, how do you split data?\nIf using folds from Alex's kernels, local validaiton is ~0.67-0.68 for one fold.
", "votes": null, "replies": [] }, { "id": 871556, "author_name": "konradb", "author_url": "", "post_date": "06/02/2020 13:09:56", "content": "I am using leave-one-region-out as a basis for my cv split - I believe it mimics the difference between training and test set more accurately.
", "votes": null, "replies": [] } ] }, { "id": 869493, "author_name": "pestipeti", "author_url": "", "post_date": "06/01/2020 03:49:05", "content": "How did you split the data?
\n\nHere is one of my comment on another topic:
\n\n\n", "votes": null, "replies": [ { "id": 869503, "author_name": "alexlwh", "author_url": "", "post_date": "06/01/2020 03:59:05", "content": "I think the reason for the high cv-lb gap is not because of a bug. See the image below; I calculated the mAP per source. \n (The final mAP is 0.602 (The LB was 0.7102) for this experiment.)
\n \n\n \n
As you can see, one of the sources (
\nrres_1) had a 0.7266 mAP and a more difficult one had 0.4588 (arvalis_2). I think this simply means that the samples in the public LB (!) have a similar difficulty level thatrres_1.
This is insightful! Didn't think it may have sth to do with the data source. \nI simply do random split in my training set, so I think my validation set is a mix of different sources, hence causing such discrepancy. \nDid you close the cv-lb gap by taking image only from rres_1 as validation set?
", "votes": null, "replies": [] }, { "id": 948177, "author_name": "alexandersoare", "author_url": "", "post_date": "07/27/2020 17:56:22", "content": "@pestipeti thanks for sharing that chart. Would you happen to know what the sources are for each trace?
", "votes": null, "replies": [] }, { "id": 948201, "author_name": "pestipeti", "author_url": "", "post_date": "07/27/2020 18:13:04", "content": "@alexandersoare \nThe last (right-most) values of the traces (from top to bottom):\n- rres (0.72659)\n- inrae (0.706732)\n- ethz (0.664106)\n- arvalis3 (pink; 0.583327)\n- usask (orange; 0.577871)\n- arvalis1 (0.545167)\n- arvalis2 (0.458834)
", "votes": null, "replies": [] }, { "id": 948253, "author_name": "alexandersoare", "author_url": "", "post_date": "07/27/2020 19:02:27", "content": "Many thanks Peter!
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "868825": "I encountered a situation where the mAP I got in my validation set is ~6% higher than that I got in public LB. Just wanna see if anyone also encounter this?", "869299": "We also encountered that issue. The CV score was around 0.83 while the public LB score was 0.67.", "869300": "Hi @minomonter \nThanks for confirming this! \nHow did you calculate your mAP metrics function? Did you borrow the one published in kernel notebooks? (e.g. [this notebook](https://www.kaggle.com/pestipeti/competition-metric-details-script))", "869317": "Interesting - I have the exact opposite problem: cv 0.5967, lb 0.7299", "869345": "Yeah, I'm having the same pattern as you. CV I average around 0.53 but obviously my submissions have been much higher! Not sure how much I can trust this evaluation script", "869388": "did you get such inconsistency with different folds of validation set?", "869408": "No, fold were relatively consistent ranging from 0.51 to 0.55.", "869493": "How did you split the data?\n\nHere is one of my comment on another topic:\n\n> I think the reason for the high cv-lb gap is not because of a bug. See the image below; I calculated the mAP per source. \n> (The final mAP is 0.602 (The LB was 0.7102) for this experiment.)\n> \n> \n> \n> \n> As you can see, one of the sources (`rres_1`) had a 0.7266 mAP and a more difficult one had 0.4588 (`arvalis_2`). I think this simply means that the samples in the **public LB (!)** have a similar difficulty level that `rres_1`.", "869503": "This is insightful! Didn't think it may have sth to do with the data source. \nI simply do random split in my training set, so I think my validation set is a mix of different sources, hence causing such discrepancy. \nDid you close the cv-lb gap by taking image only from rres_1 as validation set?", "869654": "Yes, we took that function and modified a bit for our pipeline.", "871311": "Konrad, how do you split data?\nIf using folds from Alex's kernels, local validaiton is ~0.67-0.68 for one fold.", "871343": "0.83 ? IOU@0.5 or IOU@[0.5:0.75:0.05]", "871556": "I am using leave-one-region-out as a basis for my cv split - I believe it mimics the difference between training and test set more accurately.", "948177": "pestipeti thanks for sharing that chart. Would you happen to know what the sources are for each trace?", "948201": "alexandersoare \nThe last (right-most) values of the traces (from top to bottom):\n- rres (0.72659)\n- inrae (0.706732)\n- ethz (0.664106)\n- arvalis3 (pink; 0.583327)\n- usask (orange; 0.577871)\n- arvalis1 (0.545167)\n- arvalis2 (0.458834)", "948253": "Many thanks Peter!" }, "source": "meta" }![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F864684%2Fcc16bbad8416c8992aecb2e012d7f2ea%2Fresults_1.jpg?generation=1589995713183962&alt=media\")