According to the paper, The test data are from 4 sources: UTokyo_1 (994 images), UTokyo_2 (120 images), UQ_1 (142 images) and NAU_1 (20 images). \nPublic/private test data = 0.62/0.38. \nIf I were the organizer, I would use 867 images from UTokyo_1 for the public LB and the rest for private LB. If that is the case, then the private test data includes images from 4 sources and the private score will be much different comparing to public LB score.
\n\nHow to verify my assumption?\nI was thinking of LB probing. I will make 2 submissions, the first submission will make prediction for all the test data and the second submission only predict UTokyo_1 source. If 2 submissions produce the same public LB score, then my assumption is correct.
\n\nIn order probe LB, we need to discriminate UTokyo_1 from others, do you have any suggestion to to this?
", "messages": [ { "id": 850735, "postDate": "2020-05-17T00:53:41.737Z", "content": "According to the paper, The test data are from 4 sources: UTokyo_1 (994 images), UTokyo_2 (120 images), UQ_1 (142 images) and NAU_1 (20 images). \nPublic/private test data = 0.62/0.38. \nIf I were the organizer, I would use 867 images from UTokyo_1 for the public LB and the rest for private LB. If that is the case, then the private test data includes images from 4 sources and the private score will be much different comparing to public LB score.
\n\nHow to verify my assumption?\nI was thinking of LB probing. I will make 2 submissions, the first submission will make prediction for all the test data and the second submission only predict UTokyo_1 source. If 2 submissions produce the same public LB score, then my assumption is correct.
\n\nIn order probe LB, we need to discriminate UTokyo_1 from others, do you have any suggestion to to this?
", "rawMarkdown": "According to the [paper](https://arxiv.org/ftp/arxiv/papers/2005/2005.02162.pdf), The test data are from 4 sources: UTokyo_1 (994 images), UTokyo_2 (120 images), UQ_1 (142 images) and NAU_1 (20 images). \nPublic/private test data = 0.62/0.38. \nIf I were the organizer, I would use 867 images from UTokyo_1 for the public LB and the rest for private LB. If that is the case, then the private test data includes images from 4 sources and the private score will be much different comparing to public LB score.\n\nHow to verify my assumption?\nI was thinking of LB probing. I will make 2 submissions, the first submission will make prediction for all the test data and the second submission only predict UTokyo_1 source. If 2 submissions produce the same public LB score, then my assumption is correct. \n\nIn order probe LB, we need to discriminate UTokyo_1 from others, do you have any suggestion to to this?", "votes": 2 }, { "id": 850866, "postDate": "2020-05-17T05:08:29.050Z", "content": "I guess you'd need extra output to your NN that is categorical ground truth (one hot encoding) but then you'd your metrics and loss (softmax)
", "rawMarkdown": "I guess you'd need extra output to your NN that is categorical ground truth (one hot encoding) but then you'd your metrics and loss (softmax) ", "votes": 1 }, { "id": 862975, "postDate": "2020-05-27T01:55:31.693Z", "content": "Good point, but I think it's challenging to predict only UTokyo_1 images, given that you can't infer the source from the filename. I'm pretty sure that the LB is only UTokyo_1 images looking at the CV/source and LB performance.
", "rawMarkdown": "Good point, but I think it's challenging to predict only UTokyo_1 images, given that you can't infer the source from the filename. I'm pretty sure that the LB is only UTokyo_1 images looking at the CV/source and LB performance.", "votes": 2, "replies": [ { "id": 863099, "postDate": "2020-05-27T04:28:04.977Z", "rawMarkdown": "", "isDeleted": true } ] } ], "comments": [ { "id": 850866, "author_name": "mjsML", "author_url": "", "post_date": "2020-05-17T05:08:29.050000", "content": "I guess you'd need extra output to your NN that is categorical ground truth (one hot encoding) but then you'd your metrics and loss (softmax)
", "votes": 1, "replies": [] }, { "id": 862975, "author_name": "arutema47", "author_url": "", "post_date": "2020-05-27T01:55:31.693000", "content": "Good point, but I think it's challenging to predict only UTokyo_1 images, given that you can't infer the source from the filename. I'm pretty sure that the LB is only UTokyo_1 images looking at the CV/source and LB performance.
", "votes": 2, "replies": [ { "id": 863099, "author_name": "", "author_url": "", "post_date": "2020-05-27T04:28:04.977000", "content": "", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "850735": "According to the [paper](https://arxiv.org/ftp/arxiv/papers/2005/2005.02162.pdf), The test data are from 4 sources: UTokyo_1 (994 images), UTokyo_2 (120 images), UQ_1 (142 images) and NAU_1 (20 images). \nPublic/private test data = 0.62/0.38. \nIf I were the organizer, I would use 867 images from UTokyo_1 for the public LB and the rest for private LB. If that is the case, then the private test data includes images from 4 sources and the private score will be much different comparing to public LB score.\n\nHow to verify my assumption?\nI was thinking of LB probing. I will make 2 submissions, the first submission will make prediction for all the test data and the second submission only predict UTokyo_1 source. If 2 submissions produce the same public LB score, then my assumption is correct. \n\nIn order probe LB, we need to discriminate UTokyo_1 from others, do you have any suggestion to to this?", "850866": "I guess you'd need extra output to your NN that is categorical ground truth (one hot encoding) but then you'd your metrics and loss (softmax) ", "862975": "Good point, but I think it's challenging to predict only UTokyo_1 images, given that you can't infer the source from the filename. I'm pretty sure that the LB is only UTokyo_1 images looking at the CV/source and LB performance." } }