I assume the purpose of this competition is twofold below.\n1. create accurate wheat head detection in outdoor field images \n2. create model that is robust to the growing environment
\n\nSince pseudo labeling use test domain data, it ignore robustness to growing enviroment. If it is permitted, final solution may not be worth for competition host. Pseudo labeling is not forbidden, but considering the purpose of the competition participant shouldn't use it. \nSo I'd like to ask competition host if you understands this point.
", "messages": [ { "id": "836543", "postDate": "05/07/2020 03:58:55", "content": "I assume the purpose of this competition is twofold below.\n1. create accurate wheat head detection in outdoor field images \n2. create model that is robust to the growing environment
\n\nSince pseudo labeling use test domain data, it ignore robustness to growing enviroment. If it is permitted, final solution may not be worth for competition host. Pseudo labeling is not forbidden, but considering the purpose of the competition participant shouldn't use it. \nSo I'd like to ask competition host if you understands this point.
", "rawMarkdown": "I assume the purpose of this competition is twofold below.\n1. create accurate wheat head detection in outdoor field images \n2. create model that is robust to the growing environment\n\nSince pseudo labeling use test domain data, it ignore robustness to growing enviroment. If it is permitted, final solution may not be worth for competition host. Pseudo labeling is not forbidden, but considering the purpose of the competition participant shouldn't use it. \nSo I'd like to ask competition host if you understands this point.", "votes": null }, { "id": "836610", "postDate": "05/07/2020 05:33:38", "content": "Thank you very much for your question.
\n\nIt is the reason why we hide most of the test set, to avoid advanced pseudo-labeling strategies. However, we let very few examples to let people understand what they should expect from a \"different\" source of images. Correct me if I am wrong, but pseudo-labeling is not really technically possible in the context of the competition ?
", "rawMarkdown": "Thank you very much for your question.\n\nIt is the reason why we hide most of the test set, to avoid advanced pseudo-labeling strategies. However, we let very few examples to let people understand what they should expect from a \"different\" source of images. Correct me if I am wrong, but pseudo-labeling is not really technically possible in the context of the competition ?", "votes": null }, { "id": "836637", "postDate": "05/07/2020 05:50:23", "content": "It is possible, see this notebook.
", "rawMarkdown": "It is possible, see [this](https://www.kaggle.com/nvnnghia/fasterrcnn-pseudo-labeling) notebook.", "votes": null }, { "id": "836655", "postDate": "05/07/2020 06:06:18", "content": "Thanks Peter :)\nIn test phase, kaggle server rerun our submitted code with public and private data, so we can use all test data. So If we use pseudo labeling, we can assign pseudo label to all test data and use it for training.
", "rawMarkdown": "Thanks Peter :)\nIn test phase, kaggle server rerun our submitted code with public and private data, so we can use all test data. So If we use pseudo labeling, we can assign pseudo label to all test data and use it for training.", "votes": null }, { "id": "836741", "postDate": "05/07/2020 07:48:05", "content": "Thank you for your answer !
\n\nI think that if such blind pseudo-labeling approaches works within the limit of competition rules, there is no problem ! We are agnostic regarding what we expect, and personally I would be happy if an automated pseudo-labeling approach works. From my experience, pseudo-labeling in this use case tends to amplify error.
\n\nThe reason we have built this competition is precisely to get as many novel approaches as possible. For approaches that don't reach the top of the competition but are still interesting, there is the possibility to submit a small abstract for ECCV 2020. Details will come soon.
", "rawMarkdown": "Thank you for your answer ! \n\nI think that if such blind pseudo-labeling approaches works within the limit of competition rules, there is no problem ! We are agnostic regarding what we expect, and personally I would be happy if an automated pseudo-labeling approach works. From my experience, pseudo-labeling in this use case tends to amplify error. \n\nThe reason we have built this competition is precisely to get as many novel approaches as possible. For approaches that don't reach the top of the competition but are still interesting, there is the possibility to submit a small abstract for ECCV 2020. Details will come soon.", "votes": null }, { "id": "840362", "postDate": "05/10/2020 00:35:36", "content": "@bendvd I suggest you should decrease inference time for GPU. 6 hours is too much for this competition and it makes room to apply pseudo labeling for hidden test set.
", "rawMarkdown": "bendvd I suggest you should decrease inference time for GPU. 6 hours is too much for this competition and it makes room to apply pseudo labeling for hidden test set.", "votes": null }, { "id": "842928", "postDate": "05/11/2020 17:52:01", "content": "Technically speaking it's called 'self supervised' learning and it's touted by smartest people in the field (Yann LeCun is a big fan of this with FAIR) ... Some of the imagnet pretrained weights coming out of FAIR on pytorch are trained also on a billion instagram photo that the model labelled itself.
", "rawMarkdown": "Technically speaking it's called 'self supervised' learning and it's touted by smartest people in the field (Yann LeCun is a big fan of this with FAIR) ... Some of the imagnet pretrained weights coming out of FAIR on pytorch are trained also on a billion instagram photo that the model labelled itself.", "votes": null }, { "id": "850743", "postDate": "05/17/2020 01:07:46", "content": "I'd like to understand if this is fair. The idea of psuedo-labelling makes sense when you have less labelled data and a lot of unlabelled data. If contestants use their own unlabelled dataset for pseudo-labelling, I think this should be allowed. However, using the test set itself provides the model with exposure to test set examples, and I don't think that's fair, it seems like a hack. Please correct me if my understanding is wrong.
", "rawMarkdown": "I'd like to understand if this is fair. The idea of psuedo-labelling makes sense when you have less labelled data and a lot of unlabelled data. If contestants use their own unlabelled dataset for pseudo-labelling, I think this should be allowed. However, using the test set itself provides the model with exposure to test set examples, and I don't think that's fair, it seems like a hack. Please correct me if my understanding is wrong.", "votes": null } ], "comments": [ { "id": 836610, "author_name": "bendvd", "author_url": "", "post_date": "05/07/2020 05:33:38", "content": "Thank you very much for your question.
\n\nIt is the reason why we hide most of the test set, to avoid advanced pseudo-labeling strategies. However, we let very few examples to let people understand what they should expect from a \"different\" source of images. Correct me if I am wrong, but pseudo-labeling is not really technically possible in the context of the competition ?
", "votes": null, "replies": [ { "id": 836637, "author_name": "pestipeti", "author_url": "", "post_date": "05/07/2020 05:50:23", "content": "It is possible, see this notebook.
", "votes": null, "replies": [] }, { "id": 836655, "author_name": "phalanx", "author_url": "", "post_date": "05/07/2020 06:06:18", "content": "Thanks Peter :)\nIn test phase, kaggle server rerun our submitted code with public and private data, so we can use all test data. So If we use pseudo labeling, we can assign pseudo label to all test data and use it for training.
", "votes": null, "replies": [] }, { "id": 836741, "author_name": "bendvd", "author_url": "", "post_date": "05/07/2020 07:48:05", "content": "Thank you for your answer !
\n\nI think that if such blind pseudo-labeling approaches works within the limit of competition rules, there is no problem ! We are agnostic regarding what we expect, and personally I would be happy if an automated pseudo-labeling approach works. From my experience, pseudo-labeling in this use case tends to amplify error.
\n\nThe reason we have built this competition is precisely to get as many novel approaches as possible. For approaches that don't reach the top of the competition but are still interesting, there is the possibility to submit a small abstract for ECCV 2020. Details will come soon.
", "votes": null, "replies": [] }, { "id": 840362, "author_name": "bamps53", "author_url": "", "post_date": "05/10/2020 00:35:36", "content": "@bendvd I suggest you should decrease inference time for GPU. 6 hours is too much for this competition and it makes room to apply pseudo labeling for hidden test set.
", "votes": null, "replies": [] }, { "id": 842928, "author_name": "ma7moud", "author_url": "", "post_date": "05/11/2020 17:52:01", "content": "Technically speaking it's called 'self supervised' learning and it's touted by smartest people in the field (Yann LeCun is a big fan of this with FAIR) ... Some of the imagnet pretrained weights coming out of FAIR on pytorch are trained also on a billion instagram photo that the model labelled itself.
", "votes": null, "replies": [] }, { "id": 850743, "author_name": "arvsome", "author_url": "", "post_date": "05/17/2020 01:07:46", "content": "I'd like to understand if this is fair. The idea of psuedo-labelling makes sense when you have less labelled data and a lot of unlabelled data. If contestants use their own unlabelled dataset for pseudo-labelling, I think this should be allowed. However, using the test set itself provides the model with exposure to test set examples, and I don't think that's fair, it seems like a hack. Please correct me if my understanding is wrong.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "836543": "I assume the purpose of this competition is twofold below.\n1. create accurate wheat head detection in outdoor field images \n2. create model that is robust to the growing environment\n\nSince pseudo labeling use test domain data, it ignore robustness to growing enviroment. If it is permitted, final solution may not be worth for competition host. Pseudo labeling is not forbidden, but considering the purpose of the competition participant shouldn't use it. \nSo I'd like to ask competition host if you understands this point.", "836610": "Thank you very much for your question.\n\nIt is the reason why we hide most of the test set, to avoid advanced pseudo-labeling strategies. However, we let very few examples to let people understand what they should expect from a \"different\" source of images. Correct me if I am wrong, but pseudo-labeling is not really technically possible in the context of the competition ?", "836637": "It is possible, see [this](https://www.kaggle.com/nvnnghia/fasterrcnn-pseudo-labeling) notebook.", "836655": "Thanks Peter :)\nIn test phase, kaggle server rerun our submitted code with public and private data, so we can use all test data. So If we use pseudo labeling, we can assign pseudo label to all test data and use it for training.", "836741": "Thank you for your answer ! \n\nI think that if such blind pseudo-labeling approaches works within the limit of competition rules, there is no problem ! We are agnostic regarding what we expect, and personally I would be happy if an automated pseudo-labeling approach works. From my experience, pseudo-labeling in this use case tends to amplify error. \n\nThe reason we have built this competition is precisely to get as many novel approaches as possible. For approaches that don't reach the top of the competition but are still interesting, there is the possibility to submit a small abstract for ECCV 2020. Details will come soon.", "840362": "bendvd I suggest you should decrease inference time for GPU. 6 hours is too much for this competition and it makes room to apply pseudo labeling for hidden test set.", "842928": "Technically speaking it's called 'self supervised' learning and it's touted by smartest people in the field (Yann LeCun is a big fan of this with FAIR) ... Some of the imagnet pretrained weights coming out of FAIR on pytorch are trained also on a billion instagram photo that the model labelled itself.", "850743": "I'd like to understand if this is fair. The idea of psuedo-labelling makes sense when you have less labelled data and a lot of unlabelled data. If contestants use their own unlabelled dataset for pseudo-labelling, I think this should be allowed. However, using the test set itself provides the model with exposure to test set examples, and I don't think that's fair, it seems like a hack. Please correct me if my understanding is wrong." }, "source": "meta" }