At the end of the competition a different "private" test set will be used to evaluate the different models.\nThe question is if this will include a new set of control (i.e. check_agreement and check_correlation) data sets that need to be passed?
", "messages": [ { "id": "86693", "postDate": "07/23/2015 21:37:14", "content": "At the end of the competition a different "private" test set will be used to evaluate the different models.\nThe question is if this will include a new set of control (i.e. check_agreement and check_correlation) data sets that need to be passed?
", "rawMarkdown": "At the end of the competition a different \"private\" test set will be used to evaluate the different models.\r\nThe question is if this will include a new set of control (i.e. check_agreement and check_correlation) data sets that need to be passed?", "votes": null }, { "id": "86728", "postDate": "07/24/2015 00:47:33", "content": "I might be wrong but the way I see it its that the public/private sets are only divided logically, not physically, i.e., there cannot be other control sets, because you already have all the data, and the tests are being applied to all of it. The LB only scores a few points, but that do not change the fact that you passed the tests for all the points.
", "rawMarkdown": "I might be wrong but the way I see it its that the public/private sets are only divided logically, not physically, i.e., there cannot be other control sets, because you already have all the data, and the tests are being applied to all of it. The LB only scores a few points, but that do not change the fact that you passed the tests for all the points.", "votes": null }, { "id": "86733", "postDate": "07/24/2015 02:08:35", "content": "@udibr,
\n\nSo if we've see all the data for the control, we can specifically fit these control settings (for example, using the control dataset, are we allowed?) to pass the agreement, then one can get very high score.
\n\nIn one extreme, one can build a model that could detect whether the data is not a tao->3mu decay (which should not be very difficult, using all the train + control dataset), then it produces a random forecast drawn from uniform distribution. In some sense, the control is kind of meaningless and one can get very high score.
\n\nSo it seems we should not be allowed to do any training with the control dataset... but that seems impossible, everybody will have some kind of fitting to the control data-set by trying models hundreds of times even if one did not specifically train the model based on the control dataset. it is a matter of the degree of overfitting..... In this case, we are not overfitting to the target variable, but overfitting to a constrain..
\n\n[quote=Wendy Kan;86733]
\n\n@udibr,
\n\n[/quote]
", "rawMarkdown": "So if we've see all the data for the control, we can specifically fit these control settings (for example, using the control dataset, are we allowed?) to pass the agreement, then one can get very high score. \r\n\r\nIn one extreme, one can build a model that could detect whether the data is not a tao->3mu decay (which should not be very difficult, using all the train + control dataset), then it produces a random forecast drawn from uniform distribution. In some sense, the control is kind of meaningless and one can get very high score.\r\n\r\nSo it seems we should not be allowed to do any training with the control dataset... but that seems impossible, everybody will have some kind of fitting to the control data-set by trying models hundreds of times even if one did not specifically train the model based on the control dataset. it is a matter of the degree of overfitting..... In this case, we are not overfitting to the target variable, but overfitting to a constrain..\r\n\r\n\r\n[quote=Wendy Kan;86733]\r\n\r\n@udibr,\r\n\r\n - When you make a submission, we perform the agreement and correlation tests on the submission. We only calculate the weighted AUC if both tests pass. \r\n - When the weighted AUC is calculated, the data is split into public/private sets, and those correspond to public/private LB scores. \r\n - In other words, tests are only performed once to check the physics criteria, not \"once for public and once for private\"\r\n - You can locally run the checks yourself using check_agreement and check_correlation, however, the check_agreement and check_correlation that we provide for you are subsets of the actual ones we run on our server. This is done to prevent leakage. \r\n - If I understand correctly, @NxGTR is correct. \r\n\r\n[/quote]", "votes": null }, { "id": "87183", "postDate": "07/27/2015 16:57:25", "content": "@woshialex
\n\nThis is a discussion we had when designing this competition. As described in the 4th bullet point, the control dataset that is used on our production server uses much larger versions of check_agreement and check_correlation. Yes, you may overfit to the constraint based on the subset you are given, but that isn't the compete dataset.
\n\nAlso, we're monitoring closely to see if people are trying to submit hundred of times to overfit the constraint, and not the target variable.
", "rawMarkdown": "woshialex\r\n\r\nThis is a discussion we had when designing this competition. As described in the 4th bullet point, the control dataset that is used on our production server uses much larger versions of check_agreement and check_correlation. Yes, you may overfit to the constraint based on the subset you are given, but that isn't the compete dataset. \r\n\r\nAlso, we're monitoring closely to see if people are trying to submit hundred of times to overfit the constraint, and not the target variable.", "votes": null } ], "comments": [ { "id": 86728, "author_name": "carloshuertas", "author_url": "", "post_date": "07/24/2015 00:47:33", "content": "I might be wrong but the way I see it its that the public/private sets are only divided logically, not physically, i.e., there cannot be other control sets, because you already have all the data, and the tests are being applied to all of it. The LB only scores a few points, but that do not change the fact that you passed the tests for all the points.
", "votes": null, "replies": [] }, { "id": 86733, "author_name": "wendykan", "author_url": "", "post_date": "07/24/2015 02:08:35", "content": "@udibr,
\n\nSo if we've see all the data for the control, we can specifically fit these control settings (for example, using the control dataset, are we allowed?) to pass the agreement, then one can get very high score.
\n\nIn one extreme, one can build a model that could detect whether the data is not a tao->3mu decay (which should not be very difficult, using all the train + control dataset), then it produces a random forecast drawn from uniform distribution. In some sense, the control is kind of meaningless and one can get very high score.
\n\nSo it seems we should not be allowed to do any training with the control dataset... but that seems impossible, everybody will have some kind of fitting to the control data-set by trying models hundreds of times even if one did not specifically train the model based on the control dataset. it is a matter of the degree of overfitting..... In this case, we are not overfitting to the target variable, but overfitting to a constrain..
\n\n[quote=Wendy Kan;86733]
\n\n@udibr,
\n\n[/quote]
", "votes": null, "replies": [] }, { "id": 87183, "author_name": "wendykan", "author_url": "", "post_date": "07/27/2015 16:57:25", "content": "@woshialex
\n\nThis is a discussion we had when designing this competition. As described in the 4th bullet point, the control dataset that is used on our production server uses much larger versions of check_agreement and check_correlation. Yes, you may overfit to the constraint based on the subset you are given, but that isn't the compete dataset.
\n\nAlso, we're monitoring closely to see if people are trying to submit hundred of times to overfit the constraint, and not the target variable.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "86693": "At the end of the competition a different \"private\" test set will be used to evaluate the different models.\r\nThe question is if this will include a new set of control (i.e. check_agreement and check_correlation) data sets that need to be passed?", "86728": "I might be wrong but the way I see it its that the public/private sets are only divided logically, not physically, i.e., there cannot be other control sets, because you already have all the data, and the tests are being applied to all of it. The LB only scores a few points, but that do not change the fact that you passed the tests for all the points.", "86733": "udibr,\r\n\r\n - When you make a submission, we perform the agreement and correlation tests on the submission. We only calculate the weighted AUC if both tests pass. \r\n - When the weighted AUC is calculated, the data is split into public/private sets, and those correspond to public/private LB scores. \r\n - In other words, tests are only performed once to check the physics criteria, not \"once for public and once for private\"\r\n - You can locally run the checks yourself using check_agreement and check_correlation, however, the check_agreement and check_correlation that we provide for you are subsets of the actual ones we run on our server. This is done to prevent leakage. \r\n - If I understand correctly, @NxGTR is correct.", "87028": "So if we've see all the data for the control, we can specifically fit these control settings (for example, using the control dataset, are we allowed?) to pass the agreement, then one can get very high score. \r\n\r\nIn one extreme, one can build a model that could detect whether the data is not a tao->3mu decay (which should not be very difficult, using all the train + control dataset), then it produces a random forecast drawn from uniform distribution. In some sense, the control is kind of meaningless and one can get very high score.\r\n\r\nSo it seems we should not be allowed to do any training with the control dataset... but that seems impossible, everybody will have some kind of fitting to the control data-set by trying models hundreds of times even if one did not specifically train the model based on the control dataset. it is a matter of the degree of overfitting..... In this case, we are not overfitting to the target variable, but overfitting to a constrain..\r\n\r\n\r\n[quote=Wendy Kan;86733]\r\n\r\n@udibr,\r\n\r\n - When you make a submission, we perform the agreement and correlation tests on the submission. We only calculate the weighted AUC if both tests pass. \r\n - When the weighted AUC is calculated, the data is split into public/private sets, and those correspond to public/private LB scores. \r\n - In other words, tests are only performed once to check the physics criteria, not \"once for public and once for private\"\r\n - You can locally run the checks yourself using check_agreement and check_correlation, however, the check_agreement and check_correlation that we provide for you are subsets of the actual ones we run on our server. This is done to prevent leakage. \r\n - If I understand correctly, @NxGTR is correct. \r\n\r\n[/quote]", "87183": "woshialex\r\n\r\nThis is a discussion we had when designing this competition. As described in the 4th bullet point, the control dataset that is used on our production server uses much larger versions of check_agreement and check_correlation. Yes, you may overfit to the constraint based on the subset you are given, but that isn't the compete dataset. \r\n\r\nAlso, we're monitoring closely to see if people are trying to submit hundred of times to overfit the constraint, and not the target variable." }, "source": "meta" }