In this competition our submission needs to pass two tests (agreement test and correlation test). I was wondering how threshold values are set? like how organizers have chosen <0.09 for agreement and <0.02 for correlation? Are they like pchance values used in chi-square test?
", "messages": [ { "id": "86727", "postDate": "07/24/2015 00:45:37", "content": "In this competition our submission needs to pass two tests (agreement test and correlation test). I was wondering how threshold values are set? like how organizers have chosen <0.09 for agreement and <0.02 for correlation? Are they like pchance values used in chi-square test?
", "rawMarkdown": "In this competition our submission needs to pass two tests (agreement test and correlation test). I was wondering how threshold values are set? like how organizers have chosen <0.09 for agreement and <0.02 for correlation? Are they like pchance values used in chi-square test?", "votes": null }, { "id": "86929", "postDate": "07/25/2015 10:52:41", "content": "I have a similar question. \nWhat are significance levels which correspond to these thresholds?\nWhat is the probability that a random solution passes agreement test, correlation test?
", "rawMarkdown": "I have a similar question. \r\nWhat are significance levels which correspond to these thresholds?\r\nWhat is the probability that a random solution passes agreement test, correlation test?", "votes": null }, { "id": "86978", "postDate": "07/25/2015 23:01:05", "content": "Yep, the agreement and the correlation thresholds correspond to p-values. An uncorrelated classifier (or a classifier which is in agreement on the real data and Monte Carlo data) will have metric value less than the threshold (0.002 in the case of correlation metric, 0.09 in the case of agreement metric) with probability > 0.9999 (the same holds true for a random classifier).
", "rawMarkdown": "Yep, the agreement and the correlation thresholds correspond to p-values. An uncorrelated classifier (or a classifier which is in agreement on the real data and Monte Carlo data) will have metric value less than the threshold (0.002 in the case of correlation metric, 0.09 in the case of agreement metric) with probability > 0.9999 (the same holds true for a random classifier).", "votes": null }, { "id": "86981", "postDate": "07/25/2015 23:13:51", "content": "So that means our classifier should be such that minimizes agreement and correlation values (for more certanity) and maximizes ROC score. Am I interpreting it in a correct way?
", "rawMarkdown": "So that means our classifier should be such that minimizes agreement and correlation values (for more certanity) and maximizes ROC score. Am I interpreting it in a correct way?", "votes": null }, { "id": "87012", "postDate": "07/26/2015 10:07:05", "content": "Yep. Only not ROC AUC, but weighted ROC AUC.
", "rawMarkdown": "Yep. Only not ROC AUC, but weighted ROC AUC.", "votes": null } ], "comments": [ { "id": 86929, "author_name": "nedelko", "author_url": "", "post_date": "07/25/2015 10:52:41", "content": "I have a similar question. \nWhat are significance levels which correspond to these thresholds?\nWhat is the probability that a random solution passes agreement test, correlation test?
", "votes": null, "replies": [] }, { "id": 86978, "author_name": "tataantares", "author_url": "", "post_date": "07/25/2015 23:01:05", "content": "Yep, the agreement and the correlation thresholds correspond to p-values. An uncorrelated classifier (or a classifier which is in agreement on the real data and Monte Carlo data) will have metric value less than the threshold (0.002 in the case of correlation metric, 0.09 in the case of agreement metric) with probability > 0.9999 (the same holds true for a random classifier).
", "votes": null, "replies": [] }, { "id": 86981, "author_name": "harshaneel", "author_url": "", "post_date": "07/25/2015 23:13:51", "content": "So that means our classifier should be such that minimizes agreement and correlation values (for more certanity) and maximizes ROC score. Am I interpreting it in a correct way?
", "votes": null, "replies": [] }, { "id": 87012, "author_name": "tataantares", "author_url": "", "post_date": "07/26/2015 10:07:05", "content": "Yep. Only not ROC AUC, but weighted ROC AUC.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "86727": "In this competition our submission needs to pass two tests (agreement test and correlation test). I was wondering how threshold values are set? like how organizers have chosen <0.09 for agreement and <0.02 for correlation? Are they like pchance values used in chi-square test?", "86929": "I have a similar question. \r\nWhat are significance levels which correspond to these thresholds?\r\nWhat is the probability that a random solution passes agreement test, correlation test?", "86978": "Yep, the agreement and the correlation thresholds correspond to p-values. An uncorrelated classifier (or a classifier which is in agreement on the real data and Monte Carlo data) will have metric value less than the threshold (0.002 in the case of correlation metric, 0.09 in the case of agreement metric) with probability > 0.9999 (the same holds true for a random classifier).", "86981": "So that means our classifier should be such that minimizes agreement and correlation values (for more certanity) and maximizes ROC score. Am I interpreting it in a correct way?", "87012": "Yep. Only not ROC AUC, but weighted ROC AUC." }, "source": "meta" }