{
  "id": 15473,
  "title": "Correlation test for a random classifier fails",
  "url": "/competitions/flavours-of-physics/discussion/15473",
  "author_name": "",
  "post_date": "2015-07-23T11:54:51.657Z",
  "votes": 6,
  "comment_count": 1,
  "views": 493,
  "content": "<p>Hi -</p>\n\n<p>So I have been playing around to understand the meaning the agreement and correlation tests better. In my fiddling, I wanted to to see if a random classifier (i.e. a classifier for which predict_proba(X) = [0.5, 0.5]) would pass or fail the tests. </p>\n\n<p>For the agreement test, a random classifier perfectly passes the checks (KS=0.0), as expected.</p>\n\n<p>For the CVM test, as far as I understand, it is like testing for a correlation between a constant value (0.5) and the mass. In this case however, the test fails with CvM=0.155. This is surprising since, from the description of the test provided in the challenge, we are looking for a bucket in which the prediction distribution is different from the global prediction distribution -- but this cannot happen since the prediction is the same everywhere. This is also surprising since by the standard definition of &quot;correlation&quot;, we know that a constant variable cannot be correlated to another random variable. (Or more precisely, &quot;correlation&quot; is undefined in this case.) </p>\n\n<p>Am I missing something in my interpretation of this test or is it the expected behaviour?  </p>\n\n<pre><code>from sklearn.dummy import DummyClassifier\nclf = DummyClassifier(strategy=&quot;uniform&quot;).fit(X, y)\n\nfrom evaluation import compute_ks\ncheck_agreement = pd.read_csv(&quot;data/check_agreement.csv&quot;, index_col=&quot;id&quot;)\nagreement_probs = clf.predict_proba(check_agreement[features].values)[:, 1]\n\nks = compute_ks(\n    agreement_probs[check_agreement['signal'].values == 0],\n    agreement_probs[check_agreement['signal'].values == 1],\n    check_agreement[check_agreement['signal'] == 0]['weight'].values,\n    check_agreement[check_agreement['signal'] == 1]['weight'].values)\n\nprint &quot;KS =&quot;, ks, ks &lt; 0.09\n\nfrom evaluation import compute_cvm\ncheck_correlation = pd.read_csv(&quot;data/check_correlation.csv&quot;, index_col=&quot;id&quot;)\ncorrelation_probs = clf.predict_proba(check_correlation[features])[:, 1]\ncvm = compute_cvm(correlation_probs, check_correlation['mass'])\nprint &quot;CvM =&quot;, cvm, cvm &lt; 0.002\n\nKS = 0.0 True\nCvM = 0.155842626513 False\n</code></pre>",
  "messages": [
    {
      "id": "86607",
      "postDate": "07/23/2015 11:54:51",
      "content": "<p>Hi -</p>\n\n<p>So I have been playing around to understand the meaning the agreement and correlation tests better. In my fiddling, I wanted to to see if a random classifier (i.e. a classifier for which predict_proba(X) = [0.5, 0.5]) would pass or fail the tests. </p>\n\n<p>For the agreement test, a random classifier perfectly passes the checks (KS=0.0), as expected.</p>\n\n<p>For the CVM test, as far as I understand, it is like testing for a correlation between a constant value (0.5) and the mass. In this case however, the test fails with CvM=0.155. This is surprising since, from the description of the test provided in the challenge, we are looking for a bucket in which the prediction distribution is different from the global prediction distribution -- but this cannot happen since the prediction is the same everywhere. This is also surprising since by the standard definition of &quot;correlation&quot;, we know that a constant variable cannot be correlated to another random variable. (Or more precisely, &quot;correlation&quot; is undefined in this case.) </p>\n\n<p>Am I missing something in my interpretation of this test or is it the expected behaviour?  </p>\n\n<pre><code>from sklearn.dummy import DummyClassifier\nclf = DummyClassifier(strategy=&quot;uniform&quot;).fit(X, y)\n\nfrom evaluation import compute_ks\ncheck_agreement = pd.read_csv(&quot;data/check_agreement.csv&quot;, index_col=&quot;id&quot;)\nagreement_probs = clf.predict_proba(check_agreement[features].values)[:, 1]\n\nks = compute_ks(\n    agreement_probs[check_agreement['signal'].values == 0],\n    agreement_probs[check_agreement['signal'].values == 1],\n    check_agreement[check_agreement['signal'] == 0]['weight'].values,\n    check_agreement[check_agreement['signal'] == 1]['weight'].values)\n\nprint &quot;KS =&quot;, ks, ks &lt; 0.09\n\nfrom evaluation import compute_cvm\ncheck_correlation = pd.read_csv(&quot;data/check_correlation.csv&quot;, index_col=&quot;id&quot;)\ncorrelation_probs = clf.predict_proba(check_correlation[features])[:, 1]\ncvm = compute_cvm(correlation_probs, check_correlation['mass'])\nprint &quot;CvM =&quot;, cvm, cvm &lt; 0.002\n\nKS = 0.0 True\nCvM = 0.155842626513 False\n</code></pre>",
      "rawMarkdown": "Hi -\r\n\r\nSo I have been playing around to understand the meaning the agreement and correlation tests better. In my fiddling, I wanted to to see if a random classifier (i.e. a classifier for which predict_proba(X) = [0.5, 0.5]) would pass or fail the tests. \r\n\r\nFor the agreement test, a random classifier perfectly passes the checks (KS=0.0), as expected.\r\n\r\nFor the CVM test, as far as I understand, it is like testing for a correlation between a constant value (0.5) and the mass. In this case however, the test fails with CvM=0.155. This is surprising since, from the description of the test provided in the challenge, we are looking for a bucket in which the prediction distribution is different from the global prediction distribution -- but this cannot happen since the prediction is the same everywhere. This is also surprising since by the standard definition of \"correlation\", we know that a constant variable cannot be correlated to another random variable. (Or more precisely, \"correlation\" is undefined in this case.) \r\n\r\nAm I missing something in my interpretation of this test or is it the expected behaviour?  \r\n\r\n<pre><code>from sklearn.dummy import DummyClassifier\r\nclf = DummyClassifier(strategy=\"uniform\").fit(X, y)\r\n\r\nfrom evaluation import compute_ks\r\ncheck_agreement = pd.read_csv(\"data/check_agreement.csv\", index_col=\"id\")\r\nagreement_probs = clf.predict_proba(check_agreement[features].values)[:, 1]\r\n\r\nks = compute_ks(\r\n    agreement_probs[check_agreement['signal'].values == 0],\r\n    agreement_probs[check_agreement['signal'].values == 1],\r\n    check_agreement[check_agreement['signal'] == 0]['weight'].values,\r\n    check_agreement[check_agreement['signal'] == 1]['weight'].values)\r\n\r\nprint \"KS =\", ks, ks &lt; 0.09\r\n\r\nfrom evaluation import compute_cvm\r\ncheck_correlation = pd.read_csv(\"data/check_correlation.csv\", index_col=\"id\")\r\ncorrelation_probs = clf.predict_proba(check_correlation[features])[:, 1]\r\ncvm = compute_cvm(correlation_probs, check_correlation['mass'])\r\nprint \"CvM =\", cvm, cvm &lt; 0.002\r\n\r\nKS = 0.0 True\r\nCvM = 0.155842626513 False\r\n</code></pre>",
      "votes": null
    },
    {
      "id": "86608",
      "postDate": "07/23/2015 12:14:58",
      "content": "<p>Hi, thanks for your investigations :) </p>\n\n<p>Yes, the correlation is absent between the mass and a constant prediction. Since participants could artificially decrease correlation by generating the same predictions for some events, we penalize for stuck together predictions (and the more events have the same prediction, the greater penalty) to prevent from such kind of cheating.</p>\n\n<p>However, this should not be a problem for all usual classifiers.</p>\n\n<p>If you submit random predictions (i.e. <code>numpy.random.random(n_events)</code>) correlation test will be passed.</p>",
      "rawMarkdown": "Hi, thanks for your investigations :) \r\n\r\nYes, the correlation is absent between the mass and a constant prediction. Since participants could artificially decrease correlation by generating the same predictions for some events, we penalize for stuck together predictions (and the more events have the same prediction, the greater penalty) to prevent from such kind of cheating.\r\n\r\nHowever, this should not be a problem for all usual classifiers.\r\n\r\nIf you submit random predictions (i.e. ```numpy.random.random(n_events)```) correlation test will be passed.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 86608,
      "author_name": "tataantares",
      "author_url": "",
      "post_date": "07/23/2015 12:14:58",
      "content": "<p>Hi, thanks for your investigations :) </p>\n\n<p>Yes, the correlation is absent between the mass and a constant prediction. Since participants could artificially decrease correlation by generating the same predictions for some events, we penalize for stuck together predictions (and the more events have the same prediction, the greater penalty) to prevent from such kind of cheating.</p>\n\n<p>However, this should not be a problem for all usual classifiers.</p>\n\n<p>If you submit random predictions (i.e. <code>numpy.random.random(n_events)</code>) correlation test will be passed.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "86607": "Hi -\r\n\r\nSo I have been playing around to understand the meaning the agreement and correlation tests better. In my fiddling, I wanted to to see if a random classifier (i.e. a classifier for which predict_proba(X) = [0.5, 0.5]) would pass or fail the tests. \r\n\r\nFor the agreement test, a random classifier perfectly passes the checks (KS=0.0), as expected.\r\n\r\nFor the CVM test, as far as I understand, it is like testing for a correlation between a constant value (0.5) and the mass. In this case however, the test fails with CvM=0.155. This is surprising since, from the description of the test provided in the challenge, we are looking for a bucket in which the prediction distribution is different from the global prediction distribution -- but this cannot happen since the prediction is the same everywhere. This is also surprising since by the standard definition of \"correlation\", we know that a constant variable cannot be correlated to another random variable. (Or more precisely, \"correlation\" is undefined in this case.) \r\n\r\nAm I missing something in my interpretation of this test or is it the expected behaviour?  \r\n\r\n<pre><code>from sklearn.dummy import DummyClassifier\r\nclf = DummyClassifier(strategy=\"uniform\").fit(X, y)\r\n\r\nfrom evaluation import compute_ks\r\ncheck_agreement = pd.read_csv(\"data/check_agreement.csv\", index_col=\"id\")\r\nagreement_probs = clf.predict_proba(check_agreement[features].values)[:, 1]\r\n\r\nks = compute_ks(\r\n    agreement_probs[check_agreement['signal'].values == 0],\r\n    agreement_probs[check_agreement['signal'].values == 1],\r\n    check_agreement[check_agreement['signal'] == 0]['weight'].values,\r\n    check_agreement[check_agreement['signal'] == 1]['weight'].values)\r\n\r\nprint \"KS =\", ks, ks &lt; 0.09\r\n\r\nfrom evaluation import compute_cvm\r\ncheck_correlation = pd.read_csv(\"data/check_correlation.csv\", index_col=\"id\")\r\ncorrelation_probs = clf.predict_proba(check_correlation[features])[:, 1]\r\ncvm = compute_cvm(correlation_probs, check_correlation['mass'])\r\nprint \"CvM =\", cvm, cvm &lt; 0.002\r\n\r\nKS = 0.0 True\r\nCvM = 0.155842626513 False\r\n</code></pre>",
    "86608": "Hi, thanks for your investigations :) \r\n\r\nYes, the correlation is absent between the mass and a constant prediction. Since participants could artificially decrease correlation by generating the same predictions for some events, we penalize for stuck together predictions (and the more events have the same prediction, the greater penalty) to prevent from such kind of cheating.\r\n\r\nHowever, this should not be a problem for all usual classifiers.\r\n\r\nIf you submit random predictions (i.e. ```numpy.random.random(n_events)```) correlation test will be passed."
  },
  "source": "meta"
}