{
  "id": 5063,
  "title": "Average of two AUCs? Or just one?",
  "url": "/competitions/cause-effect-pairs/discussion/5063",
  "author_name": "",
  "post_date": "2013-07-10T14:38:08.217Z",
  "votes": 7,
  "comment_count": 23,
  "views": 4802,
  "content": "<p>I'm worried that the evaluation metric used on the leaderboard is not as described here&nbsp;<a href=\"http://www.causality.inf.ethz.ch/cause-effect.php?page=tasks\">http://www.causality.inf.ethz.ch/cause-effect.php?page=tasks</a>. I have performed some tests that\r\n suggest that only one AUC is being used (perhaps due to a coding error?) instead of the average of two. I may be wrong, but based on these tests I would suggest checking the evaluation script.</p>\r\n<p>I have trained a classifier on the 2 class problem (A causes B) vs. (everything else). This gets a score of about 0.84. When I use the same method to train a classifier on the problem&nbsp;(B causes A) vs. (everything else) I get a score of about 0.63. This seems\r\n to be at odds with the symmetry of the problem and stated evaluation metric. Using my own train / test split of the training data I have computed AUCs of my method. When using&nbsp;(A causes B) vs. (everything else) for training the classifier and evaluating the\r\n AUC I get about 0.84. When using different 2 class splits for training / evaluation I get an AUC of about 0.64.</p>\r\n<p>These numbers are very similar to those returned by Kaggle - hence my concern that only one AUC is being computed (did the same AUC get calculated twice - or some other simple coding error?).</p>\r\n<p>Is there an error (either the evaluation script or description of the evaluation metric)?</p>",
  "messages": [
    {
      "id": "26943",
      "postDate": "07/10/2013 14:38:08",
      "content": "<p>I'm worried that the evaluation metric used on the leaderboard is not as described here&nbsp;<a href=\"http://www.causality.inf.ethz.ch/cause-effect.php?page=tasks\">http://www.causality.inf.ethz.ch/cause-effect.php?page=tasks</a>. I have performed some tests that\r\n suggest that only one AUC is being used (perhaps due to a coding error?) instead of the average of two. I may be wrong, but based on these tests I would suggest checking the evaluation script.</p>\r\n<p>I have trained a classifier on the 2 class problem (A causes B) vs. (everything else). This gets a score of about 0.84. When I use the same method to train a classifier on the problem&nbsp;(B causes A) vs. (everything else) I get a score of about 0.63. This seems\r\n to be at odds with the symmetry of the problem and stated evaluation metric. Using my own train / test split of the training data I have computed AUCs of my method. When using&nbsp;(A causes B) vs. (everything else) for training the classifier and evaluating the\r\n AUC I get about 0.84. When using different 2 class splits for training / evaluation I get an AUC of about 0.64.</p>\r\n<p>These numbers are very similar to those returned by Kaggle - hence my concern that only one AUC is being computed (did the same AUC get calculated twice - or some other simple coding error?).</p>\r\n<p>Is there an error (either the evaluation script or description of the evaluation metric)?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "27012",
      "postDate": "07/11/2013 15:37:35",
      "content": "<p>James, I think you have a very good question there!&nbsp;</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "27013",
      "postDate": "07/11/2013 15:51:27",
      "content": "<p>It's just a bad evaluation metric. So, just carry on - the competition is ruined once again. This was an observation that didn't need to be shared</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "27016",
      "postDate": "07/11/2013 16:13:29",
      "content": "<p>Well, this is quite serious. Any comment from the organisers ?&nbsp;</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "27017",
      "postDate": "07/11/2013 16:14:33",
      "content": "<p>I think it's great that James shared it. It's a flaw that needs fixing otherwise we are not doing science.</p>\r\n<p></p>\r\n<p>There is indeed a bug, I tried it myself -not bothering to forecast the B-&gt;A cases- and our score jumped up. This is not fair to the other contestants who are not aware of the bug, so making it public is the correct thing to do.</p>\r\n<p></p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "27019",
      "postDate": "07/11/2013 16:37:54",
      "content": "<p>Couldn't agree more. Kudos to James for sticking his neck out. Hopefully we'll hear from the admins soon.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "27022",
      "postDate": "07/11/2013 18:46:55",
      "content": "<p>It's not a bug - it's bad choice of evaluation metric</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "27023",
      "postDate": "07/11/2013 18:48:20",
      "content": "<p>Sorry for the delayed answer. I checked and indeed you seem to be right. The current score used is asymmetric. I am having Kaggle check into this. The error may have been introduced when the switch to the new validation set was made. Prior to that the symmetric\r\n score was used (as indicated on the website).</p>\r\n<p>Hopefully this will be rectified soon. Thank you for pointing out the error.</p>\r\n<p>Isabelle</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "27024",
      "postDate": "07/11/2013 18:48:53",
      "content": "<p>Are you sure? because I get the exact same in training</p>\r\n<p></p>\r\n<p>I get 86 AUC one way A-&gt;B and 82 AUC B-&gt;A in training so the score I got I was expecting</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "27025",
      "postDate": "07/11/2013 18:59:35",
      "content": "<p>The evaluation metric could be chosen to be anything, it's part of the challenge. &nbsp;However the implementation of the described metric has indeed a bug: it does not adhere to the specs, it behaves different.</p>\r\n<p></p>\r\n<p>A lot of people must have been building symmetrical model, probably tweaking them accidentally into a direction that was preferring asymmetry. It would be really helpful if *all* historical submissions were re-evaluated with the fixed metric rather than\r\n resetting us all again to below 0.50. Do you think that's possible Isabella? It would provide us with much better feedback compared to a reset.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "27027",
      "postDate": "07/11/2013 19:00:34",
      "content": "<p>[quote=Sitmo;27017]</p>\r\n<p>I think it's great that James shared it. It's a flaw that needs fixing otherwise we are not doing science</p>\r\n<p></p>\r\n<p>[/quote]</p>\r\n<p></p>\r\n<p>I didn't realise we were doing science, I thought it was a Kaggle competition where many competitions have been won by de-anonymising and data leakage (take the recent KDD competition) The science of this whole academic field has been put to the test. The\r\n way that artificial data is created is flawed, the metric is deifnitely flawed (going by my training results, the leaderboard reflects this score) and there's no current &quot;state-of-the-art&quot; benchmark for the new data. I gave up on the science ages ago.</p>\r\n<p></p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "27028",
      "postDate": "07/11/2013 19:05:42",
      "content": "<p>[quote=Sitmo;27025]</p>\r\n<p>The evaluation metric could be chosen to be anything, it's part of the challenge. &nbsp;However the implementation of the described metric has indeed a bug: it does not adhere to the specs, it behaves different.</p>\r\n<p></p>\r\n<p>A lot of people must have been building symmetrical model, probably tweaking them accidentally into a direction that was preferring asymmetry. It would be really helpful if *all* historical submissions were re-evaluated with the fixed metric rather than\r\n resetting us all again to below 0.50. Do you think that's possible Isabella? It would provide us with much better feedback compared to a reset.</p>\r\n<p>[/quote]</p>\r\n<p></p>\r\n<p>Well, it must be an R bug then because as I've said I get over 80% AUC both ways in training (and none of my predictions are &lt; 0)</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "27030",
      "postDate": "07/11/2013 19:31:36",
      "content": "<p>I think it depends if you use the ROCR package or the VERIFICATION package (making -1s to 1s). On one I get 30 AUC and the other 80. It's how it deals with the -1. Sorry James - I just couldn't understand what you were saying because of the symmetry I was\r\n getting following the same model. </p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "27035",
      "postDate": "07/11/2013 21:17:22",
      "content": "<p>For computing the AUC, the target values should be binary. Some packages use 0/1 for target values, others use -1/1.</p>\r\n<p>Let us call NEG and POS the two target values. Let us call Y the trinary targets of the challenge. You can transform Y into 2 target vectors Y1 and Y2:</p>\r\n<p>Y1 = POS if Y==1 and Y1 = NEG otherwise</p>\r\n<p>Y2 = NEG if Y==-1 and Y2 = POS otherwise</p>\r\n<p>Then compute 2 AUCS, one for Y1 and one for Y2, and average the results to obtain the score of the challenge.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "27062",
      "postDate": "07/12/2013 14:46:04",
      "content": "<p>What does the score on the leaderboard?</p>\r\n<p>A hilarious competition!&nbsp;</p>\r\n<p>Changing data, not understandable assessment.</p>\r\n<p>In general a lot of fun!</p>\r\n<p>Good luck to everyone!</p>\r\n<p></p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "27071",
      "postDate": "07/12/2013 15:32:17",
      "content": "<p>lol you just made me laugh. First time I've laughed about it since the &quot;troubles&quot;. lol Been a Kaggle member for years and this is the first time I've turned into an insane cow. Cause and Effect!</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "27072",
      "postDate": "07/12/2013 15:44:59",
      "content": "<p>As explained before, to get more robust results and benefit from the symmetry of the problem, we proposed that the score would average 2 AUCs. Kaggle kindly implemented this special score for us. Because it is not one of the standard Kaggle scores, it requires\r\n an intervention of the Kaggle admins. Due to the changes occurring around July 1 (the change in data and/or in submission method) the score might have been accidentally reverted to the regular AUC. I alerted Kaggle support and they will fix this as soon as\r\n possible. Thank you for your patience.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "27096",
      "postDate": "07/12/2013 20:46:23",
      "content": "<p>James, thanks for pointing this out. You were correct - when our scoring system was updated, the metric for this competition reverted to a normal (unidirectional) AUC.</p>\r\n<p>This competition is now being scored with the correct metric (bidirectional AUC) and all of the affected submissions have been rescored.</p>\r\n<p>Thanks for your patience, and good luck in this competition!</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "27098",
      "postDate": "07/12/2013 20:47:46",
      "content": "<p>Hi, Are your sure the new ones are correct? I was expecting about 68. (OK -mine haven't been rescored yet)</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "27101",
      "postDate": "07/12/2013 21:13:28",
      "content": "<p>Looks great!</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "27138",
      "postDate": "07/13/2013 16:09:39",
      "content": "<p>All my submissions are still &quot;Pending&quot;. Do I have to resubmit all previous models or are you doing the rescoring by punch cards?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "27209",
      "postDate": "07/15/2013 04:50:20",
      "content": "<p>[quote=Domcastro;27138]</p>\r\n<p>All my submissions are still &quot;Pending&quot;. Do I have to resubmit all previous models or are you doing the rescoring by punch cards?</p>\r\n<p>[/quote]</p>\r\n<p>Does the problem persist? I see that you have successfully made submissions since then, but I did not hear back from the admins.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "27226",
      "postDate": "07/15/2013 11:34:03",
      "content": "<p>Hi Isabelle - yes the problem persists - none of my entries have been rescored. I think it will be quicker if I re-run them all. New submissions are being scored correctly.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "27242",
      "postDate": "07/15/2013 17:08:04",
      "content": "<p>Thanks - all fixed now</p>",
      "rawMarkdown": "",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 27012,
      "author_name": "bcragin",
      "author_url": "",
      "post_date": "07/11/2013 15:37:35",
      "content": "<p>James, I think you have a very good question there!&nbsp;</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 27013,
      "author_name": "domcastro",
      "author_url": "",
      "post_date": "07/11/2013 15:51:27",
      "content": "<p>It's just a bad evaluation metric. So, just carry on - the competition is ruined once again. This was an observation that didn't need to be shared</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 27016,
      "author_name": "firfirikos",
      "author_url": "",
      "post_date": "07/11/2013 16:13:29",
      "content": "<p>Well, this is quite serious. Any comment from the organisers ?&nbsp;</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 27017,
      "author_name": "sitmocom",
      "author_url": "",
      "post_date": "07/11/2013 16:14:33",
      "content": "<p>I think it's great that James shared it. It's a flaw that needs fixing otherwise we are not doing science.</p>\r\n<p></p>\r\n<p>There is indeed a bug, I tried it myself -not bothering to forecast the B-&gt;A cases- and our score jumped up. This is not fair to the other contestants who are not aware of the bug, so making it public is the correct thing to do.</p>\r\n<p></p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 27019,
      "author_name": "kinnskogr",
      "author_url": "",
      "post_date": "07/11/2013 16:37:54",
      "content": "<p>Couldn't agree more. Kudos to James for sticking his neck out. Hopefully we'll hear from the admins soon.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 27022,
      "author_name": "domcastro",
      "author_url": "",
      "post_date": "07/11/2013 18:46:55",
      "content": "<p>It's not a bug - it's bad choice of evaluation metric</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 27023,
      "author_name": "iguyon",
      "author_url": "",
      "post_date": "07/11/2013 18:48:20",
      "content": "<p>Sorry for the delayed answer. I checked and indeed you seem to be right. The current score used is asymmetric. I am having Kaggle check into this. The error may have been introduced when the switch to the new validation set was made. Prior to that the symmetric\r\n score was used (as indicated on the website).</p>\r\n<p>Hopefully this will be rectified soon. Thank you for pointing out the error.</p>\r\n<p>Isabelle</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 27024,
      "author_name": "domcastro",
      "author_url": "",
      "post_date": "07/11/2013 18:48:53",
      "content": "<p>Are you sure? because I get the exact same in training</p>\r\n<p></p>\r\n<p>I get 86 AUC one way A-&gt;B and 82 AUC B-&gt;A in training so the score I got I was expecting</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 27025,
      "author_name": "sitmocom",
      "author_url": "",
      "post_date": "07/11/2013 18:59:35",
      "content": "<p>The evaluation metric could be chosen to be anything, it's part of the challenge. &nbsp;However the implementation of the described metric has indeed a bug: it does not adhere to the specs, it behaves different.</p>\r\n<p></p>\r\n<p>A lot of people must have been building symmetrical model, probably tweaking them accidentally into a direction that was preferring asymmetry. It would be really helpful if *all* historical submissions were re-evaluated with the fixed metric rather than\r\n resetting us all again to below 0.50. Do you think that's possible Isabella? It would provide us with much better feedback compared to a reset.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 27027,
      "author_name": "domcastro",
      "author_url": "",
      "post_date": "07/11/2013 19:00:34",
      "content": "<p>[quote=Sitmo;27017]</p>\r\n<p>I think it's great that James shared it. It's a flaw that needs fixing otherwise we are not doing science</p>\r\n<p></p>\r\n<p>[/quote]</p>\r\n<p></p>\r\n<p>I didn't realise we were doing science, I thought it was a Kaggle competition where many competitions have been won by de-anonymising and data leakage (take the recent KDD competition) The science of this whole academic field has been put to the test. The\r\n way that artificial data is created is flawed, the metric is deifnitely flawed (going by my training results, the leaderboard reflects this score) and there's no current &quot;state-of-the-art&quot; benchmark for the new data. I gave up on the science ages ago.</p>\r\n<p></p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 27028,
      "author_name": "domcastro",
      "author_url": "",
      "post_date": "07/11/2013 19:05:42",
      "content": "<p>[quote=Sitmo;27025]</p>\r\n<p>The evaluation metric could be chosen to be anything, it's part of the challenge. &nbsp;However the implementation of the described metric has indeed a bug: it does not adhere to the specs, it behaves different.</p>\r\n<p></p>\r\n<p>A lot of people must have been building symmetrical model, probably tweaking them accidentally into a direction that was preferring asymmetry. It would be really helpful if *all* historical submissions were re-evaluated with the fixed metric rather than\r\n resetting us all again to below 0.50. Do you think that's possible Isabella? It would provide us with much better feedback compared to a reset.</p>\r\n<p>[/quote]</p>\r\n<p></p>\r\n<p>Well, it must be an R bug then because as I've said I get over 80% AUC both ways in training (and none of my predictions are &lt; 0)</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 27030,
      "author_name": "domcastro",
      "author_url": "",
      "post_date": "07/11/2013 19:31:36",
      "content": "<p>I think it depends if you use the ROCR package or the VERIFICATION package (making -1s to 1s). On one I get 30 AUC and the other 80. It's how it deals with the -1. Sorry James - I just couldn't understand what you were saying because of the symmetry I was\r\n getting following the same model. </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 27035,
      "author_name": "iguyon",
      "author_url": "",
      "post_date": "07/11/2013 21:17:22",
      "content": "<p>For computing the AUC, the target values should be binary. Some packages use 0/1 for target values, others use -1/1.</p>\r\n<p>Let us call NEG and POS the two target values. Let us call Y the trinary targets of the challenge. You can transform Y into 2 target vectors Y1 and Y2:</p>\r\n<p>Y1 = POS if Y==1 and Y1 = NEG otherwise</p>\r\n<p>Y2 = NEG if Y==-1 and Y2 = POS otherwise</p>\r\n<p>Then compute 2 AUCS, one for Y1 and one for Y2, and average the results to obtain the score of the challenge.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 27062,
      "author_name": "alexxanderlarko",
      "author_url": "",
      "post_date": "07/12/2013 14:46:04",
      "content": "<p>What does the score on the leaderboard?</p>\r\n<p>A hilarious competition!&nbsp;</p>\r\n<p>Changing data, not understandable assessment.</p>\r\n<p>In general a lot of fun!</p>\r\n<p>Good luck to everyone!</p>\r\n<p></p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 27071,
      "author_name": "domcastro",
      "author_url": "",
      "post_date": "07/12/2013 15:32:17",
      "content": "<p>lol you just made me laugh. First time I've laughed about it since the &quot;troubles&quot;. lol Been a Kaggle member for years and this is the first time I've turned into an insane cow. Cause and Effect!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 27072,
      "author_name": "iguyon",
      "author_url": "",
      "post_date": "07/12/2013 15:44:59",
      "content": "<p>As explained before, to get more robust results and benefit from the symmetry of the problem, we proposed that the score would average 2 AUCs. Kaggle kindly implemented this special score for us. Because it is not one of the standard Kaggle scores, it requires\r\n an intervention of the Kaggle admins. Due to the changes occurring around July 1 (the change in data and/or in submission method) the score might have been accidentally reverted to the regular AUC. I alerted Kaggle support and they will fix this as soon as\r\n possible. Thank you for your patience.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 27096,
      "author_name": "benhamner",
      "author_url": "",
      "post_date": "07/12/2013 20:46:23",
      "content": "<p>James, thanks for pointing this out. You were correct - when our scoring system was updated, the metric for this competition reverted to a normal (unidirectional) AUC.</p>\r\n<p>This competition is now being scored with the correct metric (bidirectional AUC) and all of the affected submissions have been rescored.</p>\r\n<p>Thanks for your patience, and good luck in this competition!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 27098,
      "author_name": "domcastro",
      "author_url": "",
      "post_date": "07/12/2013 20:47:46",
      "content": "<p>Hi, Are your sure the new ones are correct? I was expecting about 68. (OK -mine haven't been rescored yet)</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 27101,
      "author_name": "sitmocom",
      "author_url": "",
      "post_date": "07/12/2013 21:13:28",
      "content": "<p>Looks great!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 27138,
      "author_name": "domcastro",
      "author_url": "",
      "post_date": "07/13/2013 16:09:39",
      "content": "<p>All my submissions are still &quot;Pending&quot;. Do I have to resubmit all previous models or are you doing the rescoring by punch cards?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 27209,
      "author_name": "iguyon",
      "author_url": "",
      "post_date": "07/15/2013 04:50:20",
      "content": "<p>[quote=Domcastro;27138]</p>\r\n<p>All my submissions are still &quot;Pending&quot;. Do I have to resubmit all previous models or are you doing the rescoring by punch cards?</p>\r\n<p>[/quote]</p>\r\n<p>Does the problem persist? I see that you have successfully made submissions since then, but I did not hear back from the admins.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 27226,
      "author_name": "domcastro",
      "author_url": "",
      "post_date": "07/15/2013 11:34:03",
      "content": "<p>Hi Isabelle - yes the problem persists - none of my entries have been rescored. I think it will be quicker if I re-run them all. New submissions are being scored correctly.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 27242,
      "author_name": "domcastro",
      "author_url": "",
      "post_date": "07/15/2013 17:08:04",
      "content": "<p>Thanks - all fixed now</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "26943": "",
    "27012": "",
    "27013": "",
    "27016": "",
    "27017": "",
    "27019": "",
    "27022": "",
    "27023": "",
    "27024": "",
    "27025": "",
    "27027": "",
    "27028": "",
    "27030": "",
    "27035": "",
    "27062": "",
    "27071": "",
    "27072": "",
    "27096": "",
    "27098": "",
    "27101": "",
    "27138": "",
    "27209": "",
    "27226": "",
    "27242": ""
  },
  "source": "meta"
}