{
  "id": 4449,
  "title": "About the rules",
  "url": "/competitions/cause-effect-pairs/discussion/4449",
  "author_name": "",
  "post_date": "2013-04-26T18:27:23.737Z",
  "votes": 1,
  "comment_count": 14,
  "views": 3875,
  "content": "<div>Dear Isabelle and other organizers,</div>\n<div>I find the rules of this contest slightly confusing. It seems hard to determine what exactly has to be submitted in order to compete towards the prizes. I'm copying some bits of the rules here, along with a list of questions:</div>\n<div>Part of the rules are on the Kaggle rules page:</div>\n<div><a href=\"http://www.kaggle.com/c/cause-effect-pairs/details/rules\" target=\"_blank\">http://www.kaggle.com/c/cause-effect-pairs/details/rules</a></div>\n<div>Here it&nbsp;says: \"Prizes will be awarded after the winners have posted their solutions to the competition forum. Winners must post or link to their solutions with seven days of the final competition deadline.\"&nbsp;</div>\n<div>1. I assume the final competition deadline is July 15th, as mentioned on Kaggle, and so the solutions have to be posted between July 8th and July 15th. Can you confirm this?</div>\n<div>2. Since you write about posting or linking to the solution, I assume the \"solution\" refers to software and not a to CSV submission on the validation data. In this case, what programming language is accepted? Some specifications would be nice. Or do you only require a high-level description at this point?</div>\n<div>3. It seems that the participants are required to submit their solution before the winner is announced. Is this true?</div>\n<div>Then there's another part on the Kaggle prizes page:</div>\n<div><a href=\"http://www.kaggle.com/c/cause-effect-pairs/details/prizes\" target=\"_blank\">http://www.kaggle.com/c/cause-effect-pairs/details/prizes</a></div>\n<div>Here it says \"to compete for the prizes, the participants must submit their software for verification purpose\".&nbsp;</div>\n<div>4. What does \"verification purpose\" mean? Does that mean that the organizers check that the submitted software outputs an earlier submitted winning label set for the validation data? Please explain.</div>\n<div>And then there's a third part on the external \"other rules\" page:</div>\n<div><a href=\"http://www.causality.inf.ethz.ch/cause-effect.php?page=rules\" target=\"_blank\">http://www.causality.inf.ethz.ch/cause-effect.php?page=rules</a></div>\n<div>Here it says something that is not mentioned on Kaggle's web site:&nbsp;<span style=\"line-height: 1.4em\">\"To compete towards the prizes, the participants must submit a 6-page paper describing their donated dataset and the findings derived from the challenge results (if they entered as a donor, track 1), or their task-solving method(s) and result(s) (if they entered as a solver, track 2), before the submission deadline (see schedule).\"</span></div>\n<div>5. Can you confirm that participants are required to write a 6-page paper to be eligible for the prizes?</div>\n<div>6. Does this paper need to fulfill some quality standards, such as having to pass a peer-review? If the rules don't specify this clearly then it seems any bogus paper is allowed.</div>\n<p>Also, according to the schedule on this page, the test data is released on July 19th and the results on the test data must be in by July 26th.</p>\n<div>7. Does this mean that the participants are required to run their software, which they have posted before on the site (see question 1), on the newly made available test data as-is? Or is it allowed to adjust the solution w.r.t. to what has been posted before?<br>\n<div>I hope you can shed some light on these issues. I believe more participants may have the same doubts, or will so as the deadline approaches.</div>\n<div>Thanks</div>\n</div>",
  "messages": [
    {
      "id": "23533",
      "postDate": "04/26/2013 18:27:23",
      "content": "<div>Dear Isabelle and other organizers,</div>\n<div>I find the rules of this contest slightly confusing. It seems hard to determine what exactly has to be submitted in order to compete towards the prizes. I'm copying some bits of the rules here, along with a list of questions:</div>\n<div>Part of the rules are on the Kaggle rules page:</div>\n<div><a href=\"http://www.kaggle.com/c/cause-effect-pairs/details/rules\" target=\"_blank\">http://www.kaggle.com/c/cause-effect-pairs/details/rules</a></div>\n<div>Here it&nbsp;says: \"Prizes will be awarded after the winners have posted their solutions to the competition forum. Winners must post or link to their solutions with seven days of the final competition deadline.\"&nbsp;</div>\n<div>1. I assume the final competition deadline is July 15th, as mentioned on Kaggle, and so the solutions have to be posted between July 8th and July 15th. Can you confirm this?</div>\n<div>2. Since you write about posting or linking to the solution, I assume the \"solution\" refers to software and not a to CSV submission on the validation data. In this case, what programming language is accepted? Some specifications would be nice. Or do you only require a high-level description at this point?</div>\n<div>3. It seems that the participants are required to submit their solution before the winner is announced. Is this true?</div>\n<div>Then there's another part on the Kaggle prizes page:</div>\n<div><a href=\"http://www.kaggle.com/c/cause-effect-pairs/details/prizes\" target=\"_blank\">http://www.kaggle.com/c/cause-effect-pairs/details/prizes</a></div>\n<div>Here it says \"to compete for the prizes, the participants must submit their software for verification purpose\".&nbsp;</div>\n<div>4. What does \"verification purpose\" mean? Does that mean that the organizers check that the submitted software outputs an earlier submitted winning label set for the validation data? Please explain.</div>\n<div>And then there's a third part on the external \"other rules\" page:</div>\n<div><a href=\"http://www.causality.inf.ethz.ch/cause-effect.php?page=rules\" target=\"_blank\">http://www.causality.inf.ethz.ch/cause-effect.php?page=rules</a></div>\n<div>Here it says something that is not mentioned on Kaggle's web site:&nbsp;<span style=\"line-height: 1.4em\">\"To compete towards the prizes, the participants must submit a 6-page paper describing their donated dataset and the findings derived from the challenge results (if they entered as a donor, track 1), or their task-solving method(s) and result(s) (if they entered as a solver, track 2), before the submission deadline (see schedule).\"</span></div>\n<div>5. Can you confirm that participants are required to write a 6-page paper to be eligible for the prizes?</div>\n<div>6. Does this paper need to fulfill some quality standards, such as having to pass a peer-review? If the rules don't specify this clearly then it seems any bogus paper is allowed.</div>\n<p>Also, according to the schedule on this page, the test data is released on July 19th and the results on the test data must be in by July 26th.</p>\n<div>7. Does this mean that the participants are required to run their software, which they have posted before on the site (see question 1), on the newly made available test data as-is? Or is it allowed to adjust the solution w.r.t. to what has been posted before?<br>\n<div>I hope you can shed some light on these issues. I believe more participants may have the same doubts, or will so as the deadline approaches.</div>\n<div>Thanks</div>\n</div>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "23654",
      "postDate": "04/29/2013 19:38:16",
      "content": "<div>\r\n<div>\r\n<blockquote>\r\n<div class=\"x_gmail_extra\">\r\n<div class=\"x_gmail_quote\">\r\n<blockquote class=\"x_gmail_quote\">\r\n<div>\r\n<div>\r\n<blockquote>\r\n<div><em>Part of the rules are on the Kaggle rules page:</em></div>\r\n<div><em><a href=\"http://www.kaggle.com/c/cause-effect-pairs/details/rules\" target=\"_blank\">http://www.kaggle.com/c/cause-effect-pairs/details/rules</a></em></div>\r\n<div><em>Here it&nbsp;says: &quot;Prizes will be awarded after the winners have posted their solutions to the competition forum. Winners must post or link to their solutions with seven days of the final competition deadline.&quot;&nbsp;</em></div>\r\n<div><em>1. I assume the final competition deadline is July 15th, as mentioned on Kaggle, and so the solutions have to be posted between July 8th and July 15th. Can you confirm this?</em></div>\r\n</blockquote>\r\n<div><span><br>\r\n</span></div>\r\n<div><span>Yes, this is correct. The deadline for software submission is July 15. You will have to upload and link your software to the validation data submission that it produced in the week preceding the deadline. In this way, the organizers can check that\r\n your software works properly before releasing the test labels and eventually get back to you if there is any problem. Corrections of the software will not be possible after the test data are released.</span></div>\r\n</div>\r\n</div>\r\n</blockquote>\r\n</div>\r\n</div>\r\n</blockquote>\r\n<blockquote>\r\n<div class=\"x_gmail_extra\">\r\n<div class=\"x_gmail_quote\">\r\n<blockquote class=\"x_gmail_quote\">\r\n<div>\r\n<div>\r\n<div><span>T<strong>he software submitted will be used for verification purposes only, unless you win. The three winners will be eligible for prizes only if they make their source code available within a week of the test set submission results, that is no later\r\n than August 2.</strong></span></div>\r\n<span><span><br>\r\n</span></span>\r\n<blockquote>\r\n<div><em>2. Since you write about posting or linking to the solution, I assume the &quot;solution&quot; refers to software and not a to CSV submission on the validation data. In this case, what programming language is accepted? Some specifications would be nice. Or do\r\n you only require a high-level description at this point?</em></div>\r\n</blockquote>\r\n<div><span><br>\r\n</span></div>\r\n<div>Recommendations on how to submit software are found at the bottom of the page:&nbsp;<a href=\"http://www.causality.inf.ethz.ch/cause-effect.php?page=tasks\" target=\"_blank\">http://www.causality.inf.ethz.ch/cause-effect.php?page=tasks</a>.</div>\r\n<div>The documentation you provide should be clear so the organizers can easily reproduce the results.</div>\r\n<div>You will have to upload the results on the test data between July 19 and 26 and link your software to the test data results.</div>\r\n<span><span><br>\r\n</span></span>\r\n<blockquote>\r\n<div><em>3. It seems that the participants are required to submit their solution before the winner is announced. Is this true?</em></div>\r\n</blockquote>\r\n<span>Yes, both software and test data results must be submitted by the participants and verified by the organizers before results are announced.</span></div>\r\n<div><span><span><br>\r\n</span></span>\r\n<blockquote>\r\n<div><em>Then there's another part on the Kaggle prizes page:</em></div>\r\n<div><em><a href=\"http://www.kaggle.com/c/cause-effect-pairs/details/prizes\" target=\"_blank\">http://www.kaggle.com/c/cause-effect-pairs/details/prizes</a></em></div>\r\n<div><em>Here it says &quot;to compete for the prizes, the participants must submit their software for verification purpose&quot;.&nbsp;</em></div>\r\n<div><em>4. What does &quot;verification purpose&quot; mean? Does that mean that the organizers check that the submitted software outputs an earlier submitted winning label set for the validation data? Please explain.</em></div>\r\n</blockquote>\r\n<div><span><br>\r\n</span></div>\r\n<div><span>The software submitted must produce the same predictions than those submitted on test data. To that end, we recommend to eliminate any stochastic component from the code and have random number generators initialized with the same seed at each run.</span></div>\r\n<span><span><br>\r\n</span></span>\r\n<blockquote>\r\n<div><em>And then there's a third part on the external &quot;other rules&quot; page:</em></div>\r\n<div><em><a href=\"http://www.causality.inf.ethz.ch/cause-effect.php?page=rules\" target=\"_blank\">http://www.causality.inf.ethz.ch/cause-effect.php?page=rules</a></em></div>\r\n<div><em>Here it says something that is not mentioned on Kaggle's web site:&nbsp;&quot;To compete towards the prizes, the participants must submit a 6-page paper describing their donated dataset and the findings derived from the challenge results (if they entered as\r\n a donor, track 1), or their task-solving method(s) and result(s) (if they entered as a solver, track 2), before the submission deadline (see schedule).&quot;</em></div>\r\n<div><em>5. Can you confirm that participants are required to write a 6-page paper to be eligible for the prizes?</em></div>\r\n</blockquote>\r\n<div><span><br>\r\n</span></div>\r\n<div><span>Yes, this is correct.</span></div>\r\n<span><span><br>\r\n</span></span>\r\n<blockquote>\r\n<div><em>6. Does this paper need to fulfill some quality standards, such as having to pass a peer-review? If the rules don't specify this clearly then it seems any bogus paper is allowed.</em></div>\r\n</blockquote>\r\n<div><span><br>\r\n</span></div>\r\n<span>The paper will be peer reviewed. Details on review criteria are found at&nbsp;<a href=\"http://www.causality.inf.ethz.ch/cause-effect.php?page=rewards\" target=\"_blank\">http://www.causality.inf.ethz.ch/cause-effect.php?page=rewards</a>.</span></div>\r\n<div><span><span>For all papers, the reviewers will use five criteria: (1)&nbsp;<strong>Impact</strong>&nbsp;(track 1) or&nbsp;<strong>Performance</strong>&nbsp;in challenge (track 1), (2)&nbsp;<strong>Novelty/Originality</strong>, (3)&nbsp;<strong>Sanity</strong>, (4)&nbsp;<strong>Insight</strong>,\r\n and (5)&nbsp;<strong>Clarity</strong>&nbsp;of presentation. In track 1 (data donation), &quot;Impact&quot; means the scientific, technical, and/or societal impact of the cause-effect relationships discovered through the challenge (preferably validated a posteriori). In track\r\n 2 (submission of methods), &quot;Performance&quot; means verified final challenge performance results on test data.</span></span></div>\r\n<div><span style=\"font-family:verdana,sans-serif\"><span><br>\r\n</span></span></div>\r\n<div><span style=\"font-family:verdana,sans-serif\"><span>The papers will need to be accepted to qualify for prizes.<br>\r\n</span></span>\r\n<blockquote>\r\n<p><em>Also, according to the schedule on this page, the test data is released on July 19th and the results on the test data must be in by July 26th.</em></p>\r\n<div><em>7. Does this mean that the participants are required to run their software, which they have posted before on the site (see question 1), on the newly made available test data as-is? Or is it allowed to adjust the solution w.r.t. to what has been posted\r\n before?</em></div>\r\n</blockquote>\r\n<div><span><br>\r\n</span></div>\r\n<span style=\"font-family:Arial,Helvetica,sans-serif; font-size:medium\"><span>The software must be run &quot;as is&quot; no adjustment should be made. The organizers will attempt to reproduce the test data predictions produced by the participants using their submitted\r\n software.</span></span></div>\r\n</div>\r\n</blockquote>\r\n</div>\r\n</div>\r\n</blockquote>\r\n</div>\r\n</div>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "23670",
      "postDate": "04/29/2013 21:58:57",
      "content": "<p>&quot;<span style=\"font-family:verdana,sans-serif\"><span>The papers will need to be accepted to qualify for prizes.&quot;</span></span></p>\r\n<p>&nbsp;</p>\r\n<p><span style=\"font-family:verdana,sans-serif\"><span>It will be difficult for a non-expert in the domain to get a paper accepted in the domain</span></span></p>\r\n<p>&nbsp;</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "23674",
      "postDate": "04/29/2013 22:36:47",
      "content": "<p>This will be &quot;conference quality&quot; papers, not &quot;journal quality&quot; papers. We will also provide the opportunity to write longer journal papers.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "23675",
      "postDate": "04/29/2013 23:03:53",
      "content": "<p>I have a few questions on this, too:</p>\r\n<p>1) Can we use R? &nbsp;(It says we have to ask...)</p>\r\n<p>2) Are you going to verify that you can retrain the models and predict the results in a fixed timeframe, or only predict using a pre-trained model?</p>\r\n<p>3) If you are retraining everything, do we have a time buget for feature creation and training?</p>\r\n<p>Thanks!</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "23679",
      "postDate": "04/30/2013 01:00:14",
      "content": "<p>Acceptance of papers to qualify for prizes is a fairly tough requirement. It is quite likely that many contestants do not have any experience writing a paper or the time to do it.</p>\r\n<p>I would request you to change it to just submitting a paper with detailed methodology. Rules are rules, but this requirement is not at all related to the contest.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "23680",
      "postDate": "04/30/2013 02:19:25",
      "content": "<p>The rules for this competition are the messiest of all the competitions I have been interested in.&nbsp; The framework setting up the competition&nbsp;is also messy, some being outside the Kaggle site.&nbsp; Early contributions to the forum by the organisers contained\r\n confusing typos.&nbsp; Altogether, in my opinion, it has not been up to Kaggle standard.</p>\r\n<p>Nevertheless it is a very interesting problem.&nbsp; I don't expect to score high enough to have to submit my solution but, since I am using 'fossil' desktop languages to do the computing, I doubt that the organisers would be able to run&nbsp;my programs anyway.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "23960",
      "postDate": "05/06/2013 22:31:55",
      "content": "<p>[quote=Sankalp;23679]</p>\r\n<p>Acceptance of papers to qualify for prizes is a fairly tough requirement. It is quite likely that many contestants do not have any experience writing a paper or the time to do it.</p>\r\n<p>I would request you to change it to just submitting a paper with detailed methodology. Rules are rules, but this requirement is not at all related to the contest.</p>\r\n<p>[/quote]</p>\r\n<p>We are very attached to that rule. In fact, we initially thought of only giving prizes for best paper awards. This challenge is about developing a new &quot;revolutionary&quot; technology. If this works, you are going to make history. And it looks like it will. I\r\n took a peek at the results and they are good both on real and artificial data.</p>\r\n<p>However, while we really want people to write a paper, we do not want to eliminate them because they have difficulties putting their &nbsp;work in writing. You will have the opportunity to revise your paper to the satisfaction of the reviewers.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "23961",
      "postDate": "05/06/2013 22:33:24",
      "content": "<p>[quote=JC36;23680]</p>\r\n<p>The rules for this competition are the messiest of all the competitions I have been interested in.&nbsp; The framework setting up the competition&nbsp;is also messy, some being outside the Kaggle site.&nbsp; Early contributions to the forum by the organisers contained\r\n confusing typos.&nbsp; Altogether, in my opinion, it has not been up to Kaggle standard.</p>\r\n<p>Nevertheless it is a very interesting problem.&nbsp; I don't expect to score high enough to have to submit my solution but, since I am using 'fossil' desktop languages to do the computing, I doubt that the organisers would be able to run&nbsp;my programs anyway.</p>\r\n<p>[/quote]</p>\r\n<p>I created a page with frequently asked questions&nbsp;http://www.causality.inf.ethz.ch/cause-effect.php?page=FAQ, let me know whether there is still anything unclear.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "24164",
      "postDate": "05/10/2013 21:33:04",
      "content": "<p>Isabelle,</p>\r\n<p>I'm still confused by the rules concerning the paper for track 2. (this might be caused by my poor English, it is at least correlated...)</p>\r\n<p>The&nbsp;<a href=\"http://www.causality.inf.ethz.ch/cause-effect.php?page=FAQ\">FAQ</a>&nbsp;say that you only need to submit a paper if you want to claim your prize.</p>\r\n<blockquote>\r\n<p><em>[quote]If you win and wish to accept your prize, you must make your code publicly available under a GPLv3 license and you must submit a 6-page paper to the proceedings.[/quote]</em></p>\r\n</blockquote>\r\n<p>and&nbsp;</p>\r\n<blockquote>\r\n<p><em><em>[quote]</em>The winners will have to produce a 6-page paper for the proceedings.<em>[/quote]</em></em></p>\r\n</blockquote>\r\n<p>Also, the <a href=\"http://www.causality.inf.ethz.ch/cause-effect.php?page=rules\">\r\nrules</a> say that we can compete anonymously by not submitting a paper.&nbsp;</p>\r\n<blockquote>\r\n<p><em><em>[quote]</em>The participants who do not submit a paper to the workshop can elect to remain anonymous<em>[/quote]</em></em></p>\r\n</blockquote>\r\n<p>On the other hand, the rules also state that we even cannot compete if we don't submit a paper.</p>\r\n<blockquote>\r\n<p><em><em>[quote]</em>To compete towards the prizes, the participants must submit a 6-page paper describing their donated dataset and the findings derived from the challenge results (if they entered as a donor, track 1), or their task-solving method(s) and\r\n result(s) (if they entered as a solver, track 2)<em>[/quote]</em></em></p>\r\n</blockquote>\r\n<p>Which one is correct? Can we wait to submit a paper until the winners have been selected?</p>\r\n<p>Thank you.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "24169",
      "postDate": "05/11/2013 00:37:34",
      "content": "<p>Yes, there is no obligation to publish a paper if you do not what to claim your prize and you can wait until the results of the challenge are announced. But of course you can submit a paper even if you are not a winner. There is a best paper award that gives\r\n you another chance.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "25383",
      "postDate": "06/11/2013 12:14:34",
      "content": "<p>Dear Isabelle, there's something equivocal about how the &quot;coefficient of causation&quot; is defined.</p>\r\n<p>Would't have been better if the values had ranged between -1 and 1, thereby easly interpretable as &quot;probability&quot; or &quot;level of confidence&quot;?</p>\r\n<p>Given I can produce an algorithm that calculates, say, a level of confidence of 75% that A causes B in a given dataset, how can I translate this in a coefficient of causation ranging from -Inf to &#43;Inf? If I'm sure that A-&gt;B, then I'll set the coefficient\r\n to &#43;Inf; if I'm sure that A|B, the I'll set it to 0; but what about the intermediate cases? Is 75% nearer to &#43;Inf than to zero (for whatever that means)? How can I translate this in a number?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "25386",
      "postDate": "06/11/2013 14:19:59",
      "content": "<p>I agree that it would be nice to be able to interpret the &quot;coefficient of causation&quot; as a probability. However, there are many constraints and difficulties in this challenge and I did not want to add another one. If a method returns normalized values, it\r\n is great. Otherwise, it is always possible to post-process the values to make the look like probabilities, in a sense that can be defined later.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "25387",
      "postDate": "06/11/2013 14:52:36",
      "content": "<p>Thank you Isabelle.</p>\r\n<p>So, if I've correctely understood, my method would be accepted if it returns normalized values rangin from -1 to 1 and this thing is specified in the method's description.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "25389",
      "postDate": "06/11/2013 15:05:24",
      "content": "<p>Absolutely, between -1 and 1 is fine.</p>",
      "rawMarkdown": "",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 23654,
      "author_name": "iguyon",
      "author_url": "",
      "post_date": "04/29/2013 19:38:16",
      "content": "<div>\r\n<div>\r\n<blockquote>\r\n<div class=\"x_gmail_extra\">\r\n<div class=\"x_gmail_quote\">\r\n<blockquote class=\"x_gmail_quote\">\r\n<div>\r\n<div>\r\n<blockquote>\r\n<div><em>Part of the rules are on the Kaggle rules page:</em></div>\r\n<div><em><a href=\"http://www.kaggle.com/c/cause-effect-pairs/details/rules\" target=\"_blank\">http://www.kaggle.com/c/cause-effect-pairs/details/rules</a></em></div>\r\n<div><em>Here it&nbsp;says: &quot;Prizes will be awarded after the winners have posted their solutions to the competition forum. Winners must post or link to their solutions with seven days of the final competition deadline.&quot;&nbsp;</em></div>\r\n<div><em>1. I assume the final competition deadline is July 15th, as mentioned on Kaggle, and so the solutions have to be posted between July 8th and July 15th. Can you confirm this?</em></div>\r\n</blockquote>\r\n<div><span><br>\r\n</span></div>\r\n<div><span>Yes, this is correct. The deadline for software submission is July 15. You will have to upload and link your software to the validation data submission that it produced in the week preceding the deadline. In this way, the organizers can check that\r\n your software works properly before releasing the test labels and eventually get back to you if there is any problem. Corrections of the software will not be possible after the test data are released.</span></div>\r\n</div>\r\n</div>\r\n</blockquote>\r\n</div>\r\n</div>\r\n</blockquote>\r\n<blockquote>\r\n<div class=\"x_gmail_extra\">\r\n<div class=\"x_gmail_quote\">\r\n<blockquote class=\"x_gmail_quote\">\r\n<div>\r\n<div>\r\n<div><span>T<strong>he software submitted will be used for verification purposes only, unless you win. The three winners will be eligible for prizes only if they make their source code available within a week of the test set submission results, that is no later\r\n than August 2.</strong></span></div>\r\n<span><span><br>\r\n</span></span>\r\n<blockquote>\r\n<div><em>2. Since you write about posting or linking to the solution, I assume the &quot;solution&quot; refers to software and not a to CSV submission on the validation data. In this case, what programming language is accepted? Some specifications would be nice. Or do\r\n you only require a high-level description at this point?</em></div>\r\n</blockquote>\r\n<div><span><br>\r\n</span></div>\r\n<div>Recommendations on how to submit software are found at the bottom of the page:&nbsp;<a href=\"http://www.causality.inf.ethz.ch/cause-effect.php?page=tasks\" target=\"_blank\">http://www.causality.inf.ethz.ch/cause-effect.php?page=tasks</a>.</div>\r\n<div>The documentation you provide should be clear so the organizers can easily reproduce the results.</div>\r\n<div>You will have to upload the results on the test data between July 19 and 26 and link your software to the test data results.</div>\r\n<span><span><br>\r\n</span></span>\r\n<blockquote>\r\n<div><em>3. It seems that the participants are required to submit their solution before the winner is announced. Is this true?</em></div>\r\n</blockquote>\r\n<span>Yes, both software and test data results must be submitted by the participants and verified by the organizers before results are announced.</span></div>\r\n<div><span><span><br>\r\n</span></span>\r\n<blockquote>\r\n<div><em>Then there's another part on the Kaggle prizes page:</em></div>\r\n<div><em><a href=\"http://www.kaggle.com/c/cause-effect-pairs/details/prizes\" target=\"_blank\">http://www.kaggle.com/c/cause-effect-pairs/details/prizes</a></em></div>\r\n<div><em>Here it says &quot;to compete for the prizes, the participants must submit their software for verification purpose&quot;.&nbsp;</em></div>\r\n<div><em>4. What does &quot;verification purpose&quot; mean? Does that mean that the organizers check that the submitted software outputs an earlier submitted winning label set for the validation data? Please explain.</em></div>\r\n</blockquote>\r\n<div><span><br>\r\n</span></div>\r\n<div><span>The software submitted must produce the same predictions than those submitted on test data. To that end, we recommend to eliminate any stochastic component from the code and have random number generators initialized with the same seed at each run.</span></div>\r\n<span><span><br>\r\n</span></span>\r\n<blockquote>\r\n<div><em>And then there's a third part on the external &quot;other rules&quot; page:</em></div>\r\n<div><em><a href=\"http://www.causality.inf.ethz.ch/cause-effect.php?page=rules\" target=\"_blank\">http://www.causality.inf.ethz.ch/cause-effect.php?page=rules</a></em></div>\r\n<div><em>Here it says something that is not mentioned on Kaggle's web site:&nbsp;&quot;To compete towards the prizes, the participants must submit a 6-page paper describing their donated dataset and the findings derived from the challenge results (if they entered as\r\n a donor, track 1), or their task-solving method(s) and result(s) (if they entered as a solver, track 2), before the submission deadline (see schedule).&quot;</em></div>\r\n<div><em>5. Can you confirm that participants are required to write a 6-page paper to be eligible for the prizes?</em></div>\r\n</blockquote>\r\n<div><span><br>\r\n</span></div>\r\n<div><span>Yes, this is correct.</span></div>\r\n<span><span><br>\r\n</span></span>\r\n<blockquote>\r\n<div><em>6. Does this paper need to fulfill some quality standards, such as having to pass a peer-review? If the rules don't specify this clearly then it seems any bogus paper is allowed.</em></div>\r\n</blockquote>\r\n<div><span><br>\r\n</span></div>\r\n<span>The paper will be peer reviewed. Details on review criteria are found at&nbsp;<a href=\"http://www.causality.inf.ethz.ch/cause-effect.php?page=rewards\" target=\"_blank\">http://www.causality.inf.ethz.ch/cause-effect.php?page=rewards</a>.</span></div>\r\n<div><span><span>For all papers, the reviewers will use five criteria: (1)&nbsp;<strong>Impact</strong>&nbsp;(track 1) or&nbsp;<strong>Performance</strong>&nbsp;in challenge (track 1), (2)&nbsp;<strong>Novelty/Originality</strong>, (3)&nbsp;<strong>Sanity</strong>, (4)&nbsp;<strong>Insight</strong>,\r\n and (5)&nbsp;<strong>Clarity</strong>&nbsp;of presentation. In track 1 (data donation), &quot;Impact&quot; means the scientific, technical, and/or societal impact of the cause-effect relationships discovered through the challenge (preferably validated a posteriori). In track\r\n 2 (submission of methods), &quot;Performance&quot; means verified final challenge performance results on test data.</span></span></div>\r\n<div><span style=\"font-family:verdana,sans-serif\"><span><br>\r\n</span></span></div>\r\n<div><span style=\"font-family:verdana,sans-serif\"><span>The papers will need to be accepted to qualify for prizes.<br>\r\n</span></span>\r\n<blockquote>\r\n<p><em>Also, according to the schedule on this page, the test data is released on July 19th and the results on the test data must be in by July 26th.</em></p>\r\n<div><em>7. Does this mean that the participants are required to run their software, which they have posted before on the site (see question 1), on the newly made available test data as-is? Or is it allowed to adjust the solution w.r.t. to what has been posted\r\n before?</em></div>\r\n</blockquote>\r\n<div><span><br>\r\n</span></div>\r\n<span style=\"font-family:Arial,Helvetica,sans-serif; font-size:medium\"><span>The software must be run &quot;as is&quot; no adjustment should be made. The organizers will attempt to reproduce the test data predictions produced by the participants using their submitted\r\n software.</span></span></div>\r\n</div>\r\n</blockquote>\r\n</div>\r\n</div>\r\n</blockquote>\r\n</div>\r\n</div>",
      "votes": null,
      "replies": []
    },
    {
      "id": 23670,
      "author_name": "domcastro",
      "author_url": "",
      "post_date": "04/29/2013 21:58:57",
      "content": "<p>&quot;<span style=\"font-family:verdana,sans-serif\"><span>The papers will need to be accepted to qualify for prizes.&quot;</span></span></p>\r\n<p>&nbsp;</p>\r\n<p><span style=\"font-family:verdana,sans-serif\"><span>It will be difficult for a non-expert in the domain to get a paper accepted in the domain</span></span></p>\r\n<p>&nbsp;</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 23674,
      "author_name": "iguyon",
      "author_url": "",
      "post_date": "04/29/2013 22:36:47",
      "content": "<p>This will be &quot;conference quality&quot; papers, not &quot;journal quality&quot; papers. We will also provide the opportunity to write longer journal papers.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 23675,
      "author_name": "edge219878",
      "author_url": "",
      "post_date": "04/29/2013 23:03:53",
      "content": "<p>I have a few questions on this, too:</p>\r\n<p>1) Can we use R? &nbsp;(It says we have to ask...)</p>\r\n<p>2) Are you going to verify that you can retrain the models and predict the results in a fixed timeframe, or only predict using a pre-trained model?</p>\r\n<p>3) If you are retraining everything, do we have a time buget for feature creation and training?</p>\r\n<p>Thanks!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 23679,
      "author_name": "sankalp",
      "author_url": "",
      "post_date": "04/30/2013 01:00:14",
      "content": "<p>Acceptance of papers to qualify for prizes is a fairly tough requirement. It is quite likely that many contestants do not have any experience writing a paper or the time to do it.</p>\r\n<p>I would request you to change it to just submitting a paper with detailed methodology. Rules are rules, but this requirement is not at all related to the contest.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 23680,
      "author_name": "jc3626439",
      "author_url": "",
      "post_date": "04/30/2013 02:19:25",
      "content": "<p>The rules for this competition are the messiest of all the competitions I have been interested in.&nbsp; The framework setting up the competition&nbsp;is also messy, some being outside the Kaggle site.&nbsp; Early contributions to the forum by the organisers contained\r\n confusing typos.&nbsp; Altogether, in my opinion, it has not been up to Kaggle standard.</p>\r\n<p>Nevertheless it is a very interesting problem.&nbsp; I don't expect to score high enough to have to submit my solution but, since I am using 'fossil' desktop languages to do the computing, I doubt that the organisers would be able to run&nbsp;my programs anyway.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 23960,
      "author_name": "iguyon",
      "author_url": "",
      "post_date": "05/06/2013 22:31:55",
      "content": "<p>[quote=Sankalp;23679]</p>\r\n<p>Acceptance of papers to qualify for prizes is a fairly tough requirement. It is quite likely that many contestants do not have any experience writing a paper or the time to do it.</p>\r\n<p>I would request you to change it to just submitting a paper with detailed methodology. Rules are rules, but this requirement is not at all related to the contest.</p>\r\n<p>[/quote]</p>\r\n<p>We are very attached to that rule. In fact, we initially thought of only giving prizes for best paper awards. This challenge is about developing a new &quot;revolutionary&quot; technology. If this works, you are going to make history. And it looks like it will. I\r\n took a peek at the results and they are good both on real and artificial data.</p>\r\n<p>However, while we really want people to write a paper, we do not want to eliminate them because they have difficulties putting their &nbsp;work in writing. You will have the opportunity to revise your paper to the satisfaction of the reviewers.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 23961,
      "author_name": "iguyon",
      "author_url": "",
      "post_date": "05/06/2013 22:33:24",
      "content": "<p>[quote=JC36;23680]</p>\r\n<p>The rules for this competition are the messiest of all the competitions I have been interested in.&nbsp; The framework setting up the competition&nbsp;is also messy, some being outside the Kaggle site.&nbsp; Early contributions to the forum by the organisers contained\r\n confusing typos.&nbsp; Altogether, in my opinion, it has not been up to Kaggle standard.</p>\r\n<p>Nevertheless it is a very interesting problem.&nbsp; I don't expect to score high enough to have to submit my solution but, since I am using 'fossil' desktop languages to do the computing, I doubt that the organisers would be able to run&nbsp;my programs anyway.</p>\r\n<p>[/quote]</p>\r\n<p>I created a page with frequently asked questions&nbsp;http://www.causality.inf.ethz.ch/cause-effect.php?page=FAQ, let me know whether there is still anything unclear.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 24164,
      "author_name": "bram278806",
      "author_url": "",
      "post_date": "05/10/2013 21:33:04",
      "content": "<p>Isabelle,</p>\r\n<p>I'm still confused by the rules concerning the paper for track 2. (this might be caused by my poor English, it is at least correlated...)</p>\r\n<p>The&nbsp;<a href=\"http://www.causality.inf.ethz.ch/cause-effect.php?page=FAQ\">FAQ</a>&nbsp;say that you only need to submit a paper if you want to claim your prize.</p>\r\n<blockquote>\r\n<p><em>[quote]If you win and wish to accept your prize, you must make your code publicly available under a GPLv3 license and you must submit a 6-page paper to the proceedings.[/quote]</em></p>\r\n</blockquote>\r\n<p>and&nbsp;</p>\r\n<blockquote>\r\n<p><em><em>[quote]</em>The winners will have to produce a 6-page paper for the proceedings.<em>[/quote]</em></em></p>\r\n</blockquote>\r\n<p>Also, the <a href=\"http://www.causality.inf.ethz.ch/cause-effect.php?page=rules\">\r\nrules</a> say that we can compete anonymously by not submitting a paper.&nbsp;</p>\r\n<blockquote>\r\n<p><em><em>[quote]</em>The participants who do not submit a paper to the workshop can elect to remain anonymous<em>[/quote]</em></em></p>\r\n</blockquote>\r\n<p>On the other hand, the rules also state that we even cannot compete if we don't submit a paper.</p>\r\n<blockquote>\r\n<p><em><em>[quote]</em>To compete towards the prizes, the participants must submit a 6-page paper describing their donated dataset and the findings derived from the challenge results (if they entered as a donor, track 1), or their task-solving method(s) and\r\n result(s) (if they entered as a solver, track 2)<em>[/quote]</em></em></p>\r\n</blockquote>\r\n<p>Which one is correct? Can we wait to submit a paper until the winners have been selected?</p>\r\n<p>Thank you.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 24169,
      "author_name": "iguyon",
      "author_url": "",
      "post_date": "05/11/2013 00:37:34",
      "content": "<p>Yes, there is no obligation to publish a paper if you do not what to claim your prize and you can wait until the results of the challenge are announced. But of course you can submit a paper even if you are not a winner. There is a best paper award that gives\r\n you another chance.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 25383,
      "author_name": "emilianobigozzi",
      "author_url": "",
      "post_date": "06/11/2013 12:14:34",
      "content": "<p>Dear Isabelle, there's something equivocal about how the &quot;coefficient of causation&quot; is defined.</p>\r\n<p>Would't have been better if the values had ranged between -1 and 1, thereby easly interpretable as &quot;probability&quot; or &quot;level of confidence&quot;?</p>\r\n<p>Given I can produce an algorithm that calculates, say, a level of confidence of 75% that A causes B in a given dataset, how can I translate this in a coefficient of causation ranging from -Inf to &#43;Inf? If I'm sure that A-&gt;B, then I'll set the coefficient\r\n to &#43;Inf; if I'm sure that A|B, the I'll set it to 0; but what about the intermediate cases? Is 75% nearer to &#43;Inf than to zero (for whatever that means)? How can I translate this in a number?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 25386,
      "author_name": "iguyon",
      "author_url": "",
      "post_date": "06/11/2013 14:19:59",
      "content": "<p>I agree that it would be nice to be able to interpret the &quot;coefficient of causation&quot; as a probability. However, there are many constraints and difficulties in this challenge and I did not want to add another one. If a method returns normalized values, it\r\n is great. Otherwise, it is always possible to post-process the values to make the look like probabilities, in a sense that can be defined later.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 25387,
      "author_name": "emilianobigozzi",
      "author_url": "",
      "post_date": "06/11/2013 14:52:36",
      "content": "<p>Thank you Isabelle.</p>\r\n<p>So, if I've correctely understood, my method would be accepted if it returns normalized values rangin from -1 to 1 and this thing is specified in the method's description.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 25389,
      "author_name": "iguyon",
      "author_url": "",
      "post_date": "06/11/2013 15:05:24",
      "content": "<p>Absolutely, between -1 and 1 is fine.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "23533": "",
    "23654": "",
    "23670": "",
    "23674": "",
    "23675": "",
    "23679": "",
    "23680": "",
    "23960": "",
    "23961": "",
    "24164": "",
    "24169": "",
    "25383": "",
    "25386": "",
    "25387": "",
    "25389": ""
  },
  "source": "meta"
}