You have until Friday May 17 to contribute your own pairs and enter the challenge in track 1, see: http://www.causality.inf.ethz.ch/cause-effect.php?page=tasks
\r\nIt is VERY important that a lot of people submit data, even artificial data. This will considerably strengthen the validity of the results of the challenge by lowering the risk that the algorithms proposed are capitalizing on artifacts of the proposed dataset.\r\n We have already become aware of a few artifacts that will be corrected in test data, such as uneven distributions of number of values across classes, which will be corrected in the test data. But it is very difficult to assertain that the data will be completely\r\n devoid of artifacts.
\r\nSo, we count on you to provide data. You may contribute:
\r\n- real data with known causal relationships
\r\n- real data with unknown causal relationships (hence there is a potential for making new interesting scientific discoveries if there is a consensus between methods)
\r\n- artificial data.
\r\nThere is no mimimum number of pairs nor minimum number for samples, but we recommend submitting at least 10 pairs and 100 samples per pair. The data you provide will not be used for the final ranking, but, we will publish the results of the best ranking\r\n methods on those data for comparison. And ...
\r\nYou have until Friday May 17 to contribute your own pairs and enter the challenge in track 1, see: http://www.causality.inf.ethz.ch/cause-effect.php?page=tasks
\r\nIt is VERY important that a lot of people submit data, even artificial data. This will considerably strengthen the validity of the results of the challenge by lowering the risk that the algorithms proposed are capitalizing on artifacts of the proposed dataset.\r\n We have already become aware of a few artifacts that will be corrected in test data, such as uneven distributions of number of values across classes, which will be corrected in the test data. But it is very difficult to assertain that the data will be completely\r\n devoid of artifacts.
\r\nSo, we count on you to provide data. You may contribute:
\r\n- real data with known causal relationships
\r\n- real data with unknown causal relationships (hence there is a potential for making new interesting scientific discoveries if there is a consensus between methods)
\r\n- artificial data.
\r\nThere is no mimimum number of pairs nor minimum number for samples, but we recommend submitting at least 10 pairs and 100 samples per pair. The data you provide will not be used for the final ranking, but, we will publish the results of the best ranking\r\n methods on those data for comparison. And ...
\r\nYou shouldn't really change the test data class distributions - the valid test set would have been selected based on these distributions. This means how a model performs on the valid set will now be different from the test set! Not really fair, as you only\r\n have one shot on the test data (and can't see results).
\r\n\r\n
", "rawMarkdown": "", "votes": null }, { "id": "23985", "postDate": "05/07/2013 14:59:38", "content": "
Isabelle,
\r\n1) are there any restrictions on the data we can submit? More specifically,
\r\n*for artificial data, should the data includes only two variables in all cases? Is there any restriction on the kind of functions?
\r\n*for real data for which we declare a known causal relationship B>A, should we be able to exclude the existence of causal relationship A>B (for example say A=age and B=life expectancy), feedback, time series, third variable, or anything else?
\r\n*for real data with unknown causal relationship, any constraints?
\r\n2) how will you determine the winner for the track 1?
\r\nThanks,
", "rawMarkdown": "", "votes": null }, { "id": "24052", "postDate": "05/09/2013 05:29:04", "content": "[quote=Jay;23985]
\r\nIsabelle,
\r\n1) are there any restrictions on the data we can submit? More specifically,
\r\n*for artificial data, should the data includes only two variables in all cases? Is there any restriction on the kind of functions?
\r\n*for real data for which we declare a known causal relationship B>A, should we be able to exclude the existence of causal relationship A>B (for example say A=age and B=life expectancy), feedback, time series, third variable, or anything else?
\r\n*for real data with unknown causal relationship, any constraints?
\r\n2) how will you determine the winner for the track 1?
\r\nThanks,
\r\n[/quote]
\r\n1) The data should contain pairs of variables that can be numerical, categorical or binary, like those of the challenge. Also exclude feed-back and time series to keep the same setting as in the challenge. If you have both A->B and A<-Z->B, this will be\r\n considered a case of A->B (A is a cause of B, not necessarily the only cause). The rationale is that we want to find cases in which A influences B. But for for cases labeled A-B, make sure they correspond to A<-Z->B, excluding any possibility of A->B or A<-B.
\r\nThere is no restriction on the kind of function or noise, but be aware that linear functions are among the most difficult cases and many non-liner functions have almost linear ranges sometimes, so you don't want too many of those because this will be redundant.\r\n The case linear function+Gaussian noise+Gaussian input is non identifiable. Non identifiability is discussed in some of the\r\nsuggested readings.
\r\nFor real data with unknown causal relationship, the target value should be 0 in the target field (taking values-1/0/1) and 0 in the "detailed field" (taking other values 1,2,3,4). Preferably include data with a\r\nstatistically significant dependency between A and B because it is very easy to generate independent variables and not so interesting. This is true for both known and unknown causal relationships.
\r\n2) The participants of track 1 will get the results of the participants on their data and will have to produce a paper explaining the design of their dataset and providing an analysis of the findings. The papers will be peer reviewed and judged by a committee.\r\n See review criteria. There will be a best paper award and a best student paper award. Other accepted papers will also be published. The authors of the best papers will be invited\r\n to write a longer paper in a special topic of JMLR on experimental design.
\r\nThere are many ways in which your contribution can be important:
\r\n- You can propose data from an application of real scientific interest. Maybe you have both observational and experimental data and want to see whether the cause-effect pair methods can uncover the causal relationships experimetally validated with the observational\r\n data only. Maybe you have unvalidated pairs, but you plan to conduct experiments to validate the findings of the challenge.
\r\n- You can propose artificial data that points to limitations of the cause-effect pair algorithms for well defined reasons.
\r\n- you can propose artificial data covering systematically an array of cases that would provide an alternative valuable benchmark to that of the challenge.
\r\nThank you in advance for your contribution.
", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 23978, "author_name": "domcastro", "author_url": "", "post_date": "05/07/2013 09:42:40", "content": "You shouldn't really change the test data class distributions - the valid test set would have been selected based on these distributions. This means how a model performs on the valid set will now be different from the test set! Not really fair, as you only\r\n have one shot on the test data (and can't see results).
\r\n\r\n
", "votes": null, "replies": [] }, { "id": 23985, "author_name": "jay670790", "author_url": "", "post_date": "05/07/2013 14:59:38", "content": "
Isabelle,
\r\n1) are there any restrictions on the data we can submit? More specifically,
\r\n*for artificial data, should the data includes only two variables in all cases? Is there any restriction on the kind of functions?
\r\n*for real data for which we declare a known causal relationship B>A, should we be able to exclude the existence of causal relationship A>B (for example say A=age and B=life expectancy), feedback, time series, third variable, or anything else?
\r\n*for real data with unknown causal relationship, any constraints?
\r\n2) how will you determine the winner for the track 1?
\r\nThanks,
", "votes": null, "replies": [] }, { "id": 24052, "author_name": "iguyon", "author_url": "", "post_date": "05/09/2013 05:29:04", "content": "[quote=Jay;23985]
\r\nIsabelle,
\r\n1) are there any restrictions on the data we can submit? More specifically,
\r\n*for artificial data, should the data includes only two variables in all cases? Is there any restriction on the kind of functions?
\r\n*for real data for which we declare a known causal relationship B>A, should we be able to exclude the existence of causal relationship A>B (for example say A=age and B=life expectancy), feedback, time series, third variable, or anything else?
\r\n*for real data with unknown causal relationship, any constraints?
\r\n2) how will you determine the winner for the track 1?
\r\nThanks,
\r\n[/quote]
\r\n1) The data should contain pairs of variables that can be numerical, categorical or binary, like those of the challenge. Also exclude feed-back and time series to keep the same setting as in the challenge. If you have both A->B and A<-Z->B, this will be\r\n considered a case of A->B (A is a cause of B, not necessarily the only cause). The rationale is that we want to find cases in which A influences B. But for for cases labeled A-B, make sure they correspond to A<-Z->B, excluding any possibility of A->B or A<-B.
\r\nThere is no restriction on the kind of function or noise, but be aware that linear functions are among the most difficult cases and many non-liner functions have almost linear ranges sometimes, so you don't want too many of those because this will be redundant.\r\n The case linear function+Gaussian noise+Gaussian input is non identifiable. Non identifiability is discussed in some of the\r\nsuggested readings.
\r\nFor real data with unknown causal relationship, the target value should be 0 in the target field (taking values-1/0/1) and 0 in the "detailed field" (taking other values 1,2,3,4). Preferably include data with a\r\nstatistically significant dependency between A and B because it is very easy to generate independent variables and not so interesting. This is true for both known and unknown causal relationships.
\r\n2) The participants of track 1 will get the results of the participants on their data and will have to produce a paper explaining the design of their dataset and providing an analysis of the findings. The papers will be peer reviewed and judged by a committee.\r\n See review criteria. There will be a best paper award and a best student paper award. Other accepted papers will also be published. The authors of the best papers will be invited\r\n to write a longer paper in a special topic of JMLR on experimental design.
\r\nThere are many ways in which your contribution can be important:
\r\n- You can propose data from an application of real scientific interest. Maybe you have both observational and experimental data and want to see whether the cause-effect pair methods can uncover the causal relationships experimetally validated with the observational\r\n data only. Maybe you have unvalidated pairs, but you plan to conduct experiments to validate the findings of the challenge.
\r\n- You can propose artificial data that points to limitations of the cause-effect pair algorithms for well defined reasons.
\r\n- you can propose artificial data covering systematically an array of cases that would provide an alternative valuable benchmark to that of the challenge.
\r\nThank you in advance for your contribution.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "23969": "", "23978": "", "23985": "", "24052": "" }, "source": "meta" }