Would it be possible to know more about the meaning of the predicted values and how the final score will be computed?
\r\nI am guessing :
\r\n- positive for A->B
\r\n- negative for A<-B
\r\n- zero for the other options?
\r\nAm I right?
\r\nThanks in advance
", "messages": [ { "id": "22080", "postDate": "04/03/2013 16:09:08", "content": "Would it be possible to know more about the meaning of the predicted values and how the final score will be computed?
\r\nI am guessing :
\r\n- positive for A->B
\r\n- negative for A<-B
\r\n- zero for the other options?
\r\nAm I right?
\r\nThanks in advance
", "rawMarkdown": "", "votes": null }, { "id": "22084", "postDate": "04/03/2013 16:19:31", "content": "For each pair of variables, we have a ternary truth value indicating whether A is a cause of B (+1), B is a cause of A (-1), or neither (0). We use the scores provided by the participants as a ranking criterion and evaluate their entries with two Area\r\n Under the ROC curve (AUC) scores:
\r\nLet Yhat be your predicted score in [-Inf, +Inf] and Y the target values in {-1, 0 1}.
\r\nWe define Y1=Y; Y1(Y==0)=-1; and Y2=Y; Y2(Y==0)=+1;
\r\nScore = 0.5*(AUC(Yhat, Y1)+AUC(Yhat, Y2));
The first score AUC(Yhat, Y1) measures the success at correctly detecting that A->B rather than [A<-B, A-B, or A|B].
\r\nThe second score AUC(Yhat, Y2) measures the success at correctly detecting that A<-B rather than [A->B, A-B, or A|B].
\r\nSince the problem is symmetric, we average the two scores.
\r\nThe organizers will also compute various other score for analysis purpose, but they will not be used to rank the participants.
I am confused about the meaning of Yhat.
\r\nFrom the competition notes: "We expect the participants to produce a score between -Inf and +Inf, large positive values indicating that A is a cause of B with certainty, large negative values indicating that B is a cause of A with certainty. Middle range\r\n scores (near zero) indicate that neither A causes B nor B causes A."
\r\nBut as you say this is *ternary* problem: if a large positive value indicates A is a cause of B with certainty, a large negative value can only mean A is not a cause of B.
\r\nIt seems I should produce *two* numbers: Z1 to indicate the degree that A is a cause of B, and Z2 to indicate the degree that B is a cause of A. Obviously, consistency requires they cannot both be large.
\r\nAs an example, consider a ternary probabilistic classifier and let Z1=logit P(A->B) and Z2=logit P(B->A).
", "rawMarkdown": "", "votes": null }, { "id": "22609", "postDate": "04/11/2013 06:14:18", "content": "The rationale is that eventually we want to do massive causal data mining in publicly available data and rank pairs of variable in order of how promising the causal relationship looks like. In this way, we can walk through the list and conduct experimental\r\n validations until our budget is exhausted or we have enough interesting results. So we need a ranking score.
\r\nThe problem is symmetric in A and B. So it makes sense to have a symmetric score, with A->B for large positive scores and A<-B for small negative scores and other cases in between. In this way, we can score the results as 2 classification problems: A->B\r\n vs. all other cases or A<-B vs all other cases as we do in the challenge. See "SCORING" on page: http://www.causality.inf.ethz.ch/cause-effect.php?page=tasks.
\r\nIf you produce 2 numbers, you can just subtract them form one another: Z1-Z2.
", "rawMarkdown": "", "votes": null }, { "id": "22658", "postDate": "04/11/2013 21:35:02", "content": "Hi Isabelle,
\r\nThanks for your reply--I know you must be pretty busy.
\r\nI really like this challenge, I am just concerned the scoring system is not evaluating actual beliefs.
\r\nHere is an example: suppose I think P(A->B) = 0.6 and P(B->A) =0.1. Then under your suggestion Yhat = logit 0.6 - logit 0.1 = 2.6. But I would get the same score if my beliefs were P(A->B) = 0.79 and P(B->A) = 0.21 or P(A->B) = 0.3 and P(B->A) = 0.03. While\r\n these always favour A->B over B->A, in the last case it much more likely that there is no direct causal connection at all!
\r\nI have no problem turning a ternary classification problem into two one-against-the-rest classification problems, it just seems I should provide a number for each problem.
", "rawMarkdown": "", "votes": null }, { "id": "22671", "postDate": "04/12/2013 00:40:15", "content": "I was thinking that you would normalize such that P(A->B)+P(B->A)=1, in which case you would obtain different values for each sets of results: 3.58, 2.64, and 6.81. But feel free to solve the problem in another way.
\r\nI think it makes sense to have just one score and not two because the problem is completely symmetric in A and B and imposing to have just one score forces you to take this constraint into account rather than considering the problem as a general 3-class\r\n problem. In terms of a margin-based optimization problem in which you impose that there is a margin area around your decision boundary, you could formulate it as imposing that A->B examples fall on one side of the margin, B->A for on the other side of the\r\n margin, and the other cases (A-B and A|B) fall inside the margin. Then you can either exchange the roles of A and B (to get twice as many examples of A->B and B->A) or enforce the symmetry of the classes as another constraint.
", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 22084, "author_name": "iguyon", "author_url": "", "post_date": "04/03/2013 16:19:31", "content": "For each pair of variables, we have a ternary truth value indicating whether A is a cause of B (+1), B is a cause of A (-1), or neither (0). We use the scores provided by the participants as a ranking criterion and evaluate their entries with two Area\r\n Under the ROC curve (AUC) scores:
\r\nLet Yhat be your predicted score in [-Inf, +Inf] and Y the target values in {-1, 0 1}.
\r\nWe define Y1=Y; Y1(Y==0)=-1; and Y2=Y; Y2(Y==0)=+1;
\r\nScore = 0.5*(AUC(Yhat, Y1)+AUC(Yhat, Y2));
The first score AUC(Yhat, Y1) measures the success at correctly detecting that A->B rather than [A<-B, A-B, or A|B].
\r\nThe second score AUC(Yhat, Y2) measures the success at correctly detecting that A<-B rather than [A->B, A-B, or A|B].
\r\nSince the problem is symmetric, we average the two scores.
\r\nThe organizers will also compute various other score for analysis purpose, but they will not be used to rank the participants.
I am confused about the meaning of Yhat.
\r\nFrom the competition notes: "We expect the participants to produce a score between -Inf and +Inf, large positive values indicating that A is a cause of B with certainty, large negative values indicating that B is a cause of A with certainty. Middle range\r\n scores (near zero) indicate that neither A causes B nor B causes A."
\r\nBut as you say this is *ternary* problem: if a large positive value indicates A is a cause of B with certainty, a large negative value can only mean A is not a cause of B.
\r\nIt seems I should produce *two* numbers: Z1 to indicate the degree that A is a cause of B, and Z2 to indicate the degree that B is a cause of A. Obviously, consistency requires they cannot both be large.
\r\nAs an example, consider a ternary probabilistic classifier and let Z1=logit P(A->B) and Z2=logit P(B->A).
", "votes": null, "replies": [] }, { "id": 22609, "author_name": "iguyon", "author_url": "", "post_date": "04/11/2013 06:14:18", "content": "The rationale is that eventually we want to do massive causal data mining in publicly available data and rank pairs of variable in order of how promising the causal relationship looks like. In this way, we can walk through the list and conduct experimental\r\n validations until our budget is exhausted or we have enough interesting results. So we need a ranking score.
\r\nThe problem is symmetric in A and B. So it makes sense to have a symmetric score, with A->B for large positive scores and A<-B for small negative scores and other cases in between. In this way, we can score the results as 2 classification problems: A->B\r\n vs. all other cases or A<-B vs all other cases as we do in the challenge. See "SCORING" on page: http://www.causality.inf.ethz.ch/cause-effect.php?page=tasks.
\r\nIf you produce 2 numbers, you can just subtract them form one another: Z1-Z2.
", "votes": null, "replies": [] }, { "id": 22658, "author_name": "mangostein", "author_url": "", "post_date": "04/11/2013 21:35:02", "content": "Hi Isabelle,
\r\nThanks for your reply--I know you must be pretty busy.
\r\nI really like this challenge, I am just concerned the scoring system is not evaluating actual beliefs.
\r\nHere is an example: suppose I think P(A->B) = 0.6 and P(B->A) =0.1. Then under your suggestion Yhat = logit 0.6 - logit 0.1 = 2.6. But I would get the same score if my beliefs were P(A->B) = 0.79 and P(B->A) = 0.21 or P(A->B) = 0.3 and P(B->A) = 0.03. While\r\n these always favour A->B over B->A, in the last case it much more likely that there is no direct causal connection at all!
\r\nI have no problem turning a ternary classification problem into two one-against-the-rest classification problems, it just seems I should provide a number for each problem.
", "votes": null, "replies": [] }, { "id": 22671, "author_name": "iguyon", "author_url": "", "post_date": "04/12/2013 00:40:15", "content": "I was thinking that you would normalize such that P(A->B)+P(B->A)=1, in which case you would obtain different values for each sets of results: 3.58, 2.64, and 6.81. But feel free to solve the problem in another way.
\r\nI think it makes sense to have just one score and not two because the problem is completely symmetric in A and B and imposing to have just one score forces you to take this constraint into account rather than considering the problem as a general 3-class\r\n problem. In terms of a margin-based optimization problem in which you impose that there is a margin area around your decision boundary, you could formulate it as imposing that A->B examples fall on one side of the margin, B->A for on the other side of the\r\n margin, and the other cases (A-B and A|B) fall inside the margin. Then you can either exchange the roles of A and B (to get twice as many examples of A->B and B->A) or enforce the symmetry of the classes as another constraint.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "22080": "", "22084": "", "22596": "", "22609": "", "22658": "", "22671": "" }, "source": "meta" }