Is it (a) correct to infer causality from non-experimental data, and (b) how can an algorithm find applicability in real-life scenario if neither the relevance to the scenario nor a time-structure is preserved/ given.
\r\nI do understand the advances in bayesian networks to do this, but i'm questioning the premise here; not the availability of techniques.
", "messages": [ { "id": "21983", "postDate": "04/02/2013 08:40:04", "content": "Is it (a) correct to infer causality from non-experimental data, and (b) how can an algorithm find applicability in real-life scenario if neither the relevance to the scenario nor a time-structure is preserved/ given.
\r\nI do understand the advances in bayesian networks to do this, but i'm questioning the premise here; not the availability of techniques.
", "rawMarkdown": "", "votes": null }, { "id": "22015", "postDate": "04/02/2013 18:32:24", "content": "We are not really trying to infer causality here. The gold standard is still experiments. We are interested in ranking pairs of variables {A, B} to prioritize experiments. There are good indications that there is much to win here because, depending on how\r\n the data were generated, there are measurable differences in the distribution of observations, as evidenced by plots of B vs. A (see the example at the bottom of the description page http://www.kaggle.com/c/cause-effect-pairs). You can however encounter non-identifiable\r\n case (cases that are perfectly symmetric in A and B in observational data, even though there is a causal relationship).
\r\nThere are many cases in which time is not preserved and yet there is an underlying causal structure of interest. These concern for instance the analysis of population data within a given time slice in epidemiology, medicine, and sociology. We are thinking\r\n of massive data mining of publicly available data to uncover interesting candidate causal relationships worth testing to improve public health, economical growth, etc.
\r\nAdding time helps in some cases, but the analysis is also far more complicated. We will get there in an upcoming challenge. In a challenge setting, we cannot disclose the scenarios to avoid biasing the results.
", "rawMarkdown": "", "votes": null }, { "id": "22120", "postDate": "04/04/2013 04:30:18", "content": "@Isabelle, In the sample plot of B vs. A (ie. altitude vs. temperature), how exactly did you come to the conclusion that A(temperature) is a function of B(altitude)?
\r\n\r\n
Ps: This is an interesting new field and this contest will mostly be a learning experience for me!
\r\n", "rawMarkdown": "", "votes": null }, { "id": "22129", "postDate": "04/04/2013 05:14:33", "content": "
There is a short and a long answer... The short answer is: these are values of average yearly temperature and altitude of German cities. Since we cannot move the cities, altitude cannot possibly be influenced by temperature, therefore temperature ought to\r\n be a consequence of altitude and not vice versa. This is different from the case in which we would have weather balloons free to move and we would record simultaneously altitude and temperature.
\r\nBut, this is a good question, at the heart of the notion of causality. A->B is verified experimentally if there is a significant dependency between A and B when and external agent imposes values to A ("manipulates it"). In some cases, we cannot perform actual\r\n experiments, but we can rely on "natural" experiments or "thought" experiments.
\r\nIn this case, we can say that there is an implicit natural experiment: we sampled points on the map at various altitudes and recorded temperature. However, the sampling may be biased because German cities were not necessarily built at random places. For\r\n instance, they may have been built at places where the temperature was pleasant. The dependency between altitude and temperature may also be explained (at least in part) by a common cause, like latitude.
\r\nBecause it is so difficult to know for sure in real data whether A->B, unless we have a well designed experiment to confirm it, our evaluation also relies on artificial pairs of variables for which we know the causal direction by construction.
", "rawMarkdown": "", "votes": null }, { "id": "22130", "postDate": "04/04/2013 05:28:33", "content": "Ok, now I understand.
\r\nBut it leads me to the next question - in the example plot, is it still possible to find causal relationship if the labels (temperature & altitude) are removed? If yes, then how?
\r\nIn addition, I see couple of cases possible here-
\r\n1. No relationship whatsoever
\r\n2. Causal relationship
\r\n3. Correlation, no causation (common underlying cause)
\r\n4. Correlation, no causation (no common cause, unrelated variables)
\r\nOption 4 might include things like financial astrology :-)
", "rawMarkdown": "", "votes": null }, { "id": "22135", "postDate": "04/04/2013 05:54:22", "content": "In the case of this example it looks like there is a non invertible function (like a little hat), so yes, in the case of this example, it is possible to detect that altitude must be the cause of temperature. Sometimes the noise can help you figure out the\r\n causal direction. Try a linear function with uniform additive noise.
\r\nWe have examples of your case (1) in the data (denoted as A|B); we included unrelated vaiables whose values were independently randomly permuted (so they are really unrelated).
\r\nWe have examples of your case (2) as discussed previously.
\r\nWe have examples of your case (3) generated from real variables X, Y, and Z, where A=f(X, Z) and B=g(Y, Z), i.e. Z is a common cause of A and B.
\r\nThere may be cases of coincidental dependency (particularly because of the small number of samples), like your case (4), but this should be really rare.
\r\nThere is also the case of feed-back loops A->B and A<-B, but we are not considering this case in this challenge.
", "rawMarkdown": "", "votes": null }, { "id": "22580", "postDate": "04/10/2013 21:41:32", "content": "Unfortunately I must disagree. In a chart such as the example, all notions of "orientation" are removed. You are defending the position that, if the chart we see is rotated, inverted, upside-down, anything, there is a way that the data itself suggests\r\n B=>A? And furthermore, out algorithms are "wrong" if we predict with equal strength A would imply B? I agree with you that in real-world this is doable, but in a number theory context devoid of anything but sets of numbers, such a task is impossible except\r\n in rare situations. Do you have a working example you can provide of HOW B=>A in your example chart? Instead of just telling us and we take it on faith? Give a chart that looks just like your example, but with NO KNOWLEDGE other than the number pairs, show\r\n B MUST imply A and not A => B? Furthermore, your footnote below the example used external evidence to conclude B=>A (namely, B is altitude and A is temperature). Only THEN did you decide which way the causation arrow pointed. I'm sorry but I need an example\r\n that actually follows the rules you yourself have set forth.", "rawMarkdown": "", "votes": null }, { "id": "22583", "postDate": "04/10/2013 22:34:16", "content": "I added a piece-wise linear fit plus orange intersections.
\r\nIf A -> B: f(A=130) = 0 and f(A=130) = 1500 (no)
\r\nIf B -> A: f(B=0) = 130 and f(B=1500) = 130 (yes)
\r\nDoes that help?
", "rawMarkdown": "", "votes": null }, { "id": "22584", "postDate": "04/10/2013 22:35:00", "content": "yes very much actually. thanks!
", "rawMarkdown": "", "votes": null }, { "id": "22790", "postDate": "04/14/2013 14:44:06", "content": "Once you add the linear fit, it does indeed make sense. However, the question is whether or not I could come up with a situation where A causes B which results in similar data being generated. What if B was the temperature in a refridgerator and A was\r\n the amount of power taking up by the refridgerator but was measured by a device which did not provide reliable readings beyond 140 units, leading to the odd kink at the bottom of the graph? It sounds contrived, but only because we now know what A and B are.\r\n I really don't see how one could infer that B causes A without some time dimension (Granger Causality) or have appropriate context (i.e. theory about the actual real world phenomenon that we are measuring.)
", "rawMarkdown": "", "votes": null }, { "id": "22797", "postDate": "04/14/2013 17:53:02", "content": "I think of it like I'm placing a bet on the outcome of the experiment that will take place in the future. I don't know what the outcome will be, but I can adjust my bet according to the evidence: high correlation makes dependence more likely and non-invertibility\r\n makes B->A more likely. So in this case I can bet on B->A with some amount of confidence, and after the experiment is run, I will find out if I won or lost.
", "rawMarkdown": "", "votes": null }, { "id": "22803", "postDate": "04/14/2013 19:27:09", "content": "How can you test for non-inveribility? I'm getting lost in maths.
", "rawMarkdown": "", "votes": null }, { "id": "22817", "postDate": "04/15/2013 02:20:56", "content": "What do you gurus say if we substitute an altimeter for a thermometer -- the type of altimeter that is based on air pressure. Then we use the altimeter to determine the height of the city.
", "rawMarkdown": "", "votes": null }, { "id": "23399", "postDate": "04/24/2013 05:29:21", "content": ""I added a piece-wise linear fit plus orange intersections.
\r\nIf A -> B: f(A=130) = 0 and f(A=130) = 1500 (no)
\r\nIf B -> A: f(B=0) = 130 and f(B=1500) = 130 (yes)
\r\nDoes that help?"
\r\n\r\n
I don't think the argument is convincing. If we take a linear transformation of A such as 5A and keep B the same, then B=10 can lead to 5A=5*130=650 and 5A=5*160=800. Right?
", "rawMarkdown": "", "votes": null }, { "id": "23404", "postDate": "04/24/2013 08:43:25", "content": "Argument is convincing.
\r\nIf we take a linear transformation of A such as 5A and keep B the same, then
\r\nIf A -> B: f(A=650) = 0 and f(A=650) = 1500 (no)\r\nIf B -> A: f(B=0) = 650 and f(B=1500) = 650 (yes)\r\n\r\n
@Wei Deng: You take A=5*160 but A is never 160 on the original red line, especially not for B=10.
", "rawMarkdown": "", "votes": null }, { "id": "23424", "postDate": "04/24/2013 17:40:47", "content": "I think I must have missed something. It is true that (A=130, B=0) and (A=130, B=1500) are two data points; however, according to the plot, we could also find two points that are on the same horizon, i.e., two data points sharing the same B value but with\r\n different A values. Right?
", "rawMarkdown": "", "votes": null }, { "id": "23431", "postDate": "04/24/2013 20:21:13", "content": "That's not what I am seeing. For what B value would that occur?
", "rawMarkdown": "", "votes": null }, { "id": "23434", "postDate": "04/24/2013 20:44:03", "content": "Sticking my nose in here, this being an idea I hinted at earlier. It's arguable I think that multiple A-function values could be ocurring even at one or the other of the two B values (roughly 200 and 1500) highlighted in Isabelle's figure. To know for sure,\r\n wouldn't you have to be able to separate the signal from the noise? The working assumption seems to be that the "true" function is "smooth" and "simple", and the dispersion of the points is a good indicator of the noise level. Seems to me it's that somewhat\r\n imprecise assumption that leads you to dismiss the multiple values as noise in one case but not in the other.
", "rawMarkdown": "", "votes": null }, { "id": "23439", "postDate": "04/24/2013 22:54:31", "content": "@Steven: If you draw a horizontal line that is slightly above the line B=0, then it is evident that that line touches at least two data points.
\r\n@Bruce: "And the dispersion of the points is a good indicator of the noise level." Because both A and B are free to take a linear transformation of themselves, there is no metric, i.e., we cannot say if 500 is a big number or not.
", "rawMarkdown": "", "votes": null }, { "id": "23449", "postDate": "04/25/2013 08:49:48", "content": "@Wei Deng: Oh yes, you're right. If you're talking about the data, then there are values of B for which there are several values of A. But the point in this thread is that you can approximate the data by some lines like the red ones, and on the red lines\r\n there is no point of B for which there are multiple possibilities for A.
", "rawMarkdown": "", "votes": null }, { "id": "23450", "postDate": "04/25/2013 09:17:09", "content": "@Steven: Now I see. Thank you!
\r\nSomehow I feel that the context for this project is insufficient. So far I doubt the validity of this project, but I could be wrong.
", "rawMarkdown": "", "votes": null }, { "id": "23452", "postDate": "04/25/2013 10:03:17", "content": "I think the rules ("other rules") are rather confusing. I may open another thread for that.
", "rawMarkdown": "", "votes": null }, { "id": "23455", "postDate": "04/25/2013 13:16:59", "content": "Blimey - just read the other rules! Very harsh to expect people to write a 6 page paper just to be considered for the prizes! Don't mind submitting a model / software but won't be writing a paper unless I'm in top 3 (which we won't know until the conference)
", "rawMarkdown": "", "votes": null }, { "id": "23464", "postDate": "04/25/2013 15:11:09", "content": "Perhaps I'm missing something, but doesn't this "invertible function" framework for determining causality break down with bimodal distributions? A simple example would be androgen levels as a function of age (sex unknown). I'm assuming levels would be similar\r\n pre-puberty and diverge in males vs femles post-puberty, yielding a pretty clear counterexample...
", "rawMarkdown": "", "votes": null }, { "id": "23477", "postDate": "04/25/2013 20:10:28", "content": "For a dataset that both A and B are numerical variables, I think at best we can ask if there is some reasonable function f(x, y) such that "up to noise", that f(A, B) = 0 fits the data and can be used for prediction. The question of whether one causes another\r\n seems impractical, especially if there is no background information of a dataset and no units for observations.
", "rawMarkdown": "", "votes": null }, { "id": "23484", "postDate": "04/25/2013 22:01:41", "content": "In fact, to generalize the puberty counterexample above, it seems that anytime there is an interaction between one of the known variables and a third unknown variable, a similar noninvertible function can easily arise, regardless of causality.
\r\nThese are elementary examples, and could be applied to the temperature-altitude problem, e.g. samples come from a random selection of cities in Germany and Sweden. The scatterplot would be two decreasing lines + noise, noninvertible with respect to\r\n either variable if there was any overlap when projected onto the X or Y axes respectively. This is because a third variable (latitude) interacts with height (same height is colder in northern Sweden than southern Germany).
\r\nIf we know what the variables represent, we can (potentially) make assumptions or modify our model in ways that solve this problem, but when we don't know the back story, we're helpless. Unless we can somehow be guaranteed that NO interactions are possible\r\n with ANY unknown variables.
", "rawMarkdown": "", "votes": null }, { "id": "23503", "postDate": "04/26/2013 06:34:25", "content": "Very interesting challenge. I'll be interested to see if there is success in developing these algorythms (both in and outside of this challenge).
\r\nI'd like to mention how your German city example does illustrate how hard this is. The challenge states that altitude causes temperature to decrease.
\r\nI'd like to suggest that there is not a single causal relationship between temperature and altitude. Both temperature and pressure are influenced by the forces gravity and fluid mechanics. Temperature is also influenced by the greenhouse effect.
\r\nOn my first reflection, saying that altitude causes lower temperatures is an example of the Texas Sharp Shooter fallacy. Identifying the positions where temperature will be higher because of other forces is a bit of drawing\r\n the target around the bullethole.
", "rawMarkdown": "", "votes": null }, { "id": "23531", "postDate": "04/26/2013 16:49:50", "content": "I will also throw in something that bothers me about this example. I get the non-invertibility point, but couldn't this example pattern easily arise if there is an underlying time variable that causes both A and B? The graph looks much like what you get\r\n if you plot unemployment rates and job openings. In earlier times, there was a clear inverse relationship. In the last recession, job openings have risen with little reduction in unemployment (this will give you two predicted unemployment rates corresponding\r\n to a single job opening rate). There is no apparent causation from A to B or vice versa, but an underlying time structure that is affecting both.
\r\nPerhaps I don't understand the task. Is it to come up with a classification scheme that is most likley to reflect the actual (known, in this case) causative relationship even if there are multiple possible relationships if we don't know what was really\r\n used? If so, I suppose it could be an interesting exercise, but somewhat unclear how useful it is to me. Wouldn't it depend on how complex the underlying structure was? For example, a classification scheme might be successful provided that there is no time\r\n variable, or provided that there is only a single variable correlated with both A and B, but might not work as well in a more complicated structure. How useful would the results be in that case?
\r\nI could be missing something fundamental, but I don't see how non-invertibility implies the diredtion of causation. Can someone help me here?
", "rawMarkdown": "", "votes": null }, { "id": "23535", "postDate": "04/26/2013 19:25:23", "content": "I am assuming that the task is to come up with a model for a very narrow definition of 'causality' - the relationship is direct and there are no other factors involved or they are controlled to be the same. So, in the economy example more job openings could\r\n 'cause' unemployment rate to go down if other things like the pool of workers, suitability for job, salary, willingness to work etc are the same or controlled.
\r\nIf the definition of causality is different, the task wouldn't make sense.
", "rawMarkdown": "", "votes": null }, { "id": "23538", "postDate": "04/26/2013 22:56:12", "content": "I would have immediately rejected this notion that invertibility has any relationship with causality were it not for the credentials of some of the proponents of this idea. Sankalp and previous posts have identified assumptions that seem necessary (although\r\n I'm not convinced of their sufficiency) for this framework to be meaningful. However, these assumptions are very restrictive and seem to be in direct conflict with the stated goal of searching large complex observational datasets in which variables have not\r\n been carefully controlled. In fact, it seems like the rigorously controlled scenario described by Sankalp et al. would be very unlikely to arise except in controlled experiments, which would obviously defeat the point, or in situations that are so simple these\r\n methods wouldn't be needed.
\r\nI would like to hear the competition administrator's response to the issues discussed in the past few posts, especially the assertion I described above that an interaction with a 3rd variable can yield noninvertible functions that are clearly unrelated to\r\n causality. This flaw seems to cast doubt on this entire endeavor.
", "rawMarkdown": "", "votes": null }, { "id": "23573", "postDate": "04/28/2013 17:15:45", "content": "\r\n
Look, I think the answer is simple: in general no, it's not possible.
\r\nBut... suppose that the admins promise you that the data come from a restricted set of possibilities, such that you can be sure a causal relationship exists each time a variable x is larger than a variable y. Then the task would be both possible\r\n and easy: just make your algorithm looks at whether you can be confident one variable is larger.
\r\nFrom that our task is clear: to figure out, or to guess, what the admins actually promised, and to design an algorithm able to detect the consequences of these promises.
\r\nIn the long run, what admins have in mind is that it may be that the set of naturely occuring data can more often than not be caracterized as a restricted set of possibilities, such that causal relationships more often than\r\n not let some imprint in the data. Whether it's a correct view or a dead end is by itself an interesting empirical question, however beyond the scope of the present contest.
", "rawMarkdown": "", "votes": null }, { "id": "23575", "postDate": "04/28/2013 17:51:07", "content": "Jay, you are misunderstanding the competition, and perhaps the meaning of causality when you state that
\r\n[quote=Jay;23573]
\r\nyou can be sure a causal relationship exists each time a variable x is larger than a variable y
\r\n[/quote]
\r\nThe competition is about inferring causality based on any applicable methods, but the prevailing strategy seems to be based on invertible vs noninvertible functions. I have shown in previous posts on this thread that this framework may have fundamental flaws\r\n and would be interested in having a discussion about my concerns.
", "rawMarkdown": "", "votes": null }, { "id": "23577", "postDate": "04/28/2013 18:15:28", "content": "ahfoss
\r\nI'm with you and would still like to see more discussion. I understand invertibility but do not see that as necessary or sufficient to establish causality. If there are further restrictions imposed about the potential nature of the relationships (such\r\n as limits on feedback loops, lack of time series, lack of additional factors, etc.) then I can see invertibility becoming a way to classify the causal relationships. But, then I fail to see the usefulness of this competition for shedding light on observed\r\n data - surely no such restrictions are imposed on observed data.
\r\nI want to avoid the jargon (because I quickly won't understand it myself), but I did take the initial quiz listed on the competition and got the correct answer. It was easy given the clue about invertibility. However, the more I thought about it, the less\r\n I believed it really was the correct answer. That example reminded me of this data:
\r\nwww.businessweek.com/articles/2013-02-07/the-rise-of-long-term-joblessness
\r\nAccording to the competition example, it should be the case that long-term unemployment rate causes the job openings rate but obviously both are caused by other factors. So, what am I missing?
", "rawMarkdown": "", "votes": null }, { "id": "23578", "postDate": "04/28/2013 18:25:34", "content": "Ahfoss, sorry I was unclear, but you are confusing "suppose that the admins promise that..." with "Jay states that...". Dude, were you\r\nreally believing I understand causality as "one variable must be larger than the other"! :-)
\r\n\r\n
Ok let me rephrase:
\r\nYour concern (and other similar (such as Dale's)) is perfectly valid in general.\r\n
\r\nIt probably does not apply to the data of this contest.\r\n
\r\nWhether it applies to the set of naturally occuring data is an interesting empirical question, imho beyond the scope of this contest.
\r\n\r\n
Regards,
", "rawMarkdown": "", "votes": null }, { "id": "23579", "postDate": "04/28/2013 18:43:22", "content": "Dale, I don't think you're missing anything. I think that if we're only given two variables and no subject matter knowledge, it's *possible* that under very restrictive conditions there could be a relationship between invertibility and causality, but I'm\r\n still not convinced that this is possible. Even if it is possible, the amount of knowledge required to make those assumptions would probably mean we already know enough to identify causality outright, without needing to rely on something as tenuous as the\r\n functional form of the data.
\r\nNow, if we suppose that we have a multivariate dataset that is rich enough to include all relevant variables, then the problem may be tractable with established causal modeling techniques... but the more I think about this "blind" bivariate competition the\r\n more it seems to lack validity.
\r\nInferring causality from unknown bivariate datasets would be revolutionary of the "holy-grail" sort, so I think the burden of proof rests squarely on the shoulders of its proponents. I invite them to chime in here and defend the invertibility method. Perhaps\r\n we're both missing something?
", "rawMarkdown": "", "votes": null }, { "id": "23580", "postDate": "04/28/2013 18:55:45", "content": "Jay, I apologize for the misquote. I'll remove it if you want me to.
\r\nSo if I understand you correctly, you are saying that the competition admins claim that attributes XYZ imply causality, and we are to take that for granted and simply search for XYZ.
\r\nI am saying that XYZ does not imply causality. This seems like a very important question, in fact, a fundamental one. Not for purposes of scoring well in the competition, of course, but to verify that this competition is not based on misguided statistical\r\n assertion.
", "rawMarkdown": "", "votes": null }, { "id": "23590", "postDate": "04/28/2013 20:36:57", "content": "No problem, I know my english is not as clear as I'd like.\r\n
\r\nYes, thanks for the rephrasing: we have to guess which attributes XYZ admins think as implying causality (or, more properly, what attributes XYZ imply causality for the set of data the admins selected), and yes non invertibility\r\n seems one of these features, and no there is no guarantee it is true in general. To take the temperature/altitude exemple, the apparence of non invertibility could well be explained\r\n by a third variable, say city location as inland versus seaside (to be honnest I think this is the case and that this example is terribly misguided*).
\r\nBut I won't say overall the contest is based on misguided statistical assumption, first because admins "are not really trying to infer causality here. The gold standard is still experiments.\r\n We are interested in ranking pairs of variables {A, B} to prioritize experiments.", and second because it might be possible that some of the tricks they have in mind actually work more often than not for real data (or, more subttle, for real data humans tend\r\n to record).
\r\n\r\n
\r\n
* Edit: to illustrate why, see the plot attached. It's a series of south american cities chosen est or west of the Andes (namely, a third variable impact the relationship between temperature and altitude). We'd be supposed to conclude temperature cause altitude...
\r\n\r\n
Edit2: if real data is not convincing enough, here's a noise free model where one is supposed to conclude that B cause A. Actually A and a third variable C cause B.
\r\n", "rawMarkdown": "", "votes": null }, { "id": "23594", "postDate": "04/28/2013 21:46:54", "content": "
I'll stick my nose back in here with a different example considering 2 functions:
\r\n1) B = A and 2) B = A^2
\r\nBoth function are noisless and in both cases B is dependent on A. The invertibility of (1) makes it impossible to determine the direction of the dependence and the noninvertibility of (2) makes it possible. So we can say that a) it's not always possible\r\n to determine the direction of dependence, and b) it's sometimes possible to determine the direction of dependence. That's one of the claims of the background material: noninvertibility adds information even in the absense of noise.
\r\nNotice how I avoided the dreaded 'c' word. The possibility that A and B are both dependent on an unobserved variable C is a separate problem from the one of determining the direction of observable dependence. I think this competition will be won by the\r\n team that best quantifies that possibility.
\r\nEdit: I screwed up the functions when I first posted
", "rawMarkdown": "", "votes": null }, { "id": "23596", "postDate": "04/28/2013 21:56:14", "content": "Jay, you bring up excellent points. I'm glad I understand your points better now.
\r\nSo yes, on the one hand, if it works, it works! The proof will be in the pudding (as they say). However, to be taken seriously by my fellow statisticians, it will have to be rigorously proven why and when this method works. The fact that even the defining\r\n example (temp-altitude) seems flawed to you, me, and others in this thread is not encouraging. Can anybody think of a single example that actually works, that we can't similarly deconstruct and dismiss? If there is some pattern that arises in "typical" examples\r\n collected by researchers, what could this be, and when does it arise? These are interesting and important questions.
\r\nBut, you correctly point out that the admins have hedged, and are "not really trying to infer causality," but they then describe a procedure that is textbook statistical inference: drawing general conclusions (i.e. direction of causality) from specific\r\n observations (data), with associated measures of uncertainty (the ranking/prioritizing business). So let's just be clear that, although we are not making\r\nabsolute statements, we are most definitely inferring causality.
\r\nBut theoretical concerns aside, this just doesn't pass the smell test for me. In a recent Monte Carlo simulation I ran, a bivariate scatter plot of the two variables of interest resembled four intersecting nonlinear curves plus noise, due to interactions\r\n with a few other variables of itnerest. It doesn't get any less invertible than this, and it was a relatively simple epidemiological simulation with a simple causal structure.
", "rawMarkdown": "", "votes": null }, { "id": "23597", "postDate": "04/28/2013 22:02:20", "content": "On the Identifiability of the Post-Nonlinear Causal Model: http://arxiv.org/ftp/arxiv/papers/1205/1205.2599.pdf
\r\nThis article seems to be giving the conditions when the causal direction is identifiable. (disclaimer: just found it, haven't fully read/understood all of it)
", "rawMarkdown": "", "votes": null }, { "id": "23598", "postDate": "04/28/2013 22:14:32", "content": "This is getting fun.
\r\nJay now has evidence, according to the invertibility heuristic, that temperature causes altitude. See his post above. I'm assuming the quadratic pattern would hold with more data, but even if it doesn't, we can easily imagine situations in which it would.
\r\nEdge2, assuming you mean "statistical dependence" when you say dependence (which by definition is not directional i.e. symmetric), it seems that you are just misusing the word as a surrogate for the "dreaded 'c' word." Also, Jay's graph invalidates a quadratic\r\n function as implying causality, unless you believe his evidence that temperature causes altitude.
", "rawMarkdown": "", "votes": null }, { "id": "23601", "postDate": "04/28/2013 22:42:54", "content": "Herra posts an interesting paper; from my very brief skimming it looks like they are treating invertibility as a given and not a measurable index of causality (e.g. in equations 1, 2, and 3); they also give certain combinations of distributions and functional\r\n forms for which their PNL model cannot identify causality. Although not particularly relevant to our discussion of noninvertibility (I think), it seems like a potentially promising avenue to pursue for competitors.
\r\nLooking through the references led me to this paper which may be useful to some competitors (looks like the additive noise model outperforms the others):
\r\nhttp://www.is.tuebingen.mpg.de/fileadmin/user_upload/files/JMLR-2010-Mooij.pdf
\r\nAs well as this website which seems to be the origin of the altitude-temperature example:
\r\nhttp://people.kyb.tuebingen.mpg.de/jorism/causality-data/
", "rawMarkdown": "", "votes": null }, { "id": "23602", "postDate": "04/28/2013 22:51:00", "content": "[quote=ahfoss;23601]
\r\nAlthough not particularly relevant to our discussion of noninvertibility (I think), it seems like a potentially promising avenue to pursue for competitors.
\r\n[/quote]
\r\nTheir model is a bit more general than the ones thought here (ie. they include noise and invertible post-nonlinear distortion f2), but I think their results are relevant also to this noninvertibility (of f1) discussion. See corollary 10.
", "rawMarkdown": "", "votes": null }, { "id": "23603", "postDate": "04/28/2013 23:01:34", "content": "ahfoss, I'm not talking about statistical dependence, I'm talking about B being the dependent variable. But you're right that I'm using it just to avoid using cause/effect. I thought it would make it easier to make the point but that was misguided. Let's\r\n try this again:
\r\n1) B = A, 2) B = A^2
\r\nIn 1), it is impossible to determine the causal direction because the function is invertible. In 2), B cannot cause A because the function is not invertible. Again, the claim is simply that noninvertibility adds information even in the absense of noise. I'm\r\n not sure how you can determine that this claim is somehow invalidated by a few noisy samples.
", "rawMarkdown": "", "votes": null }, { "id": "23604", "postDate": "04/28/2013 23:37:43", "content": "Here's a few more thoughts before I bow back out of this discussion. I somehow keep forgetting how much back and forth is required before it is even clear what I'm talking about. What starts as a 30 minute diversion quickly turns into an entire afternoon.\r\n
\r\nFirst, I say 'invertibility adds information even in the absense of noise' because a lot of methods require noise to determine causal direction. I forgot to mention that above. Second, I think an important point that is easy to overlook is that a lot of\r\n published methods only separate A->B from B->A. They don't consider the cases A-B and A|B. In other words, they take data that is assumed to be causal, then separate it into the cause and effect. The problem we face in this\r\n competition is much harder than that because we also need to separate (A->B,B->A) from A|B and A-B.
\r\nEdit: (sigh... I thought I was done :) )
\r\nIt's possible that invertibility only adds information when separating A->B from B->A, and may be meaningless in separating (A->B, B->A) from A-B. That is, if we know that A and B are causally related, invertibility is an\r\n important clue that can help separate the cause from the effect, but that may not be the case when we are trying to determine if the data are causally related or not.
", "rawMarkdown": "", "votes": null }, { "id": "23606", "postDate": "04/29/2013 00:15:00", "content": "I thought I was done as well - but I think I agree with your last statements (Edge2). If that is correct, then it is the example they provide that is really misleading. It is in the video as well which makes it doubly misleading.
", "rawMarkdown": "", "votes": null }, { "id": "23617", "postDate": "04/29/2013 12:50:34", "content": "Edge2, I think I understand you better now! Thanks for clarifying. I think that I agree with your statement that invertibility matters if we are discriminating between A->B and B->A ONLY, i.e. no other variables, no feedback loops, etc. This dovetails with\r\n our previous discussion, and I think we're approaching a unanimous forum-wise convergence on these very strong assumptions for invertibility to mean anything at all.
\r\nNevertheless, these assumptions only seem to be met in very unrealistic conditions, or in controlled experiments, so I still don't see how this invertibility stuff stands a chance of being useful.
\r\nIn our discussions, we have identified multiple papers that seem to hold much more potential (see previous posts).
\r\nIt seems that we are also near unanimity in deciding that the temperature altitude example is bunk. Jay has constructed "proofs" by invertibility that temperature "causes" altitude, and I and others have pointed out that a third variable can easily yield\r\n noninvertible functions in variables that are clearly causal (see previous posts).
\r\nThe competition admins MUST either explain where we are wrong, or apologize for the misleading example. I love the idea of Kaggle and the community that has arisen around it, and I strongly feel that Kaggle will lose credibility if shoddy methods/logic become\r\n commonplace. We must raise our concerns when questionable assertions are made!
\r\nI've got a busy 2 weeks ahead, so I too, must bow out for now. Wei D., Dale L., Sankalp, Jay, Edge2, Herra H.: It has been my pleasure to discuss these interesting ideas with clever and thoughtful people. Best of luck.
", "rawMarkdown": "", "votes": null }, { "id": "23633", "postDate": "04/29/2013 16:13:00", "content": "Thank you for your guys' input on this! On the other hand, my philosophy hasn't changed so far because some essential questions are still there. One thing is identifiability, i.e., {A, B} should contain exactly the same amount of information as {A', B'}\r\n if A' (B') is a linear transformation of A (B). (Here A, B are numerical variables without units.) Do you guys agree? So any acceptable method should give exactly the same judgement on {A, B} and {A', B'}. It follows we have to standardize A and B to have\r\n say mean 0 and variance 1.
\r\nOne thing that MIGHT make this project valid is that we are only required to infer whether or not A is caused by B (more precisely I should say whether or not A is a function of B) as opposed to which function it is.
\r\nIn terms of function, in my opinion, we should ask the following question instead, namely, whether or not there is a reasonable bivariate function f(x,y) such that f(A,B) = 0 fits the data. While y = x^2 is a perfect function, x = plus/minus y^0.5 is also\r\n a very reasonable function although it is not invertible. We cannot conclude if something is more fundamental (here cause is more fundamental than effect) just based on this "dis-invertibility".
\r\nJust my 2 cents.
", "rawMarkdown": "", "votes": null }, { "id": "23957", "postDate": "05/06/2013 22:08:29", "content": "I produced a new versions of my slides with voice over:
\r\nSlide presentation: overview of the challenge (pptx with voice over, May 1st version).
\r\nThe presentation lasts about 1/2 hour, but you can skip over things you know already (particulalry if you watched the video because all the beginning is the same). It starts being relevant to this discussion at slide 12. I explain methods people have been\r\n using to orient the causal direction between pairs of dependent variables using additive noise models, latent variable models, and information theoretic methods (this is very qualitative, you need to look at the papers to really get the formal approach).
", "rawMarkdown": "", "votes": null }, { "id": "23958", "postDate": "05/06/2013 22:15:10", "content": "I am organizing a series of teleconference seminars so you can ask questions to the experts:
\r\nhttps://sites.google.com/a/chalearn.org/readinggroup/
\r\nI invited Dominik Janzig, Joris Mooij and Kun Zhang to talk to you about addiive noise models, latent variable models and information theoretic approaches. We will meet every Thursday at 9 am PT for 1 hour by teleconference:
\r\nConference Dial-in Number: +1 (610) 214-0000
\r\nParticipant Access Code: 549086#
This week (May 9) I will just go over my slide deck so you can ask me questions.
", "rawMarkdown": "", "votes": null }, { "id": "23973", "postDate": "05/07/2013 03:44:11", "content": "Thanks for the reply, Isabelle. I look forward to perusing the slides once my schedule simmers down. The conference call sounds great, but it's a difficult time to dial in for those of us who work 9-5 anywhere from PST to EST. Forum posts are great since\r\n we can access them any odd time we may have to spare... e.g. now, quarter tomidnight :).
\r\nPerhaps your colleagues could start a separate string of posts, or maybe post a transcript of the conf. call proceedings?
", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 22015, "author_name": "iguyon", "author_url": "", "post_date": "04/02/2013 18:32:24", "content": "We are not really trying to infer causality here. The gold standard is still experiments. We are interested in ranking pairs of variables {A, B} to prioritize experiments. There are good indications that there is much to win here because, depending on how\r\n the data were generated, there are measurable differences in the distribution of observations, as evidenced by plots of B vs. A (see the example at the bottom of the description page http://www.kaggle.com/c/cause-effect-pairs). You can however encounter non-identifiable\r\n case (cases that are perfectly symmetric in A and B in observational data, even though there is a causal relationship).
\r\nThere are many cases in which time is not preserved and yet there is an underlying causal structure of interest. These concern for instance the analysis of population data within a given time slice in epidemiology, medicine, and sociology. We are thinking\r\n of massive data mining of publicly available data to uncover interesting candidate causal relationships worth testing to improve public health, economical growth, etc.
\r\nAdding time helps in some cases, but the analysis is also far more complicated. We will get there in an upcoming challenge. In a challenge setting, we cannot disclose the scenarios to avoid biasing the results.
", "votes": null, "replies": [] }, { "id": 22120, "author_name": "r00kie", "author_url": "", "post_date": "04/04/2013 04:30:18", "content": "@Isabelle, In the sample plot of B vs. A (ie. altitude vs. temperature), how exactly did you come to the conclusion that A(temperature) is a function of B(altitude)?
\r\n\r\n
Ps: This is an interesting new field and this contest will mostly be a learning experience for me!
\r\n", "votes": null, "replies": [] }, { "id": 22129, "author_name": "iguyon", "author_url": "", "post_date": "04/04/2013 05:14:33", "content": "
There is a short and a long answer... The short answer is: these are values of average yearly temperature and altitude of German cities. Since we cannot move the cities, altitude cannot possibly be influenced by temperature, therefore temperature ought to\r\n be a consequence of altitude and not vice versa. This is different from the case in which we would have weather balloons free to move and we would record simultaneously altitude and temperature.
\r\nBut, this is a good question, at the heart of the notion of causality. A->B is verified experimentally if there is a significant dependency between A and B when and external agent imposes values to A ("manipulates it"). In some cases, we cannot perform actual\r\n experiments, but we can rely on "natural" experiments or "thought" experiments.
\r\nIn this case, we can say that there is an implicit natural experiment: we sampled points on the map at various altitudes and recorded temperature. However, the sampling may be biased because German cities were not necessarily built at random places. For\r\n instance, they may have been built at places where the temperature was pleasant. The dependency between altitude and temperature may also be explained (at least in part) by a common cause, like latitude.
\r\nBecause it is so difficult to know for sure in real data whether A->B, unless we have a well designed experiment to confirm it, our evaluation also relies on artificial pairs of variables for which we know the causal direction by construction.
", "votes": null, "replies": [] }, { "id": 22130, "author_name": "r00kie", "author_url": "", "post_date": "04/04/2013 05:28:33", "content": "Ok, now I understand.
\r\nBut it leads me to the next question - in the example plot, is it still possible to find causal relationship if the labels (temperature & altitude) are removed? If yes, then how?
\r\nIn addition, I see couple of cases possible here-
\r\n1. No relationship whatsoever
\r\n2. Causal relationship
\r\n3. Correlation, no causation (common underlying cause)
\r\n4. Correlation, no causation (no common cause, unrelated variables)
\r\nOption 4 might include things like financial astrology :-)
", "votes": null, "replies": [] }, { "id": 22135, "author_name": "iguyon", "author_url": "", "post_date": "04/04/2013 05:54:22", "content": "In the case of this example it looks like there is a non invertible function (like a little hat), so yes, in the case of this example, it is possible to detect that altitude must be the cause of temperature. Sometimes the noise can help you figure out the\r\n causal direction. Try a linear function with uniform additive noise.
\r\nWe have examples of your case (1) in the data (denoted as A|B); we included unrelated vaiables whose values were independently randomly permuted (so they are really unrelated).
\r\nWe have examples of your case (2) as discussed previously.
\r\nWe have examples of your case (3) generated from real variables X, Y, and Z, where A=f(X, Z) and B=g(Y, Z), i.e. Z is a common cause of A and B.
\r\nThere may be cases of coincidental dependency (particularly because of the small number of samples), like your case (4), but this should be really rare.
\r\nThere is also the case of feed-back loops A->B and A<-B, but we are not considering this case in this challenge.
", "votes": null, "replies": [] }, { "id": 22580, "author_name": "chaseshaw", "author_url": "", "post_date": "04/10/2013 21:41:32", "content": "Unfortunately I must disagree. In a chart such as the example, all notions of "orientation" are removed. You are defending the position that, if the chart we see is rotated, inverted, upside-down, anything, there is a way that the data itself suggests\r\n B=>A? And furthermore, out algorithms are "wrong" if we predict with equal strength A would imply B? I agree with you that in real-world this is doable, but in a number theory context devoid of anything but sets of numbers, such a task is impossible except\r\n in rare situations. Do you have a working example you can provide of HOW B=>A in your example chart? Instead of just telling us and we take it on faith? Give a chart that looks just like your example, but with NO KNOWLEDGE other than the number pairs, show\r\n B MUST imply A and not A => B? Furthermore, your footnote below the example used external evidence to conclude B=>A (namely, B is altitude and A is temperature). Only THEN did you decide which way the causation arrow pointed. I'm sorry but I need an example\r\n that actually follows the rules you yourself have set forth.", "votes": null, "replies": [] }, { "id": 22583, "author_name": "edge219878", "author_url": "", "post_date": "04/10/2013 22:34:16", "content": "I added a piece-wise linear fit plus orange intersections.
\r\nIf A -> B: f(A=130) = 0 and f(A=130) = 1500 (no)
\r\nIf B -> A: f(B=0) = 130 and f(B=1500) = 130 (yes)
\r\nDoes that help?
", "votes": null, "replies": [] }, { "id": 22584, "author_name": "chaseshaw", "author_url": "", "post_date": "04/10/2013 22:35:00", "content": "yes very much actually. thanks!
", "votes": null, "replies": [] }, { "id": 22790, "author_name": "", "author_url": "", "post_date": "04/14/2013 14:44:06", "content": "Once you add the linear fit, it does indeed make sense. However, the question is whether or not I could come up with a situation where A causes B which results in similar data being generated. What if B was the temperature in a refridgerator and A was\r\n the amount of power taking up by the refridgerator but was measured by a device which did not provide reliable readings beyond 140 units, leading to the odd kink at the bottom of the graph? It sounds contrived, but only because we now know what A and B are.\r\n I really don't see how one could infer that B causes A without some time dimension (Granger Causality) or have appropriate context (i.e. theory about the actual real world phenomenon that we are measuring.)
", "votes": null, "replies": [] }, { "id": 22797, "author_name": "edge219878", "author_url": "", "post_date": "04/14/2013 17:53:02", "content": "I think of it like I'm placing a bet on the outcome of the experiment that will take place in the future. I don't know what the outcome will be, but I can adjust my bet according to the evidence: high correlation makes dependence more likely and non-invertibility\r\n makes B->A more likely. So in this case I can bet on B->A with some amount of confidence, and after the experiment is run, I will find out if I won or lost.
", "votes": null, "replies": [] }, { "id": 22803, "author_name": "domcastro", "author_url": "", "post_date": "04/14/2013 19:27:09", "content": "How can you test for non-inveribility? I'm getting lost in maths.
", "votes": null, "replies": [] }, { "id": 22817, "author_name": "jc3626439", "author_url": "", "post_date": "04/15/2013 02:20:56", "content": "What do you gurus say if we substitute an altimeter for a thermometer -- the type of altimeter that is based on air pressure. Then we use the altimeter to determine the height of the city.
", "votes": null, "replies": [] }, { "id": 23399, "author_name": "weiden", "author_url": "", "post_date": "04/24/2013 05:29:21", "content": ""I added a piece-wise linear fit plus orange intersections.
\r\nIf A -> B: f(A=130) = 0 and f(A=130) = 1500 (no)
\r\nIf B -> A: f(B=0) = 130 and f(B=1500) = 130 (yes)
\r\nDoes that help?"
\r\n\r\n
I don't think the argument is convincing. If we take a linear transformation of A such as 5A and keep B the same, then B=10 can lead to 5A=5*130=650 and 5A=5*160=800. Right?
", "votes": null, "replies": [] }, { "id": 23404, "author_name": "transfunctioner", "author_url": "", "post_date": "04/24/2013 08:43:25", "content": "Argument is convincing.
\r\nIf we take a linear transformation of A such as 5A and keep B the same, then
\r\nIf A -> B: f(A=650) = 0 and f(A=650) = 1500 (no)\r\nIf B -> A: f(B=0) = 650 and f(B=1500) = 650 (yes)\r\n\r\n
@Wei Deng: You take A=5*160 but A is never 160 on the original red line, especially not for B=10.
", "votes": null, "replies": [] }, { "id": 23424, "author_name": "weiden", "author_url": "", "post_date": "04/24/2013 17:40:47", "content": "I think I must have missed something. It is true that (A=130, B=0) and (A=130, B=1500) are two data points; however, according to the plot, we could also find two points that are on the same horizon, i.e., two data points sharing the same B value but with\r\n different A values. Right?
", "votes": null, "replies": [] }, { "id": 23431, "author_name": "transfunctioner", "author_url": "", "post_date": "04/24/2013 20:21:13", "content": "That's not what I am seeing. For what B value would that occur?
", "votes": null, "replies": [] }, { "id": 23434, "author_name": "bcragin", "author_url": "", "post_date": "04/24/2013 20:44:03", "content": "Sticking my nose in here, this being an idea I hinted at earlier. It's arguable I think that multiple A-function values could be ocurring even at one or the other of the two B values (roughly 200 and 1500) highlighted in Isabelle's figure. To know for sure,\r\n wouldn't you have to be able to separate the signal from the noise? The working assumption seems to be that the "true" function is "smooth" and "simple", and the dispersion of the points is a good indicator of the noise level. Seems to me it's that somewhat\r\n imprecise assumption that leads you to dismiss the multiple values as noise in one case but not in the other.
", "votes": null, "replies": [] }, { "id": 23439, "author_name": "weiden", "author_url": "", "post_date": "04/24/2013 22:54:31", "content": "@Steven: If you draw a horizontal line that is slightly above the line B=0, then it is evident that that line touches at least two data points.
\r\n@Bruce: "And the dispersion of the points is a good indicator of the noise level." Because both A and B are free to take a linear transformation of themselves, there is no metric, i.e., we cannot say if 500 is a big number or not.
", "votes": null, "replies": [] }, { "id": 23449, "author_name": "transfunctioner", "author_url": "", "post_date": "04/25/2013 08:49:48", "content": "@Wei Deng: Oh yes, you're right. If you're talking about the data, then there are values of B for which there are several values of A. But the point in this thread is that you can approximate the data by some lines like the red ones, and on the red lines\r\n there is no point of B for which there are multiple possibilities for A.
", "votes": null, "replies": [] }, { "id": 23450, "author_name": "weiden", "author_url": "", "post_date": "04/25/2013 09:17:09", "content": "@Steven: Now I see. Thank you!
\r\nSomehow I feel that the context for this project is insufficient. So far I doubt the validity of this project, but I could be wrong.
", "votes": null, "replies": [] }, { "id": 23452, "author_name": "transfunctioner", "author_url": "", "post_date": "04/25/2013 10:03:17", "content": "I think the rules ("other rules") are rather confusing. I may open another thread for that.
", "votes": null, "replies": [] }, { "id": 23455, "author_name": "domcastro", "author_url": "", "post_date": "04/25/2013 13:16:59", "content": "Blimey - just read the other rules! Very harsh to expect people to write a 6 page paper just to be considered for the prizes! Don't mind submitting a model / software but won't be writing a paper unless I'm in top 3 (which we won't know until the conference)
", "votes": null, "replies": [] }, { "id": 23464, "author_name": "bacobits", "author_url": "", "post_date": "04/25/2013 15:11:09", "content": "Perhaps I'm missing something, but doesn't this "invertible function" framework for determining causality break down with bimodal distributions? A simple example would be androgen levels as a function of age (sex unknown). I'm assuming levels would be similar\r\n pre-puberty and diverge in males vs femles post-puberty, yielding a pretty clear counterexample...
", "votes": null, "replies": [] }, { "id": 23477, "author_name": "weiden", "author_url": "", "post_date": "04/25/2013 20:10:28", "content": "For a dataset that both A and B are numerical variables, I think at best we can ask if there is some reasonable function f(x, y) such that "up to noise", that f(A, B) = 0 fits the data and can be used for prediction. The question of whether one causes another\r\n seems impractical, especially if there is no background information of a dataset and no units for observations.
", "votes": null, "replies": [] }, { "id": 23484, "author_name": "bacobits", "author_url": "", "post_date": "04/25/2013 22:01:41", "content": "In fact, to generalize the puberty counterexample above, it seems that anytime there is an interaction between one of the known variables and a third unknown variable, a similar noninvertible function can easily arise, regardless of causality.
\r\nThese are elementary examples, and could be applied to the temperature-altitude problem, e.g. samples come from a random selection of cities in Germany and Sweden. The scatterplot would be two decreasing lines + noise, noninvertible with respect to\r\n either variable if there was any overlap when projected onto the X or Y axes respectively. This is because a third variable (latitude) interacts with height (same height is colder in northern Sweden than southern Germany).
\r\nIf we know what the variables represent, we can (potentially) make assumptions or modify our model in ways that solve this problem, but when we don't know the back story, we're helpless. Unless we can somehow be guaranteed that NO interactions are possible\r\n with ANY unknown variables.
", "votes": null, "replies": [] }, { "id": 23503, "author_name": "davidwilson0", "author_url": "", "post_date": "04/26/2013 06:34:25", "content": "Very interesting challenge. I'll be interested to see if there is success in developing these algorythms (both in and outside of this challenge).
\r\nI'd like to mention how your German city example does illustrate how hard this is. The challenge states that altitude causes temperature to decrease.
\r\nI'd like to suggest that there is not a single causal relationship between temperature and altitude. Both temperature and pressure are influenced by the forces gravity and fluid mechanics. Temperature is also influenced by the greenhouse effect.
\r\nOn my first reflection, saying that altitude causes lower temperatures is an example of the Texas Sharp Shooter fallacy. Identifying the positions where temperature will be higher because of other forces is a bit of drawing\r\n the target around the bullethole.
", "votes": null, "replies": [] }, { "id": 23531, "author_name": "dlehman", "author_url": "", "post_date": "04/26/2013 16:49:50", "content": "I will also throw in something that bothers me about this example. I get the non-invertibility point, but couldn't this example pattern easily arise if there is an underlying time variable that causes both A and B? The graph looks much like what you get\r\n if you plot unemployment rates and job openings. In earlier times, there was a clear inverse relationship. In the last recession, job openings have risen with little reduction in unemployment (this will give you two predicted unemployment rates corresponding\r\n to a single job opening rate). There is no apparent causation from A to B or vice versa, but an underlying time structure that is affecting both.
\r\nPerhaps I don't understand the task. Is it to come up with a classification scheme that is most likley to reflect the actual (known, in this case) causative relationship even if there are multiple possible relationships if we don't know what was really\r\n used? If so, I suppose it could be an interesting exercise, but somewhat unclear how useful it is to me. Wouldn't it depend on how complex the underlying structure was? For example, a classification scheme might be successful provided that there is no time\r\n variable, or provided that there is only a single variable correlated with both A and B, but might not work as well in a more complicated structure. How useful would the results be in that case?
\r\nI could be missing something fundamental, but I don't see how non-invertibility implies the diredtion of causation. Can someone help me here?
", "votes": null, "replies": [] }, { "id": 23535, "author_name": "sankalp", "author_url": "", "post_date": "04/26/2013 19:25:23", "content": "I am assuming that the task is to come up with a model for a very narrow definition of 'causality' - the relationship is direct and there are no other factors involved or they are controlled to be the same. So, in the economy example more job openings could\r\n 'cause' unemployment rate to go down if other things like the pool of workers, suitability for job, salary, willingness to work etc are the same or controlled.
\r\nIf the definition of causality is different, the task wouldn't make sense.
", "votes": null, "replies": [] }, { "id": 23538, "author_name": "bacobits", "author_url": "", "post_date": "04/26/2013 22:56:12", "content": "I would have immediately rejected this notion that invertibility has any relationship with causality were it not for the credentials of some of the proponents of this idea. Sankalp and previous posts have identified assumptions that seem necessary (although\r\n I'm not convinced of their sufficiency) for this framework to be meaningful. However, these assumptions are very restrictive and seem to be in direct conflict with the stated goal of searching large complex observational datasets in which variables have not\r\n been carefully controlled. In fact, it seems like the rigorously controlled scenario described by Sankalp et al. would be very unlikely to arise except in controlled experiments, which would obviously defeat the point, or in situations that are so simple these\r\n methods wouldn't be needed.
\r\nI would like to hear the competition administrator's response to the issues discussed in the past few posts, especially the assertion I described above that an interaction with a 3rd variable can yield noninvertible functions that are clearly unrelated to\r\n causality. This flaw seems to cast doubt on this entire endeavor.
", "votes": null, "replies": [] }, { "id": 23573, "author_name": "jay670790", "author_url": "", "post_date": "04/28/2013 17:15:45", "content": "\r\n
Look, I think the answer is simple: in general no, it's not possible.
\r\nBut... suppose that the admins promise you that the data come from a restricted set of possibilities, such that you can be sure a causal relationship exists each time a variable x is larger than a variable y. Then the task would be both possible\r\n and easy: just make your algorithm looks at whether you can be confident one variable is larger.
\r\nFrom that our task is clear: to figure out, or to guess, what the admins actually promised, and to design an algorithm able to detect the consequences of these promises.
\r\nIn the long run, what admins have in mind is that it may be that the set of naturely occuring data can more often than not be caracterized as a restricted set of possibilities, such that causal relationships more often than\r\n not let some imprint in the data. Whether it's a correct view or a dead end is by itself an interesting empirical question, however beyond the scope of the present contest.
", "votes": null, "replies": [] }, { "id": 23575, "author_name": "bacobits", "author_url": "", "post_date": "04/28/2013 17:51:07", "content": "Jay, you are misunderstanding the competition, and perhaps the meaning of causality when you state that
\r\n[quote=Jay;23573]
\r\nyou can be sure a causal relationship exists each time a variable x is larger than a variable y
\r\n[/quote]
\r\nThe competition is about inferring causality based on any applicable methods, but the prevailing strategy seems to be based on invertible vs noninvertible functions. I have shown in previous posts on this thread that this framework may have fundamental flaws\r\n and would be interested in having a discussion about my concerns.
", "votes": null, "replies": [] }, { "id": 23577, "author_name": "dlehman", "author_url": "", "post_date": "04/28/2013 18:15:28", "content": "ahfoss
\r\nI'm with you and would still like to see more discussion. I understand invertibility but do not see that as necessary or sufficient to establish causality. If there are further restrictions imposed about the potential nature of the relationships (such\r\n as limits on feedback loops, lack of time series, lack of additional factors, etc.) then I can see invertibility becoming a way to classify the causal relationships. But, then I fail to see the usefulness of this competition for shedding light on observed\r\n data - surely no such restrictions are imposed on observed data.
\r\nI want to avoid the jargon (because I quickly won't understand it myself), but I did take the initial quiz listed on the competition and got the correct answer. It was easy given the clue about invertibility. However, the more I thought about it, the less\r\n I believed it really was the correct answer. That example reminded me of this data:
\r\nwww.businessweek.com/articles/2013-02-07/the-rise-of-long-term-joblessness
\r\nAccording to the competition example, it should be the case that long-term unemployment rate causes the job openings rate but obviously both are caused by other factors. So, what am I missing?
", "votes": null, "replies": [] }, { "id": 23578, "author_name": "jay670790", "author_url": "", "post_date": "04/28/2013 18:25:34", "content": "Ahfoss, sorry I was unclear, but you are confusing "suppose that the admins promise that..." with "Jay states that...". Dude, were you\r\nreally believing I understand causality as "one variable must be larger than the other"! :-)
\r\n\r\n
Ok let me rephrase:
\r\nYour concern (and other similar (such as Dale's)) is perfectly valid in general.\r\n
\r\nIt probably does not apply to the data of this contest.\r\n
\r\nWhether it applies to the set of naturally occuring data is an interesting empirical question, imho beyond the scope of this contest.
\r\n\r\n
Regards,
", "votes": null, "replies": [] }, { "id": 23579, "author_name": "bacobits", "author_url": "", "post_date": "04/28/2013 18:43:22", "content": "Dale, I don't think you're missing anything. I think that if we're only given two variables and no subject matter knowledge, it's *possible* that under very restrictive conditions there could be a relationship between invertibility and causality, but I'm\r\n still not convinced that this is possible. Even if it is possible, the amount of knowledge required to make those assumptions would probably mean we already know enough to identify causality outright, without needing to rely on something as tenuous as the\r\n functional form of the data.
\r\nNow, if we suppose that we have a multivariate dataset that is rich enough to include all relevant variables, then the problem may be tractable with established causal modeling techniques... but the more I think about this "blind" bivariate competition the\r\n more it seems to lack validity.
\r\nInferring causality from unknown bivariate datasets would be revolutionary of the "holy-grail" sort, so I think the burden of proof rests squarely on the shoulders of its proponents. I invite them to chime in here and defend the invertibility method. Perhaps\r\n we're both missing something?
", "votes": null, "replies": [] }, { "id": 23580, "author_name": "bacobits", "author_url": "", "post_date": "04/28/2013 18:55:45", "content": "Jay, I apologize for the misquote. I'll remove it if you want me to.
\r\nSo if I understand you correctly, you are saying that the competition admins claim that attributes XYZ imply causality, and we are to take that for granted and simply search for XYZ.
\r\nI am saying that XYZ does not imply causality. This seems like a very important question, in fact, a fundamental one. Not for purposes of scoring well in the competition, of course, but to verify that this competition is not based on misguided statistical\r\n assertion.
", "votes": null, "replies": [] }, { "id": 23590, "author_name": "jay670790", "author_url": "", "post_date": "04/28/2013 20:36:57", "content": "No problem, I know my english is not as clear as I'd like.\r\n
\r\nYes, thanks for the rephrasing: we have to guess which attributes XYZ admins think as implying causality (or, more properly, what attributes XYZ imply causality for the set of data the admins selected), and yes non invertibility\r\n seems one of these features, and no there is no guarantee it is true in general. To take the temperature/altitude exemple, the apparence of non invertibility could well be explained\r\n by a third variable, say city location as inland versus seaside (to be honnest I think this is the case and that this example is terribly misguided*).
\r\nBut I won't say overall the contest is based on misguided statistical assumption, first because admins "are not really trying to infer causality here. The gold standard is still experiments.\r\n We are interested in ranking pairs of variables {A, B} to prioritize experiments.", and second because it might be possible that some of the tricks they have in mind actually work more often than not for real data (or, more subttle, for real data humans tend\r\n to record).
\r\n\r\n
\r\n
* Edit: to illustrate why, see the plot attached. It's a series of south american cities chosen est or west of the Andes (namely, a third variable impact the relationship between temperature and altitude). We'd be supposed to conclude temperature cause altitude...
\r\n\r\n
Edit2: if real data is not convincing enough, here's a noise free model where one is supposed to conclude that B cause A. Actually A and a third variable C cause B.
\r\n", "votes": null, "replies": [] }, { "id": 23594, "author_name": "edge219878", "author_url": "", "post_date": "04/28/2013 21:46:54", "content": "
I'll stick my nose back in here with a different example considering 2 functions:
\r\n1) B = A and 2) B = A^2
\r\nBoth function are noisless and in both cases B is dependent on A. The invertibility of (1) makes it impossible to determine the direction of the dependence and the noninvertibility of (2) makes it possible. So we can say that a) it's not always possible\r\n to determine the direction of dependence, and b) it's sometimes possible to determine the direction of dependence. That's one of the claims of the background material: noninvertibility adds information even in the absense of noise.
\r\nNotice how I avoided the dreaded 'c' word. The possibility that A and B are both dependent on an unobserved variable C is a separate problem from the one of determining the direction of observable dependence. I think this competition will be won by the\r\n team that best quantifies that possibility.
\r\nEdit: I screwed up the functions when I first posted
", "votes": null, "replies": [] }, { "id": 23596, "author_name": "bacobits", "author_url": "", "post_date": "04/28/2013 21:56:14", "content": "Jay, you bring up excellent points. I'm glad I understand your points better now.
\r\nSo yes, on the one hand, if it works, it works! The proof will be in the pudding (as they say). However, to be taken seriously by my fellow statisticians, it will have to be rigorously proven why and when this method works. The fact that even the defining\r\n example (temp-altitude) seems flawed to you, me, and others in this thread is not encouraging. Can anybody think of a single example that actually works, that we can't similarly deconstruct and dismiss? If there is some pattern that arises in "typical" examples\r\n collected by researchers, what could this be, and when does it arise? These are interesting and important questions.
\r\nBut, you correctly point out that the admins have hedged, and are "not really trying to infer causality," but they then describe a procedure that is textbook statistical inference: drawing general conclusions (i.e. direction of causality) from specific\r\n observations (data), with associated measures of uncertainty (the ranking/prioritizing business). So let's just be clear that, although we are not making\r\nabsolute statements, we are most definitely inferring causality.
\r\nBut theoretical concerns aside, this just doesn't pass the smell test for me. In a recent Monte Carlo simulation I ran, a bivariate scatter plot of the two variables of interest resembled four intersecting nonlinear curves plus noise, due to interactions\r\n with a few other variables of itnerest. It doesn't get any less invertible than this, and it was a relatively simple epidemiological simulation with a simple causal structure.
", "votes": null, "replies": [] }, { "id": 23597, "author_name": "herrahuu", "author_url": "", "post_date": "04/28/2013 22:02:20", "content": "On the Identifiability of the Post-Nonlinear Causal Model: http://arxiv.org/ftp/arxiv/papers/1205/1205.2599.pdf
\r\nThis article seems to be giving the conditions when the causal direction is identifiable. (disclaimer: just found it, haven't fully read/understood all of it)
", "votes": null, "replies": [] }, { "id": 23598, "author_name": "bacobits", "author_url": "", "post_date": "04/28/2013 22:14:32", "content": "This is getting fun.
\r\nJay now has evidence, according to the invertibility heuristic, that temperature causes altitude. See his post above. I'm assuming the quadratic pattern would hold with more data, but even if it doesn't, we can easily imagine situations in which it would.
\r\nEdge2, assuming you mean "statistical dependence" when you say dependence (which by definition is not directional i.e. symmetric), it seems that you are just misusing the word as a surrogate for the "dreaded 'c' word." Also, Jay's graph invalidates a quadratic\r\n function as implying causality, unless you believe his evidence that temperature causes altitude.
", "votes": null, "replies": [] }, { "id": 23601, "author_name": "bacobits", "author_url": "", "post_date": "04/28/2013 22:42:54", "content": "Herra posts an interesting paper; from my very brief skimming it looks like they are treating invertibility as a given and not a measurable index of causality (e.g. in equations 1, 2, and 3); they also give certain combinations of distributions and functional\r\n forms for which their PNL model cannot identify causality. Although not particularly relevant to our discussion of noninvertibility (I think), it seems like a potentially promising avenue to pursue for competitors.
\r\nLooking through the references led me to this paper which may be useful to some competitors (looks like the additive noise model outperforms the others):
\r\nhttp://www.is.tuebingen.mpg.de/fileadmin/user_upload/files/JMLR-2010-Mooij.pdf
\r\nAs well as this website which seems to be the origin of the altitude-temperature example:
\r\nhttp://people.kyb.tuebingen.mpg.de/jorism/causality-data/
", "votes": null, "replies": [] }, { "id": 23602, "author_name": "herrahuu", "author_url": "", "post_date": "04/28/2013 22:51:00", "content": "[quote=ahfoss;23601]
\r\nAlthough not particularly relevant to our discussion of noninvertibility (I think), it seems like a potentially promising avenue to pursue for competitors.
\r\n[/quote]
\r\nTheir model is a bit more general than the ones thought here (ie. they include noise and invertible post-nonlinear distortion f2), but I think their results are relevant also to this noninvertibility (of f1) discussion. See corollary 10.
", "votes": null, "replies": [] }, { "id": 23603, "author_name": "edge219878", "author_url": "", "post_date": "04/28/2013 23:01:34", "content": "ahfoss, I'm not talking about statistical dependence, I'm talking about B being the dependent variable. But you're right that I'm using it just to avoid using cause/effect. I thought it would make it easier to make the point but that was misguided. Let's\r\n try this again:
\r\n1) B = A, 2) B = A^2
\r\nIn 1), it is impossible to determine the causal direction because the function is invertible. In 2), B cannot cause A because the function is not invertible. Again, the claim is simply that noninvertibility adds information even in the absense of noise. I'm\r\n not sure how you can determine that this claim is somehow invalidated by a few noisy samples.
", "votes": null, "replies": [] }, { "id": 23604, "author_name": "edge219878", "author_url": "", "post_date": "04/28/2013 23:37:43", "content": "Here's a few more thoughts before I bow back out of this discussion. I somehow keep forgetting how much back and forth is required before it is even clear what I'm talking about. What starts as a 30 minute diversion quickly turns into an entire afternoon.\r\n
\r\nFirst, I say 'invertibility adds information even in the absense of noise' because a lot of methods require noise to determine causal direction. I forgot to mention that above. Second, I think an important point that is easy to overlook is that a lot of\r\n published methods only separate A->B from B->A. They don't consider the cases A-B and A|B. In other words, they take data that is assumed to be causal, then separate it into the cause and effect. The problem we face in this\r\n competition is much harder than that because we also need to separate (A->B,B->A) from A|B and A-B.
\r\nEdit: (sigh... I thought I was done :) )
\r\nIt's possible that invertibility only adds information when separating A->B from B->A, and may be meaningless in separating (A->B, B->A) from A-B. That is, if we know that A and B are causally related, invertibility is an\r\n important clue that can help separate the cause from the effect, but that may not be the case when we are trying to determine if the data are causally related or not.
", "votes": null, "replies": [] }, { "id": 23606, "author_name": "dlehman", "author_url": "", "post_date": "04/29/2013 00:15:00", "content": "I thought I was done as well - but I think I agree with your last statements (Edge2). If that is correct, then it is the example they provide that is really misleading. It is in the video as well which makes it doubly misleading.
", "votes": null, "replies": [] }, { "id": 23617, "author_name": "bacobits", "author_url": "", "post_date": "04/29/2013 12:50:34", "content": "Edge2, I think I understand you better now! Thanks for clarifying. I think that I agree with your statement that invertibility matters if we are discriminating between A->B and B->A ONLY, i.e. no other variables, no feedback loops, etc. This dovetails with\r\n our previous discussion, and I think we're approaching a unanimous forum-wise convergence on these very strong assumptions for invertibility to mean anything at all.
\r\nNevertheless, these assumptions only seem to be met in very unrealistic conditions, or in controlled experiments, so I still don't see how this invertibility stuff stands a chance of being useful.
\r\nIn our discussions, we have identified multiple papers that seem to hold much more potential (see previous posts).
\r\nIt seems that we are also near unanimity in deciding that the temperature altitude example is bunk. Jay has constructed "proofs" by invertibility that temperature "causes" altitude, and I and others have pointed out that a third variable can easily yield\r\n noninvertible functions in variables that are clearly causal (see previous posts).
\r\nThe competition admins MUST either explain where we are wrong, or apologize for the misleading example. I love the idea of Kaggle and the community that has arisen around it, and I strongly feel that Kaggle will lose credibility if shoddy methods/logic become\r\n commonplace. We must raise our concerns when questionable assertions are made!
\r\nI've got a busy 2 weeks ahead, so I too, must bow out for now. Wei D., Dale L., Sankalp, Jay, Edge2, Herra H.: It has been my pleasure to discuss these interesting ideas with clever and thoughtful people. Best of luck.
", "votes": null, "replies": [] }, { "id": 23633, "author_name": "weiden", "author_url": "", "post_date": "04/29/2013 16:13:00", "content": "Thank you for your guys' input on this! On the other hand, my philosophy hasn't changed so far because some essential questions are still there. One thing is identifiability, i.e., {A, B} should contain exactly the same amount of information as {A', B'}\r\n if A' (B') is a linear transformation of A (B). (Here A, B are numerical variables without units.) Do you guys agree? So any acceptable method should give exactly the same judgement on {A, B} and {A', B'}. It follows we have to standardize A and B to have\r\n say mean 0 and variance 1.
\r\nOne thing that MIGHT make this project valid is that we are only required to infer whether or not A is caused by B (more precisely I should say whether or not A is a function of B) as opposed to which function it is.
\r\nIn terms of function, in my opinion, we should ask the following question instead, namely, whether or not there is a reasonable bivariate function f(x,y) such that f(A,B) = 0 fits the data. While y = x^2 is a perfect function, x = plus/minus y^0.5 is also\r\n a very reasonable function although it is not invertible. We cannot conclude if something is more fundamental (here cause is more fundamental than effect) just based on this "dis-invertibility".
\r\nJust my 2 cents.
", "votes": null, "replies": [] }, { "id": 23957, "author_name": "iguyon", "author_url": "", "post_date": "05/06/2013 22:08:29", "content": "I produced a new versions of my slides with voice over:
\r\nSlide presentation: overview of the challenge (pptx with voice over, May 1st version).
\r\nThe presentation lasts about 1/2 hour, but you can skip over things you know already (particulalry if you watched the video because all the beginning is the same). It starts being relevant to this discussion at slide 12. I explain methods people have been\r\n using to orient the causal direction between pairs of dependent variables using additive noise models, latent variable models, and information theoretic methods (this is very qualitative, you need to look at the papers to really get the formal approach).
", "votes": null, "replies": [] }, { "id": 23958, "author_name": "iguyon", "author_url": "", "post_date": "05/06/2013 22:15:10", "content": "I am organizing a series of teleconference seminars so you can ask questions to the experts:
\r\nhttps://sites.google.com/a/chalearn.org/readinggroup/
\r\nI invited Dominik Janzig, Joris Mooij and Kun Zhang to talk to you about addiive noise models, latent variable models and information theoretic approaches. We will meet every Thursday at 9 am PT for 1 hour by teleconference:
\r\nConference Dial-in Number: +1 (610) 214-0000
\r\nParticipant Access Code: 549086#
This week (May 9) I will just go over my slide deck so you can ask me questions.
", "votes": null, "replies": [] }, { "id": 23973, "author_name": "bacobits", "author_url": "", "post_date": "05/07/2013 03:44:11", "content": "Thanks for the reply, Isabelle. I look forward to perusing the slides once my schedule simmers down. The conference call sounds great, but it's a difficult time to dial in for those of us who work 9-5 anywhere from PST to EST. Forum posts are great since\r\n we can access them any odd time we may have to spare... e.g. now, quarter tomidnight :).
\r\nPerhaps your colleagues could start a separate string of posts, or maybe post a transcript of the conf. call proceedings?
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "21983": "", "22015": "", "22120": "", "22129": "", "22130": "", "22135": "", "22580": "", "22583": "", "22584": "", "22790": "", "22797": "", "22803": "", "22817": "", "23399": "", "23404": "", "23424": "", "23431": "", "23434": "", "23439": "", "23449": "", "23450": "", "23452": "", "23455": "", "23464": "", "23477": "", "23484": "", "23503": "", "23531": "", "23535": "", "23538": "", "23573": "", "23575": "", "23577": "", "23578": "", "23579": "", "23580": "", "23590": "", "23594": "", "23596": "", "23597": "", "23598": "", "23601": "", "23602": "", "23603": "", "23604": "", "23606": "", "23617": "", "23633": "", "23957": "", "23958": "", "23973": "" }, "source": "meta" }