{
  "id": 4531,
  "title": "Report data artefacts",
  "url": "/competitions/cause-effect-pairs/discussion/4531",
  "author_name": "",
  "post_date": "2013-05-08T15:37:17.803Z",
  "votes": null,
  "comment_count": 11,
  "views": 3126,
  "content": "<p>As you know, part of the data of the challenge is real data and part of it is artificial data. Both types of data have advantages and disadvantages:</p>\r\n<p>- Real data are less prone to artefacts, but difficult to obtain and the truth value of the causal relationships is uncertain.&nbsp;</p>\r\n<p>- Artificial data provide us with full control over signal/noise ratio, number of samples, and we know the truth values of causal relationships by design, but there is a danger that we inadvertently introduce artefacts.</p>\r\n<p>So results if results are good on both types of data, this is a reassuring indication that the methods actually work. Our analysis of the validation data results indicates that this is the case. However, for some participants, there is a larger difference\r\n in performance between real data and artificial data than for others. We invite you to help troubleshooting the problem by reporting anything you find that may be indicative of a possible artefact in data. For instance, one participant found that&nbsp;there is\r\n an imbalance in the distribution of values of the variables in the various classes.</p>\r\n<p>You may either reply to this post or write to causality at chalearn do org.</p>\r\n<p>If you think that the final evaluation should be done only on real data, you may also comment on this.</p>\r\n<p>Thank you in advance for your help. Everyone will benefit from a fair and sound evaluation.</p>",
  "messages": [
    {
      "id": "24015",
      "postDate": "05/08/2013 15:37:17",
      "content": "<p>As you know, part of the data of the challenge is real data and part of it is artificial data. Both types of data have advantages and disadvantages:</p>\r\n<p>- Real data are less prone to artefacts, but difficult to obtain and the truth value of the causal relationships is uncertain.&nbsp;</p>\r\n<p>- Artificial data provide us with full control over signal/noise ratio, number of samples, and we know the truth values of causal relationships by design, but there is a danger that we inadvertently introduce artefacts.</p>\r\n<p>So results if results are good on both types of data, this is a reassuring indication that the methods actually work. Our analysis of the validation data results indicates that this is the case. However, for some participants, there is a larger difference\r\n in performance between real data and artificial data than for others. We invite you to help troubleshooting the problem by reporting anything you find that may be indicative of a possible artefact in data. For instance, one participant found that&nbsp;there is\r\n an imbalance in the distribution of values of the variables in the various classes.</p>\r\n<p>You may either reply to this post or write to causality at chalearn do org.</p>\r\n<p>If you think that the final evaluation should be done only on real data, you may also comment on this.</p>\r\n<p>Thank you in advance for your help. Everyone will benefit from a fair and sound evaluation.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "24018",
      "postDate": "05/08/2013 16:22:42",
      "content": "<p>If the final evaluation is real data, then the training and valid data should also only be real</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "24020",
      "postDate": "05/08/2013 16:53:11",
      "content": "<p>I'm inclined to agree. Alternatively, we could continue to use the current training and valid data sets but with the real/artificial status of each case being supplied to the user. The leaderboard would then need to provide separate scores for the real and\r\n artificial components. &nbsp;This would make the whole exercise a lot more meaningful to the competitors, many of whom were at least initially skeptical.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "24025",
      "postDate": "05/08/2013 17:24:22",
      "content": "<p><span>Hi all,</span></p>\r\n<div>I am taking part in this competition because I am interested in this scientific problem. Thus, I will support any measure taken to avoid us wasting our time discovering and exploding artefacts in artificial data.</div>\r\n<div>At this moment, if I find a feature that improves my score a lot, sometimes may be hard to know if I am exploding an artefact on artificial data or if I have found a feature that is present in nature. So, I think it may be useful if you let us know which\r\n of the training samples are real data and which of them are artificial data.</div>\r\n<div>Also, I will support&nbsp;re-issuing new &quot;fixed&quot; versions of the data (training and validation), otherwise the leaderboard will remain useless.</div>\r\n<div>Regarding the possibility of making the final evaluation only on real data: That is the whole point, isn't it? Predicting causality in real data. Thus, I agree with making the final evaluation only on real data, but, as I said before, you should let us\r\n know which of the training samples are real data.&nbsp;</div>\r\n<div>Bruce mentioned the possibility of keeping&nbsp;separate scores for the real and artificial components. I think that could be also a quite interesting option.</div>\r\n<div>Anyway, we are dealing with a very complex scientific problem, so I find completely normal to encounter this kind of issues. Let's do our best and see where we can get!</div>\r\n<div>Best.</div>\r\n<div>&nbsp;</div>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "24027",
      "postDate": "05/08/2013 17:45:55",
      "content": "<p>Ok - don't know if this helps as I don't know if I'm someone who has predicted the artificial data. I have just made 2 submissions (9a and 9b) 9a included number of unique values and the max values. 9b includes unique / total ratio only. 9a scored about\r\n 88.248 and 9b scored about 87.883. If submission 9b is better at predicting artificial data then there are more artefacts than just unique values and max value.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "24053",
      "postDate": "05/09/2013 05:41:48",
      "content": "<p>Thanks a lot, this is reassuring. What would also help is if you and other partiicipants would conduct an analysis of what their most informative features are. If one looks suspicious (that is if it does not make sense that this feature should be relevant\r\n to causal relationships), please post a note here or send email to causality at chalearn dot org. Even if we end up using only real data for the final ranking of participants, the artificial data will be valuable for training and for analyzing the methods\r\n because it covers systematically a wide variety of cases.</p>\r\n<p>Between now and May 17 (the track 1 deadline), I think people should focus on contributing data to track 1 and sending us feed-back on possible artefacts on the data we provided. We will not re-issue a new version of the data until then. There is a great\r\n potential in this challenge to do as much collaborative work as competing work and beat this problem together.</p>\r\n<p>Thank you for all the hard work you did so far and for the contributions you will make.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "24121",
      "postDate": "05/09/2013 21:22:35",
      "content": "<p>My 2 &quot;purest&quot; features according to the Random Forest is Mean of A and&nbsp; Mean of B. I removed them from the dataset and ran 9a (above) The score is 0.88205!</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "24122",
      "postDate": "05/09/2013 21:40:21",
      "content": "<p>What do you make of this?</p>\r\n<p>Here is one way in which you could help: try various ways of preprocessing the data such that the distribution of unique values and max value is homegeneized accross classes and see whether this deteriorates the results. If you find a way to do that without\r\n significantly deteriorating results, this would be a candidate to fix the problem we have. I find that this is not so trivial, but maybe I am on the wrong track.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "24123",
      "postDate": "05/09/2013 21:48:10",
      "content": "<p>Sorry, of course the mean plays a role similar to the max since the variables are scaled between 0 and max, so the mean also needs to be distributed homogeneously.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "24143",
      "postDate": "05/10/2013 07:30:03",
      "content": "<p>But all of these are standard variables - they would also be needed to classify the real data? Anyway, have you noticed the jumps in the leaderboard since the artefacts have been revealed?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "24172",
      "postDate": "05/11/2013 01:19:54",
      "content": "<p>It seems to make sense that all methods should return a coefficient that is invariant to variable scaling and shift. Hence, we can standardize all the training data to make sure that mean and scale are not used as features. We would subtract the mean, divide\r\n by the standard deviation, multiply by a large integer N and round to get a given precision. N=5000 seems to be adequate.</p>\r\n<p>In validation and test data, it may make sense NOT to standardize the data to check whether the methods actually implement scale and shift invariance.</p>\r\n<p>This standardization will not solve the problem of the number of unique values and other features of the marginal distributions of A and B that may betray the causal category. Due to the way the data are generated, it is difficult to make it that the marginal\r\n distributions are similar for all causal categories. And maybe they should not be similar. I checked that, in real data, when the values are quantized to a given number of unique values, most of the time both A and B are quantized in the same way (approximately\r\n the same number of unique values) and the distribution of differences in unique values between cause and effect is mostly symmetrical. My inclination is to construct artificial data in which the number of unique values is not informative of causal effect.\r\n It can be in the case of experimental data (typically in a planned experiments, measurements are repeated for the same input). But why should it be for observational data?</p>\r\n<p>Can you think of other invariants?</p>\r\n<p></p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "25467",
      "postDate": "06/12/2013 17:37:03",
      "content": "<p>I generated visualizations of the training data if anyone is interested.&nbsp; http://www.bakermountain.org/CEdata_train.pdf.gz</p>",
      "rawMarkdown": "",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 24018,
      "author_name": "domcastro",
      "author_url": "",
      "post_date": "05/08/2013 16:22:42",
      "content": "<p>If the final evaluation is real data, then the training and valid data should also only be real</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 24020,
      "author_name": "bcragin",
      "author_url": "",
      "post_date": "05/08/2013 16:53:11",
      "content": "<p>I'm inclined to agree. Alternatively, we could continue to use the current training and valid data sets but with the real/artificial status of each case being supplied to the user. The leaderboard would then need to provide separate scores for the real and\r\n artificial components. &nbsp;This would make the whole exercise a lot more meaningful to the competitors, many of whom were at least initially skeptical.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 24025,
      "author_name": "",
      "author_url": "",
      "post_date": "05/08/2013 17:24:22",
      "content": "<p><span>Hi all,</span></p>\r\n<div>I am taking part in this competition because I am interested in this scientific problem. Thus, I will support any measure taken to avoid us wasting our time discovering and exploding artefacts in artificial data.</div>\r\n<div>At this moment, if I find a feature that improves my score a lot, sometimes may be hard to know if I am exploding an artefact on artificial data or if I have found a feature that is present in nature. So, I think it may be useful if you let us know which\r\n of the training samples are real data and which of them are artificial data.</div>\r\n<div>Also, I will support&nbsp;re-issuing new &quot;fixed&quot; versions of the data (training and validation), otherwise the leaderboard will remain useless.</div>\r\n<div>Regarding the possibility of making the final evaluation only on real data: That is the whole point, isn't it? Predicting causality in real data. Thus, I agree with making the final evaluation only on real data, but, as I said before, you should let us\r\n know which of the training samples are real data.&nbsp;</div>\r\n<div>Bruce mentioned the possibility of keeping&nbsp;separate scores for the real and artificial components. I think that could be also a quite interesting option.</div>\r\n<div>Anyway, we are dealing with a very complex scientific problem, so I find completely normal to encounter this kind of issues. Let's do our best and see where we can get!</div>\r\n<div>Best.</div>\r\n<div>&nbsp;</div>",
      "votes": null,
      "replies": []
    },
    {
      "id": 24027,
      "author_name": "domcastro",
      "author_url": "",
      "post_date": "05/08/2013 17:45:55",
      "content": "<p>Ok - don't know if this helps as I don't know if I'm someone who has predicted the artificial data. I have just made 2 submissions (9a and 9b) 9a included number of unique values and the max values. 9b includes unique / total ratio only. 9a scored about\r\n 88.248 and 9b scored about 87.883. If submission 9b is better at predicting artificial data then there are more artefacts than just unique values and max value.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 24053,
      "author_name": "iguyon",
      "author_url": "",
      "post_date": "05/09/2013 05:41:48",
      "content": "<p>Thanks a lot, this is reassuring. What would also help is if you and other partiicipants would conduct an analysis of what their most informative features are. If one looks suspicious (that is if it does not make sense that this feature should be relevant\r\n to causal relationships), please post a note here or send email to causality at chalearn dot org. Even if we end up using only real data for the final ranking of participants, the artificial data will be valuable for training and for analyzing the methods\r\n because it covers systematically a wide variety of cases.</p>\r\n<p>Between now and May 17 (the track 1 deadline), I think people should focus on contributing data to track 1 and sending us feed-back on possible artefacts on the data we provided. We will not re-issue a new version of the data until then. There is a great\r\n potential in this challenge to do as much collaborative work as competing work and beat this problem together.</p>\r\n<p>Thank you for all the hard work you did so far and for the contributions you will make.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 24121,
      "author_name": "domcastro",
      "author_url": "",
      "post_date": "05/09/2013 21:22:35",
      "content": "<p>My 2 &quot;purest&quot; features according to the Random Forest is Mean of A and&nbsp; Mean of B. I removed them from the dataset and ran 9a (above) The score is 0.88205!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 24122,
      "author_name": "iguyon",
      "author_url": "",
      "post_date": "05/09/2013 21:40:21",
      "content": "<p>What do you make of this?</p>\r\n<p>Here is one way in which you could help: try various ways of preprocessing the data such that the distribution of unique values and max value is homegeneized accross classes and see whether this deteriorates the results. If you find a way to do that without\r\n significantly deteriorating results, this would be a candidate to fix the problem we have. I find that this is not so trivial, but maybe I am on the wrong track.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 24123,
      "author_name": "iguyon",
      "author_url": "",
      "post_date": "05/09/2013 21:48:10",
      "content": "<p>Sorry, of course the mean plays a role similar to the max since the variables are scaled between 0 and max, so the mean also needs to be distributed homogeneously.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 24143,
      "author_name": "domcastro",
      "author_url": "",
      "post_date": "05/10/2013 07:30:03",
      "content": "<p>But all of these are standard variables - they would also be needed to classify the real data? Anyway, have you noticed the jumps in the leaderboard since the artefacts have been revealed?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 24172,
      "author_name": "iguyon",
      "author_url": "",
      "post_date": "05/11/2013 01:19:54",
      "content": "<p>It seems to make sense that all methods should return a coefficient that is invariant to variable scaling and shift. Hence, we can standardize all the training data to make sure that mean and scale are not used as features. We would subtract the mean, divide\r\n by the standard deviation, multiply by a large integer N and round to get a given precision. N=5000 seems to be adequate.</p>\r\n<p>In validation and test data, it may make sense NOT to standardize the data to check whether the methods actually implement scale and shift invariance.</p>\r\n<p>This standardization will not solve the problem of the number of unique values and other features of the marginal distributions of A and B that may betray the causal category. Due to the way the data are generated, it is difficult to make it that the marginal\r\n distributions are similar for all causal categories. And maybe they should not be similar. I checked that, in real data, when the values are quantized to a given number of unique values, most of the time both A and B are quantized in the same way (approximately\r\n the same number of unique values) and the distribution of differences in unique values between cause and effect is mostly symmetrical. My inclination is to construct artificial data in which the number of unique values is not informative of causal effect.\r\n It can be in the case of experimental data (typically in a planned experiments, measurements are repeated for the same input). But why should it be for observational data?</p>\r\n<p>Can you think of other invariants?</p>\r\n<p></p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 25467,
      "author_name": "ralphw",
      "author_url": "",
      "post_date": "06/12/2013 17:37:03",
      "content": "<p>I generated visualizations of the training data if anyone is interested.&nbsp; http://www.bakermountain.org/CEdata_train.pdf.gz</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "24015": "",
    "24018": "",
    "24020": "",
    "24025": "",
    "24027": "",
    "24053": "",
    "24121": "",
    "24122": "",
    "24123": "",
    "24143": "",
    "24172": "",
    "25467": ""
  },
  "source": "meta"
}