{
  "id": 5055,
  "title": "bias in original data",
  "url": "/competitions/cause-effect-pairs/discussion/5055",
  "author_name": "",
  "post_date": "2013-07-09T19:00:10.963Z",
  "votes": null,
  "comment_count": 3,
  "views": 1278,
  "content": "<p>I would appreciate an explanation of the bias in the original data, and the changes made with the final data release.</p>",
  "messages": [
    {
      "id": "26899",
      "postDate": "07/09/2013 19:00:10",
      "content": "<p>I would appreciate an explanation of the bias in the original data, and the changes made with the final data release.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "27007",
      "postDate": "07/11/2013 10:48:19",
      "content": "<p>What I mean to say is, maybe I don't have the standing to ask because I haven't made a submission yet, but I'm working on one, and the bias in the original data caused my model to act funny, so I'm interested in what it was. &nbsp;Are you going to describe it\r\n when the competition is over, or is it already described somewhere? &nbsp;I'm interested in what were the differences between the first and second data sets, from your point of view. &nbsp;</p>\r\n<p>I didn't mean to sound ominous. &nbsp;I'm really grateful to whoever told you about the problem, and to you for issuing a new data set. &nbsp;You have created a wonderful thing here. &nbsp;Getting one file is hard, you've gotten thousands - you're a hero and I hope everybody\r\n else is having as much fun as I am. &nbsp;</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "27036",
      "postDate": "07/11/2013 21:33:42",
      "content": "<p>In the original data, the mean, max and other statistics of the variables were not uniformly distributed across the 4 causal classes. Since those statistics are not expected to carry causal information, most programs start normalizing variables. To avoid\r\n that programs training on the data we provided rely on such spurious features, the new version of the data standardizes the variables (subtract the mean and divide by the standard deviation).</p>\r\n<p>Another problem with the original data is the the number of unique values of variable was not distributed uniformly across all 4 causal classes. This is a less obvious problem because even real data show some uneven distribution of unique values and there\r\n is no simple fix. However, we devised a method to roughly even out the distribution of unique values across the 4 classes. Furthermore, the difference in number of unique values between variable A and B in artificial and real data look similar now.</p>\r\n<p>Of course, there may still be other less obvious spurious statistics. But we made available sample data (SUP2) generated with the new scheme for a month before we issued the final data and noone reported anything obvious, so hopefully, if there remain artifacts,\r\n they are not too bad. The SUP3 data uses a slightly different scheme that we devised for real data.</p>\r\n<p>Because some participants complained bitterly that we change the data, we committed not to change it anymore until the end of the challenge. However, we encourage people to report problems they may find to enrich the analysis and help us design better datasets\r\n in the future.</p>\r\n<p>Thank you for your participation and good luck!</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "27040",
      "postDate": "07/11/2013 23:51:03",
      "content": "<p>Thanks Isabelle. &nbsp;My view is, nothing new ever works, glitches are to be expected, and issuing new data was totally the right decision.</p>",
      "rawMarkdown": "",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 27007,
      "author_name": "waynehaythorn",
      "author_url": "",
      "post_date": "07/11/2013 10:48:19",
      "content": "<p>What I mean to say is, maybe I don't have the standing to ask because I haven't made a submission yet, but I'm working on one, and the bias in the original data caused my model to act funny, so I'm interested in what it was. &nbsp;Are you going to describe it\r\n when the competition is over, or is it already described somewhere? &nbsp;I'm interested in what were the differences between the first and second data sets, from your point of view. &nbsp;</p>\r\n<p>I didn't mean to sound ominous. &nbsp;I'm really grateful to whoever told you about the problem, and to you for issuing a new data set. &nbsp;You have created a wonderful thing here. &nbsp;Getting one file is hard, you've gotten thousands - you're a hero and I hope everybody\r\n else is having as much fun as I am. &nbsp;</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 27036,
      "author_name": "iguyon",
      "author_url": "",
      "post_date": "07/11/2013 21:33:42",
      "content": "<p>In the original data, the mean, max and other statistics of the variables were not uniformly distributed across the 4 causal classes. Since those statistics are not expected to carry causal information, most programs start normalizing variables. To avoid\r\n that programs training on the data we provided rely on such spurious features, the new version of the data standardizes the variables (subtract the mean and divide by the standard deviation).</p>\r\n<p>Another problem with the original data is the the number of unique values of variable was not distributed uniformly across all 4 causal classes. This is a less obvious problem because even real data show some uneven distribution of unique values and there\r\n is no simple fix. However, we devised a method to roughly even out the distribution of unique values across the 4 classes. Furthermore, the difference in number of unique values between variable A and B in artificial and real data look similar now.</p>\r\n<p>Of course, there may still be other less obvious spurious statistics. But we made available sample data (SUP2) generated with the new scheme for a month before we issued the final data and noone reported anything obvious, so hopefully, if there remain artifacts,\r\n they are not too bad. The SUP3 data uses a slightly different scheme that we devised for real data.</p>\r\n<p>Because some participants complained bitterly that we change the data, we committed not to change it anymore until the end of the challenge. However, we encourage people to report problems they may find to enrich the analysis and help us design better datasets\r\n in the future.</p>\r\n<p>Thank you for your participation and good luck!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 27040,
      "author_name": "waynehaythorn",
      "author_url": "",
      "post_date": "07/11/2013 23:51:03",
      "content": "<p>Thanks Isabelle. &nbsp;My view is, nothing new ever works, glitches are to be expected, and issuing new data was totally the right decision.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "26899": "",
    "27007": "",
    "27036": "",
    "27040": ""
  },
  "source": "meta"
}