{
  "id": 6758,
  "title": "Responses normalizations",
  "url": "/competitions/galaxy-zoo-the-galaxy-challenge/discussion/6758",
  "author_name": "",
  "post_date": "2014-01-06T14:13:40.390Z",
  "votes": null,
  "comment_count": 13,
  "views": 2559,
  "content": "<p>I have just started looking at the data, trying to understand the normalization of the 37 classes on the decision tree. For this purpose, I run a check on the solutions_training.csv file, just to see that I got the right constraints on the probabilities.</p>\n<p>The attached python script simply runs through each item and checks if the following identities hold (within a tolerance epsilon=1e-4):</p>\n<p><code>sum Class1 = 1.0</code></p>\n<p><code>sum Class2 = Class1.2</code></p>\n<p><code>sum Class3 = Class2.2</code></p>\n<p><code>sum Class4 = sum Class3</code><code></code></p>\n<p><code>sum Class5 = sum Class11 + Class4.2</code><code></code></p>\n<p><code>sum Class6 = 1.0</code><code></code></p>\n<p><code>sum Class7 = Class1.1</code></p>\n<p><code>sum Class8 = Class6.1</code><code></code></p>\n<p><code>sum Class9 = Class2.1</code><code></code></p>\n<p><code>sum Class10 = Class4.1</code><code></code></p>\n<p><code>sum Class11 = sum Class10</code></p>\n<p>where sum Class1 = Class1.1+Class1.2+Class1.3, sum Class2=Class2.1+Class2.2 and so on. I started placing what I expected to be a&nbsp;generous limit on the equalities above, like epsilon=1e-4, but surprisingly, out of 70948 items, there are a lot of violations, mostly for the normalization of Class8 and Class9 nodes.</p>\n<p>Here is the output of the attached python script:</p>\n<p><code>violations of Class2 constraint 64 times out of 70948</code><code></code></p>\n<p><code>violations of Class11 constraint 31 times out of 70948</code></p>\n<p><code>violations of Class8 constraint 981 times out of 70948</code></p>\n<p><code>violations of Class9 constraint 1693 times out of 70948</code></p>\n<p><code>violations of Class7 constraint 159 times out of 70948</code></p>\n<p><code>violations of Class4 constraint 11 times out of 70948</code></p>\n<p><code>violations of Class5 constraint 316 times out of 70948</code></p>\n<p><code>violations of Class10 constraint 296 times out of 70948</code></p>\n<p><code>violations of Class3 constraint 91 times out of 70948</code></p>\n<p>In some cases the violation error is as high as 1, and is generally of order 0.01.</p>\n<p>I would like to ask if I got something wrong in the normalizations above, I am bit confused.</p>\n<p>Thank you</p>\n<p>&nbsp;</p>\n<p>&nbsp;</p>",
  "messages": [
    {
      "id": "37031",
      "postDate": "01/06/2014 14:13:40",
      "content": "<p>I have just started looking at the data, trying to understand the normalization of the 37 classes on the decision tree. For this purpose, I run a check on the solutions_training.csv file, just to see that I got the right constraints on the probabilities.</p>\n<p>The attached python script simply runs through each item and checks if the following identities hold (within a tolerance epsilon=1e-4):</p>\n<p><code>sum Class1 = 1.0</code></p>\n<p><code>sum Class2 = Class1.2</code></p>\n<p><code>sum Class3 = Class2.2</code></p>\n<p><code>sum Class4 = sum Class3</code><code></code></p>\n<p><code>sum Class5 = sum Class11 + Class4.2</code><code></code></p>\n<p><code>sum Class6 = 1.0</code><code></code></p>\n<p><code>sum Class7 = Class1.1</code></p>\n<p><code>sum Class8 = Class6.1</code><code></code></p>\n<p><code>sum Class9 = Class2.1</code><code></code></p>\n<p><code>sum Class10 = Class4.1</code><code></code></p>\n<p><code>sum Class11 = sum Class10</code></p>\n<p>where sum Class1 = Class1.1+Class1.2+Class1.3, sum Class2=Class2.1+Class2.2 and so on. I started placing what I expected to be a&nbsp;generous limit on the equalities above, like epsilon=1e-4, but surprisingly, out of 70948 items, there are a lot of violations, mostly for the normalization of Class8 and Class9 nodes.</p>\n<p>Here is the output of the attached python script:</p>\n<p><code>violations of Class2 constraint 64 times out of 70948</code><code></code></p>\n<p><code>violations of Class11 constraint 31 times out of 70948</code></p>\n<p><code>violations of Class8 constraint 981 times out of 70948</code></p>\n<p><code>violations of Class9 constraint 1693 times out of 70948</code></p>\n<p><code>violations of Class7 constraint 159 times out of 70948</code></p>\n<p><code>violations of Class4 constraint 11 times out of 70948</code></p>\n<p><code>violations of Class5 constraint 316 times out of 70948</code></p>\n<p><code>violations of Class10 constraint 296 times out of 70948</code></p>\n<p><code>violations of Class3 constraint 91 times out of 70948</code></p>\n<p>In some cases the violation error is as high as 1, and is generally of order 0.01.</p>\n<p>I would like to ask if I got something wrong in the normalizations above, I am bit confused.</p>\n<p>Thank you</p>\n<p>&nbsp;</p>\n<p>&nbsp;</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "37032",
      "postDate": "01/06/2014 14:37:55",
      "content": "<p>I did the same thing:&nbsp;https://www.kaggle.com/c/galaxy-zoo-the-galaxy-challenge/forums/t/6706/is-question-6-also-answered-for-stars-artifacts-answer-1-3/36798#post36798</p>\n<p>The organisators are looking into it but we haven't heard back so far.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "37033",
      "postDate": "01/06/2014 14:42:21",
      "content": "<p>They are, and they apologize for the delay. Hope to have something today or tomorrow.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "37034",
      "postDate": "01/06/2014 15:01:10",
      "content": "<p>I think fixing this issue is quite important, since the errors are of the order of a few percent, and the top scorers of the leaderboard already got below 10 % RMSE.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "37041",
      "postDate": "01/06/2014 18:39:43",
      "content": "<p>Should we be expecting new data? IMO, unless there is a serious problem discovered early on (like leakage) then I don't think the data should be changed after the competition has been started (and I don't think these violations qualify as a serious problem)</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "37042",
      "postDate": "01/06/2014 18:44:52",
      "content": "<p>[quote=Ryan Kiros;37041]</p>\n<p>Should we be expecting new data? IMO, unless there is a serious problem discovered early on (like leakage) then I don't think the data should be changed after the competition has been started (and I don't think these violations qualify as a serious problem)</p>\n<p>[/quote]</p>\n<p>Well, I see this as a serious problem. Trying to actually model the way the data is generated and should be behaving might not lead to a winning solution here because of a big problem with the data. I guess that the objective of this competition is to get a good model for classifying galaxies (which will be better if the data is actually correct) and that the primary objective is not to give money to people with the deepest neural network ;).</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "37045",
      "postDate": "01/06/2014 19:11:55",
      "content": "<p>The reason I don't think this is a serious problem is as follows: even without trying to incorporate any constraints into my model, it still learns to make predictions that approximate them (enough so that I don't have to do any normalization at all). If these violations were inhibiting the ability to model the tree structure then I suspect this wouldn't be possible.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "37048",
      "postDate": "01/06/2014 19:38:07",
      "content": "<p>There's also the 'unknown unknowns' problem: unless and until the root cause of already identified inconsistencies is discovered - and remedied - we cannot know how else the posted data differs from what the volunteers ('zooites') actually did (i.e. what the true distribution of their, collective, classification clicks is). Accurately predicting flawed data isn't very helpful, is it?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "37049",
      "postDate": "01/06/2014 19:44:29",
      "content": "<p>imo, these violations should be eliminated from the data and new data should be given. It wont be helpful for the organization to get predictions and model on a flawed data.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "37051",
      "postDate": "01/06/2014 20:02:53",
      "content": "<p>&nbsp;</p>\n<p>I'm not doing the compeitition so I can't really speak to how serious these violations may or may not be, but I think one could argue that dealing with dirty data is all part of the job.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "37052",
      "postDate": "01/06/2014 20:05:22",
      "content": "<p>@david, have you read &quot;the galaxy zoo decision tree&quot; page?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "37053",
      "postDate": "01/06/2014 20:10:03",
      "content": "<p>There are 37 response values to forecast, and there are 11 constraints, which brings the number of independent variables down to 26. We should expect that the reliability of the generalization will be affected by this difference.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "37054",
      "postDate": "01/06/2014 20:23:59",
      "content": "<p>What if the organizers just re-scored the leaderboard so that test points that have violations are not counted?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "37055",
      "postDate": "01/06/2014 20:27:47",
      "content": "<p>[quote=David McGarry;37051]</p>\n<p>I'm not doing the compeitition so I can't really speak to how serious these violations may or may not be, but I think one could argue that dealing with dirty data is all part of the job.[/quote]</p>\n<p>You could so argue, but in terms of the stated challenge/aim - <a href=\"http://www.kaggle.com/c/galaxy-zoo-the-galaxy-challenge\">Classify the morphologies of distant galaxies in our Universe</a> - I think astronomers would rather not have to deal with fixable flaws in data reduction/preparation.</p>\n<p>&nbsp;</p>",
      "rawMarkdown": "",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 37032,
      "author_name": "sedielem",
      "author_url": "",
      "post_date": "01/06/2014 14:37:55",
      "content": "<p>I did the same thing:&nbsp;https://www.kaggle.com/c/galaxy-zoo-the-galaxy-challenge/forums/t/6706/is-question-6-also-answered-for-stars-artifacts-answer-1-3/36798#post36798</p>\n<p>The organisators are looking into it but we haven't heard back so far.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 37033,
      "author_name": "willettk",
      "author_url": "",
      "post_date": "01/06/2014 14:42:21",
      "content": "<p>They are, and they apologize for the delay. Hope to have something today or tomorrow.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 37034,
      "author_name": "mdemenech",
      "author_url": "",
      "post_date": "01/06/2014 15:01:10",
      "content": "<p>I think fixing this issue is quite important, since the errors are of the order of a few percent, and the top scorers of the leaderboard already got below 10 % RMSE.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 37041,
      "author_name": "ryankiros",
      "author_url": "",
      "post_date": "01/06/2014 18:39:43",
      "content": "<p>Should we be expecting new data? IMO, unless there is a serious problem discovered early on (like leakage) then I don't think the data should be changed after the competition has been started (and I don't think these violations qualify as a serious problem)</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 37042,
      "author_name": "utdiscant",
      "author_url": "",
      "post_date": "01/06/2014 18:44:52",
      "content": "<p>[quote=Ryan Kiros;37041]</p>\n<p>Should we be expecting new data? IMO, unless there is a serious problem discovered early on (like leakage) then I don't think the data should be changed after the competition has been started (and I don't think these violations qualify as a serious problem)</p>\n<p>[/quote]</p>\n<p>Well, I see this as a serious problem. Trying to actually model the way the data is generated and should be behaving might not lead to a winning solution here because of a big problem with the data. I guess that the objective of this competition is to get a good model for classifying galaxies (which will be better if the data is actually correct) and that the primary objective is not to give money to people with the deepest neural network ;).</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 37045,
      "author_name": "ryankiros",
      "author_url": "",
      "post_date": "01/06/2014 19:11:55",
      "content": "<p>The reason I don't think this is a serious problem is as follows: even without trying to incorporate any constraints into my model, it still learns to make predictions that approximate them (enough so that I don't have to do any normalization at all). If these violations were inhibiting the ability to model the tree structure then I suspect this wouldn't be possible.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 37048,
      "author_name": "jeantate",
      "author_url": "",
      "post_date": "01/06/2014 19:38:07",
      "content": "<p>There's also the 'unknown unknowns' problem: unless and until the root cause of already identified inconsistencies is discovered - and remedied - we cannot know how else the posted data differs from what the volunteers ('zooites') actually did (i.e. what the true distribution of their, collective, classification clicks is). Accurately predicting flawed data isn't very helpful, is it?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 37049,
      "author_name": "abhishek",
      "author_url": "",
      "post_date": "01/06/2014 19:44:29",
      "content": "<p>imo, these violations should be eliminated from the data and new data should be given. It wont be helpful for the organization to get predictions and model on a flawed data.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 37051,
      "author_name": "dmcgarry",
      "author_url": "",
      "post_date": "01/06/2014 20:02:53",
      "content": "<p>&nbsp;</p>\n<p>I'm not doing the compeitition so I can't really speak to how serious these violations may or may not be, but I think one could argue that dealing with dirty data is all part of the job.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 37052,
      "author_name": "abhishek",
      "author_url": "",
      "post_date": "01/06/2014 20:05:22",
      "content": "<p>@david, have you read &quot;the galaxy zoo decision tree&quot; page?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 37053,
      "author_name": "mdemenech",
      "author_url": "",
      "post_date": "01/06/2014 20:10:03",
      "content": "<p>There are 37 response values to forecast, and there are 11 constraints, which brings the number of independent variables down to 26. We should expect that the reliability of the generalization will be affected by this difference.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 37054,
      "author_name": "ryankiros",
      "author_url": "",
      "post_date": "01/06/2014 20:23:59",
      "content": "<p>What if the organizers just re-scored the leaderboard so that test points that have violations are not counted?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 37055,
      "author_name": "jeantate",
      "author_url": "",
      "post_date": "01/06/2014 20:27:47",
      "content": "<p>[quote=David McGarry;37051]</p>\n<p>I'm not doing the compeitition so I can't really speak to how serious these violations may or may not be, but I think one could argue that dealing with dirty data is all part of the job.[/quote]</p>\n<p>You could so argue, but in terms of the stated challenge/aim - <a href=\"http://www.kaggle.com/c/galaxy-zoo-the-galaxy-challenge\">Classify the morphologies of distant galaxies in our Universe</a> - I think astronomers would rather not have to deal with fixable flaws in data reduction/preparation.</p>\n<p>&nbsp;</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "37031": "",
    "37032": "",
    "37033": "",
    "37034": "",
    "37041": "",
    "37042": "",
    "37045": "",
    "37048": "",
    "37049": "",
    "37051": "",
    "37052": "",
    "37053": "",
    "37054": "",
    "37055": ""
  },
  "source": "meta"
}