{
  "id": 720,
  "title": "Var2 and Var4",
  "url": "/competitions/ClaimPredictionChallenge/discussion/720",
  "author_name": "",
  "post_date": "2011-07-19T19:51:53.450Z",
  "votes": null,
  "comment_count": 4,
  "views": 4901,
  "content": "<p>I see values for Var2 and Var4 that are above 1.0. I thought that the quantitative variables were normalized to have 0 mean and stdev 1. Am I missing something?</p>",
  "messages": [
    {
      "id": "4656",
      "postDate": "07/19/2011 19:51:53",
      "content": "<p>I see values for Var2 and Var4 that are above 1.0. I thought that the quantitative variables were normalized to have 0 mean and stdev 1. Am I missing something?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "4657",
      "postDate": "07/19/2011 20:06:50",
      "content": "<p>A variable with mean 0 and stdev 1 can definitely take on values above 1.</p>\r\n<p>There's no reason the max should be 1.</p>\r\n<p>Thanks for participating, and good luck! :)</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "4661",
      "postDate": "07/20/2011 13:50:14",
      "content": "<p>I've computed the sample mean and the sample standard deviation for Var2 and get mean=-0.29 and stdev=2.79E-4</p>\r\n<p>That's not what it's supposed to be it seems to me. </p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "4667",
      "postDate": "07/20/2011 19:04:10",
      "content": "<p>In most cases they look pretty close to me to what they are claiming - I computed it for all variables they are claiming are normalized that way as follows:</p>\r\n<p>&gt; apply(cars[,c(22:29,31:34)],2,sd)<br>\r\n&nbsp;&nbsp;&nbsp;&nbsp; Var1&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Var2&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Var3&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Var4&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Var5&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Var6&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Var7&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Var8&nbsp;&nbsp;&nbsp; NVVar1&nbsp;&nbsp;&nbsp; NVVar2&nbsp;&nbsp;&nbsp; NVVar3&nbsp;&nbsp;&nbsp; NVVar4\r\n<br>\r\n0.9800609 0.9684165 1.0189020 0.9680170 0.9910490 0.9792078 1.0064329 1.0039540 1.0310404 1.0382125 1.0277485 1.0342738\r\n<br>\r\n&gt; apply(cars.test[,c(22:29,31:34)],2,sd)<br>\r\n&nbsp;&nbsp;&nbsp;&nbsp; Var1&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Var2&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Var3&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Var4&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Var5&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Var6&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Var7&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Var8&nbsp;&nbsp;&nbsp; NVVar1&nbsp;&nbsp;&nbsp; NVVar2&nbsp;&nbsp;&nbsp; NVVar3&nbsp;&nbsp;&nbsp; NVVar4\r\n<br>\r\n0.4480300 0.8989782 0.6923027 0.8814873 0.6157877 0.5500501 0.5029692 1.0958680 0.9301153 0.9855027 0.8958106 1.0426176</p>\r\n<p>and:</p>\r\n<p>&gt; apply(cars[,c(22:29,31:34)],2,mean)<br>\r\n&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Var1&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Var2&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Var3&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Var4&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Var5&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Var6&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Var7&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Var8&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; NVVar1&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; NVVar2&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; NVVar3&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; NVVar4\r\n<br>\r\n-0.010119254 -0.065087023 -0.025433912 -0.054567923&nbsp; 0.003838594 -0.040122715 -0.024212876 -0.058560590&nbsp; 0.014684099&nbsp; 0.017511687&nbsp; 0.013542262&nbsp; 0.018513759\r\n<br>\r\n&gt; apply(cars.test[,c(22:29,31:34)],2,mean)<br>\r\n&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Var1&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Var2&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Var3&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Var4&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Var5&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Var6&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Var7&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Var8&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; NVVar1&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; NVVar2&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; NVVar3&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; NVVar4\r\n<br>\r\n-0.291216266 -0.052893232 -0.208794360 -0.088931682 -0.172702121 -0.371343400 -0.552815601&nbsp; 0.087707187 -0.027083245 -0.010025619 -0.045309592&nbsp; 0.009566612</p>\r\n<p>It appears that the normalization was done based on the training set (just a guess based off the more stable values there).&nbsp; There seems to be a big difference on a few of these - which would make sense if they could be related to date type info.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "4679",
      "postDate": "07/21/2011 18:18:42",
      "content": "<p>Thanks, my mistake was to look at the test set.</p>",
      "rawMarkdown": "",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 4657,
      "author_name": "allstateinsurancecompany",
      "author_url": "",
      "post_date": "07/19/2011 20:06:50",
      "content": "<p>A variable with mean 0 and stdev 1 can definitely take on values above 1.</p>\r\n<p>There's no reason the max should be 1.</p>\r\n<p>Thanks for participating, and good luck! :)</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 4661,
      "author_name": "meliponemoody",
      "author_url": "",
      "post_date": "07/20/2011 13:50:14",
      "content": "<p>I've computed the sample mean and the sample standard deviation for Var2 and get mean=-0.29 and stdev=2.79E-4</p>\r\n<p>That's not what it's supposed to be it seems to me. </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 4667,
      "author_name": "chrisraimondi",
      "author_url": "",
      "post_date": "07/20/2011 19:04:10",
      "content": "<p>In most cases they look pretty close to me to what they are claiming - I computed it for all variables they are claiming are normalized that way as follows:</p>\r\n<p>&gt; apply(cars[,c(22:29,31:34)],2,sd)<br>\r\n&nbsp;&nbsp;&nbsp;&nbsp; Var1&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Var2&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Var3&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Var4&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Var5&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Var6&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Var7&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Var8&nbsp;&nbsp;&nbsp; NVVar1&nbsp;&nbsp;&nbsp; NVVar2&nbsp;&nbsp;&nbsp; NVVar3&nbsp;&nbsp;&nbsp; NVVar4\r\n<br>\r\n0.9800609 0.9684165 1.0189020 0.9680170 0.9910490 0.9792078 1.0064329 1.0039540 1.0310404 1.0382125 1.0277485 1.0342738\r\n<br>\r\n&gt; apply(cars.test[,c(22:29,31:34)],2,sd)<br>\r\n&nbsp;&nbsp;&nbsp;&nbsp; Var1&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Var2&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Var3&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Var4&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Var5&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Var6&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Var7&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Var8&nbsp;&nbsp;&nbsp; NVVar1&nbsp;&nbsp;&nbsp; NVVar2&nbsp;&nbsp;&nbsp; NVVar3&nbsp;&nbsp;&nbsp; NVVar4\r\n<br>\r\n0.4480300 0.8989782 0.6923027 0.8814873 0.6157877 0.5500501 0.5029692 1.0958680 0.9301153 0.9855027 0.8958106 1.0426176</p>\r\n<p>and:</p>\r\n<p>&gt; apply(cars[,c(22:29,31:34)],2,mean)<br>\r\n&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Var1&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Var2&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Var3&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Var4&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Var5&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Var6&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Var7&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Var8&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; NVVar1&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; NVVar2&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; NVVar3&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; NVVar4\r\n<br>\r\n-0.010119254 -0.065087023 -0.025433912 -0.054567923&nbsp; 0.003838594 -0.040122715 -0.024212876 -0.058560590&nbsp; 0.014684099&nbsp; 0.017511687&nbsp; 0.013542262&nbsp; 0.018513759\r\n<br>\r\n&gt; apply(cars.test[,c(22:29,31:34)],2,mean)<br>\r\n&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Var1&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Var2&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Var3&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Var4&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Var5&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Var6&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Var7&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Var8&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; NVVar1&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; NVVar2&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; NVVar3&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; NVVar4\r\n<br>\r\n-0.291216266 -0.052893232 -0.208794360 -0.088931682 -0.172702121 -0.371343400 -0.552815601&nbsp; 0.087707187 -0.027083245 -0.010025619 -0.045309592&nbsp; 0.009566612</p>\r\n<p>It appears that the normalization was done based on the training set (just a guess based off the more stable values there).&nbsp; There seems to be a big difference on a few of these - which would make sense if they could be related to date type info.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 4679,
      "author_name": "meliponemoody",
      "author_url": "",
      "post_date": "07/21/2011 18:18:42",
      "content": "<p>Thanks, my mistake was to look at the test set.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "4656": "",
    "4657": "",
    "4661": "",
    "4667": "",
    "4679": ""
  },
  "source": "meta"
}