{
  "id": 23645,
  "title": "Numeric features normalized?",
  "url": "/competitions/bosch-production-line-performance/discussion/23645",
  "author_name": "",
  "post_date": "2016-09-14T13:44:32.317Z",
  "votes": 2,
  "comment_count": 4,
  "views": 696,
  "content": "<p>All the numeric features have values between -1 and +1, and each of them have a range &lt;=1 (most of them close to 1), but they use very different parts of the -1 - +1 range.</p>\n\n<p>They seem to be normalized on a feature level, but then again modified so we don't have normalization on production level (to disable comparison between features?).</p>\n\n<p>Can an admin give us some information about the process?</p>\n\n<p>Thank you!</p>",
  "messages": [
    {
      "id": "135486",
      "postDate": "09/14/2016 13:44:32",
      "content": "<p>All the numeric features have values between -1 and +1, and each of them have a range &lt;=1 (most of them close to 1), but they use very different parts of the -1 - +1 range.</p>\n\n<p>They seem to be normalized on a feature level, but then again modified so we don't have normalization on production level (to disable comparison between features?).</p>\n\n<p>Can an admin give us some information about the process?</p>\n\n<p>Thank you!</p>",
      "rawMarkdown": "All the numeric features have values between -1 and +1, and each of them have a range <=1 (most of them close to 1), but they use very different parts of the -1 - +1 range.\r\n\r\nThey seem to be normalized on a feature level, but then again modified so we don't have normalization on production level (to disable comparison between features?).\r\n\r\nCan an admin give us some information about the process?\r\n\r\nThank you!",
      "votes": null
    },
    {
      "id": "135545",
      "postDate": "09/14/2016 22:09:15",
      "content": "<p>[quote=Tamas Barasz;135486] \nCan an admin give us some information about the process? <br>\n[/quote] <br>\nUsually in Kaggle competitions, the information that we have is the information that we get. It isn't really normal for admins to hand out extra information in the forums. If they did, people who followed the forums closely would have extra, highly material, information relative to people just worked on their models.</p>",
      "rawMarkdown": "[quote=Tamas Barasz;135486] \r\nCan an admin give us some information about the process?    \r\n[/quote]    \r\nUsually in Kaggle competitions, the information that we have is the information that we get. It isn't really normal for admins to hand out extra information in the forums. If they did, people who followed the forums closely would have extra, highly material, information relative to people just worked on their models.",
      "votes": null
    },
    {
      "id": "135548",
      "postDate": "09/14/2016 22:18:07",
      "content": "<p>[quote=Tamas Barasz;135486]\nAll the numeric features have values between -1 and +1, and each of them have a range &lt;=1 (most of them close to 1), but they use very different parts of the -1 - +1 range.</p>\n\n<p>They seem to be normalized on a feature level, but then again modified so we don't have normalization on production level (to disable comparison between features?).\n[/quote]\nI have part of train_numeric.csv.zip in a pandas data frame. I get the following:    </p>\n\n<pre><code>&gt;&gt;&gt;data.min().min()\n-1.0\n&gt;&gt;&gt;data.max().max()\n1.0\n&gt;&gt;&gt;data.mean().mean()\n7.276766738814088e-05\n</code></pre>\n\n<p>Based on that, I would say that the data was mean-centered and then min-max scaled (that is, each feature was divided by max(max(feature), -min(feature)) for non-missing values of the feature).    </p>",
      "rawMarkdown": "[quote=Tamas Barasz;135486]\r\nAll the numeric features have values between -1 and +1, and each of them have a range <=1 (most of them close to 1), but they use very different parts of the -1 - +1 range.\r\n\r\nThey seem to be normalized on a feature level, but then again modified so we don't have normalization on production level (to disable comparison between features?).\r\n[/quote]\r\nI have part of train_numeric.csv.zip in a pandas data frame. I get the following:    \r\n\r\n    >>>data.min().min()\r\n    -1.0\r\n    >>>data.max().max()\r\n    1.0\r\n    >>>data.mean().mean()\r\n    7.276766738814088e-05\r\n\r\nBased on that, I would say that the data was mean-centered and then min-max scaled (that is, each feature was divided by max(max(feature), -min(feature)) for non-missing values of the feature).",
      "votes": null
    },
    {
      "id": "135602",
      "postDate": "09/15/2016 07:42:57",
      "content": "<p>[quote=David Thaler;135548]</p>\n\n<p>Based on that, I would say that the data was mean-centered and then min-max scaled (that is, each feature was divided by max(max(feature), -min(feature)) for non-missing values of the feature).    </p>\n\n<p>[/quote]</p>\n\n<p>Thank you David,\nyes that sounds reasonable, but in that case each feature separately would have a range of 1 (max(feature) -min(feature) = 1) (or 0 in some cases). But this isn't the case, check out the csv attached to my original post with min max and range for every feature. More than half the features have a range &lt; 1, down even to 0.001</p>",
      "rawMarkdown": "[quote=David Thaler;135548]\r\n\r\nBased on that, I would say that the data was mean-centered and then min-max scaled (that is, each feature was divided by max(max(feature), -min(feature)) for non-missing values of the feature).    \r\n\r\n[/quote]\r\n\r\nThank you David,\r\nyes that sounds reasonable, but in that case each feature separately would have a range of 1 (max(feature) -min(feature) = 1) (or 0 in some cases). But this isn't the case, check out the csv attached to my original post with min max and range for every feature. More than half the features have a range < 1, down even to 0.001",
      "votes": null
    },
    {
      "id": "135651",
      "postDate": "09/15/2016 16:27:18",
      "content": "<p>If you keep looking, you'll note that each individual feature has a maximum range = max - min &lt;= 1. Why these ranges aren't all aligned to [0, 1] or [-1, 1] is a mystery. Potentially some features are comparable relative to one another. (e.g., some of these could be temperature readings, overall normalized to [-1, 1] but individually occupying a smaller range). FWIW, the test ranges are also slightly different from the train ranges.</p>",
      "rawMarkdown": "If you keep looking, you'll note that each individual feature has a maximum range = max - min <= 1. Why these ranges aren't all aligned to [0, 1] or [-1, 1] is a mystery. Potentially some features are comparable relative to one another. (e.g., some of these could be temperature readings, overall normalized to [-1, 1] but individually occupying a smaller range). FWIW, the test ranges are also slightly different from the train ranges.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 135545,
      "author_name": "davidthaler",
      "author_url": "",
      "post_date": "09/14/2016 22:09:15",
      "content": "<p>[quote=Tamas Barasz;135486] \nCan an admin give us some information about the process? <br>\n[/quote] <br>\nUsually in Kaggle competitions, the information that we have is the information that we get. It isn't really normal for admins to hand out extra information in the forums. If they did, people who followed the forums closely would have extra, highly material, information relative to people just worked on their models.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 135548,
      "author_name": "davidthaler",
      "author_url": "",
      "post_date": "09/14/2016 22:18:07",
      "content": "<p>[quote=Tamas Barasz;135486]\nAll the numeric features have values between -1 and +1, and each of them have a range &lt;=1 (most of them close to 1), but they use very different parts of the -1 - +1 range.</p>\n\n<p>They seem to be normalized on a feature level, but then again modified so we don't have normalization on production level (to disable comparison between features?).\n[/quote]\nI have part of train_numeric.csv.zip in a pandas data frame. I get the following:    </p>\n\n<pre><code>&gt;&gt;&gt;data.min().min()\n-1.0\n&gt;&gt;&gt;data.max().max()\n1.0\n&gt;&gt;&gt;data.mean().mean()\n7.276766738814088e-05\n</code></pre>\n\n<p>Based on that, I would say that the data was mean-centered and then min-max scaled (that is, each feature was divided by max(max(feature), -min(feature)) for non-missing values of the feature).    </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 135602,
      "author_name": "barasztamas",
      "author_url": "",
      "post_date": "09/15/2016 07:42:57",
      "content": "<p>[quote=David Thaler;135548]</p>\n\n<p>Based on that, I would say that the data was mean-centered and then min-max scaled (that is, each feature was divided by max(max(feature), -min(feature)) for non-missing values of the feature).    </p>\n\n<p>[/quote]</p>\n\n<p>Thank you David,\nyes that sounds reasonable, but in that case each feature separately would have a range of 1 (max(feature) -min(feature) = 1) (or 0 in some cases). But this isn't the case, check out the csv attached to my original post with min max and range for every feature. More than half the features have a range &lt; 1, down even to 0.001</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 135651,
      "author_name": "brundage",
      "author_url": "",
      "post_date": "09/15/2016 16:27:18",
      "content": "<p>If you keep looking, you'll note that each individual feature has a maximum range = max - min &lt;= 1. Why these ranges aren't all aligned to [0, 1] or [-1, 1] is a mystery. Potentially some features are comparable relative to one another. (e.g., some of these could be temperature readings, overall normalized to [-1, 1] but individually occupying a smaller range). FWIW, the test ranges are also slightly different from the train ranges.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "135486": "All the numeric features have values between -1 and +1, and each of them have a range <=1 (most of them close to 1), but they use very different parts of the -1 - +1 range.\r\n\r\nThey seem to be normalized on a feature level, but then again modified so we don't have normalization on production level (to disable comparison between features?).\r\n\r\nCan an admin give us some information about the process?\r\n\r\nThank you!",
    "135545": "[quote=Tamas Barasz;135486] \r\nCan an admin give us some information about the process?    \r\n[/quote]    \r\nUsually in Kaggle competitions, the information that we have is the information that we get. It isn't really normal for admins to hand out extra information in the forums. If they did, people who followed the forums closely would have extra, highly material, information relative to people just worked on their models.",
    "135548": "[quote=Tamas Barasz;135486]\r\nAll the numeric features have values between -1 and +1, and each of them have a range <=1 (most of them close to 1), but they use very different parts of the -1 - +1 range.\r\n\r\nThey seem to be normalized on a feature level, but then again modified so we don't have normalization on production level (to disable comparison between features?).\r\n[/quote]\r\nI have part of train_numeric.csv.zip in a pandas data frame. I get the following:    \r\n\r\n    >>>data.min().min()\r\n    -1.0\r\n    >>>data.max().max()\r\n    1.0\r\n    >>>data.mean().mean()\r\n    7.276766738814088e-05\r\n\r\nBased on that, I would say that the data was mean-centered and then min-max scaled (that is, each feature was divided by max(max(feature), -min(feature)) for non-missing values of the feature).",
    "135602": "[quote=David Thaler;135548]\r\n\r\nBased on that, I would say that the data was mean-centered and then min-max scaled (that is, each feature was divided by max(max(feature), -min(feature)) for non-missing values of the feature).    \r\n\r\n[/quote]\r\n\r\nThank you David,\r\nyes that sounds reasonable, but in that case each feature separately would have a range of 1 (max(feature) -min(feature) = 1) (or 0 in some cases). But this isn't the case, check out the csv attached to my original post with min max and range for every feature. More than half the features have a range < 1, down even to 0.001",
    "135651": "If you keep looking, you'll note that each individual feature has a maximum range = max - min <= 1. Why these ranges aren't all aligned to [0, 1] or [-1, 1] is a mystery. Potentially some features are comparable relative to one another. (e.g., some of these could be temperature readings, overall normalized to [-1, 1] but individually occupying a smaller range). FWIW, the test ranges are also slightly different from the train ranges."
  },
  "source": "meta"
}