{
  "id": 4683,
  "title": "white noise in NN weights",
  "url": "/competitions/digit-recognizer/discussion/4683",
  "author_name": "KL Tah",
  "post_date": "2013-05-27T14:57:23.153000",
  "votes": 0,
  "comment_count": 1,
  "views": null,
  "content": "<p>I'm using neural networks for this problem. When I visualize my weights, I see some of them with just white noise and barely any sculpture are all (see image attached). Why is this and how can I correct it?</p>\r\n<p></p>",
  "messages": [
    {
      "id": 24879,
      "postDate": "2013-05-29T03:18:09.937Z",
      "content": "<p>Do you initialize your weights to random noise? It looks like some of your units are never getting used/updated by the network and if you initalize to random noise, your looking at units that are somehow never being used/updated.</p>\r\n<p>Are you normalizing input? Sometimes if your input is wildly out of range of what your activation function can handle a unit could get railed always off? What is your learning rate? If you have too large of a learning rate (or super small) you might get\r\n behavior like that. What kind of activation function are you using?</p>"
    },
    {
      "id": 24843,
      "postDate": "2013-05-27T14:57:23.153Z",
      "content": "<p>I'm using neural networks for this problem. When I visualize my weights, I see some of them with just white noise and barely any sculpture are all (see image attached). Why is this and how can I correct it?</p>\r\n<p></p>"
    }
  ],
  "comments": [
    {
      "id": 24879,
      "author_name": "Alec Radford",
      "author_url": "",
      "post_date": "2013-05-29T03:18:09.937000",
      "content": "<p>Do you initialize your weights to random noise? It looks like some of your units are never getting used/updated by the network and if you initalize to random noise, your looking at units that are somehow never being used/updated.</p>\r\n<p>Are you normalizing input? Sometimes if your input is wildly out of range of what your activation function can handle a unit could get railed always off? What is your learning rate? If you have too large of a learning rate (or super small) you might get\r\n behavior like that. What kind of activation function are you using?</p>",
      "votes": 0,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "24879": "",
    "24843": ""
  }
}