{ "id": 189450, "title": "Role of use_bias = False/True", "url": "/competitions/gan-getting-started/discussion/189450", "author_name": "", "post_date": "2020-10-07T16:31:36.630171600Z", "votes": 2, "comment_count": 3, "views": 0, "content": "

Hello all,

\n

I looked at few kernels and the Tensorflow official guide and found myself wondering about the given statement :

\n

tf.keras.layers.Conv2D(filters, size, strides=2, padding='same',\n kernel_initializer=initializer, use_bias=False))

\n

It's pretty clear as it's a simple convolution operation.

\n

My question is what is the role of the use_bias parameter? How would things change if we set it to true in one case and false in another. No notebook I have seen till date mentions this.

\n

It would be a great help if someone comes to my rescue here!!

\n

Thanks and Regards

", "messages": [ { "id": "1041244", "postDate": "10/07/2020 16:31:36", "content": "

Hello all,

\n

I looked at few kernels and the Tensorflow official guide and found myself wondering about the given statement :

\n

tf.keras.layers.Conv2D(filters, size, strides=2, padding='same',\n kernel_initializer=initializer, use_bias=False))

\n

It's pretty clear as it's a simple convolution operation.

\n

My question is what is the role of the use_bias parameter? How would things change if we set it to true in one case and false in another. No notebook I have seen till date mentions this.

\n

It would be a great help if someone comes to my rescue here!!

\n

Thanks and Regards

", "rawMarkdown": "Hello all,\n\nI looked at few kernels and the Tensorflow official guide and found myself wondering about the given statement : \n\n**tf.keras.layers.Conv2D(filters, size, strides=2, padding='same',\n kernel_initializer=initializer, use_bias=False))**\n\nIt's pretty clear as it's a simple convolution operation.\n\nMy question is what is the role of the **use_bias** parameter? How would things change if we set it to true in one case and false in another. No notebook I have seen till date mentions this.\n\nIt would be a great help if someone comes to my rescue here!!\n\nThanks and Regards", "votes": null }, { "id": "1046835", "postDate": "10/12/2020 03:20:18", "content": "

@fireheart7 Its a great question. By convention Convolution layers have bias terms with them. Like Dense (Typical neural network layer; where the op is W*x + b where W is weights and b is bias), the Convolution layers also have bias with him in their op Conv(w, x) + b where b is bias and b is of n_filter sized 1D vector.

\n

And the use_bias arg in the Conv2D call is to enable/disable this bias term. If you disable it, then the op will be just Conv(w, x) and if you enable it, then the op will be Conv(w,x) + b.

\n

Typically, Its better to have Conv layers with bias in them (that's why by default, tf.keras.layers.Conv*D calls have use_bias set to true). But if there is a batch/instance norm after the Conv (or Dense) layer, the bias term becomes redundant as batch norm operation norms the conv's output and add its own bias. To know why that is the case, checkout this video from Andrew Ng in Deep Learning Specialization course. So it's good idea to set the use_bias term to false when the Conv layer is followed by batch norm.

", "rawMarkdown": "fireheart7 Its a great question. By convention Convolution layers have bias terms with them. Like Dense (Typical neural network layer; where the op is W*x + b where W is weights and b is bias), the Convolution layers also have bias with him in their op Conv(w, x) + b where b is bias and b is of n_filter sized 1D vector.\n\nAnd the use_bias arg in the Conv2D call is to enable/disable this bias term. If you disable it, then the op will be just Conv(w, x) and if you enable it, then the op will be Conv(w,x) + b.\n\nTypically, Its better to have Conv layers with bias in them (that's why by default, tf.keras.layers.Conv*D calls have use_bias set to true). But if there is a batch/instance norm after the Conv (or Dense) layer, the bias term becomes redundant as batch norm operation norms the conv's output and add its own bias. To know why that is the case, checkout [this video from Andrew Ng in Deep Learning Specialization course](https://www.youtube.com/watch?v=em6dfRxYkYU). So it's good idea to set the use_bias term to false when the Conv layer is followed by batch norm.", "votes": null }, { "id": "1067251", "postDate": "11/02/2020 12:14:16", "content": "

This is extremely helpful. Thank you for this wonderful answer.

\n

Cheers!

", "rawMarkdown": "This is extremely helpful. Thank you for this wonderful answer.\n\nCheers!", "votes": null }, { "id": "1139845", "postDate": "01/05/2021 16:54:48", "content": "

For anyone who wants to watch the Andrew Ng video, skip to 7:29 for the explanation of why bias doesn't matter for BN. His explanation is (paraphrased): \"The bias term doesn't matter because the BN is going to normalize the input (Z) to a mean of 0 and a standard variance, so adding a constant to Z before inputting it into BN doesn't change anything.\"

", "rawMarkdown": "For anyone who wants to watch the Andrew Ng video, skip to 7:29 for the explanation of why bias doesn't matter for BN. His explanation is (paraphrased): \"The bias term doesn't matter because the BN is going to normalize the input (Z) to a mean of 0 and a standard variance, so adding a constant to Z before inputting it into BN doesn't change anything.\"", "votes": null } ], "comments": [ { "id": 1046835, "author_name": "mnjmmhn", "author_url": "", "post_date": "10/12/2020 03:20:18", "content": "

@fireheart7 Its a great question. By convention Convolution layers have bias terms with them. Like Dense (Typical neural network layer; where the op is W*x + b where W is weights and b is bias), the Convolution layers also have bias with him in their op Conv(w, x) + b where b is bias and b is of n_filter sized 1D vector.

\n

And the use_bias arg in the Conv2D call is to enable/disable this bias term. If you disable it, then the op will be just Conv(w, x) and if you enable it, then the op will be Conv(w,x) + b.

\n

Typically, Its better to have Conv layers with bias in them (that's why by default, tf.keras.layers.Conv*D calls have use_bias set to true). But if there is a batch/instance norm after the Conv (or Dense) layer, the bias term becomes redundant as batch norm operation norms the conv's output and add its own bias. To know why that is the case, checkout this video from Andrew Ng in Deep Learning Specialization course. So it's good idea to set the use_bias term to false when the Conv layer is followed by batch norm.

", "votes": null, "replies": [ { "id": 1067251, "author_name": "fireheart7", "author_url": "", "post_date": "11/02/2020 12:14:16", "content": "

This is extremely helpful. Thank you for this wonderful answer.

\n

Cheers!

", "votes": null, "replies": [] }, { "id": 1139845, "author_name": "urosogrizovic", "author_url": "", "post_date": "01/05/2021 16:54:48", "content": "

For anyone who wants to watch the Andrew Ng video, skip to 7:29 for the explanation of why bias doesn't matter for BN. His explanation is (paraphrased): \"The bias term doesn't matter because the BN is going to normalize the input (Z) to a mean of 0 and a standard variance, so adding a constant to Z before inputting it into BN doesn't change anything.\"

", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1041244": "Hello all,\n\nI looked at few kernels and the Tensorflow official guide and found myself wondering about the given statement : \n\n**tf.keras.layers.Conv2D(filters, size, strides=2, padding='same',\n kernel_initializer=initializer, use_bias=False))**\n\nIt's pretty clear as it's a simple convolution operation.\n\nMy question is what is the role of the **use_bias** parameter? How would things change if we set it to true in one case and false in another. No notebook I have seen till date mentions this.\n\nIt would be a great help if someone comes to my rescue here!!\n\nThanks and Regards", "1046835": "fireheart7 Its a great question. By convention Convolution layers have bias terms with them. Like Dense (Typical neural network layer; where the op is W*x + b where W is weights and b is bias), the Convolution layers also have bias with him in their op Conv(w, x) + b where b is bias and b is of n_filter sized 1D vector.\n\nAnd the use_bias arg in the Conv2D call is to enable/disable this bias term. If you disable it, then the op will be just Conv(w, x) and if you enable it, then the op will be Conv(w,x) + b.\n\nTypically, Its better to have Conv layers with bias in them (that's why by default, tf.keras.layers.Conv*D calls have use_bias set to true). But if there is a batch/instance norm after the Conv (or Dense) layer, the bias term becomes redundant as batch norm operation norms the conv's output and add its own bias. To know why that is the case, checkout [this video from Andrew Ng in Deep Learning Specialization course](https://www.youtube.com/watch?v=em6dfRxYkYU). So it's good idea to set the use_bias term to false when the Conv layer is followed by batch norm.", "1067251": "This is extremely helpful. Thank you for this wonderful answer.\n\nCheers!", "1139845": "For anyone who wants to watch the Andrew Ng video, skip to 7:29 for the explanation of why bias doesn't matter for BN. His explanation is (paraphrased): \"The bias term doesn't matter because the BN is going to normalize the input (Z) to a mean of 0 and a standard variance, so adding a constant to Z before inputting it into BN doesn't change anything.\"" }, "source": "meta" }