Hello everyone, this is my first project on neural networks and so every time I tweak the model a bit and try to understand how the accuracy get affected.
\nNow when I fitted a Network without Scaling the data( i.e without dividing by 255), I was getting Train_accuracy of 0.94 and Val_accuracy of 0.97, while when I fit the same network with scaling data, I was getting Train_accuracy of 0.979 and Val_accuracy of 0.970.
\nSo I wanna understand why scaling affects the model so much while every features are in same range (0-255).
", "messages": [ { "id": 1145469, "postDate": "2021-01-09T07:00:42.953Z", "content": "Hello everyone, this is my first project on neural networks and so every time I tweak the model a bit and try to understand how the accuracy get affected.
\nNow when I fitted a Network without Scaling the data( i.e without dividing by 255), I was getting Train_accuracy of 0.94 and Val_accuracy of 0.97, while when I fit the same network with scaling data, I was getting Train_accuracy of 0.979 and Val_accuracy of 0.970.
\nSo I wanna understand why scaling affects the model so much while every features are in same range (0-255).
", "rawMarkdown": "Hello everyone, this is my first project on neural networks and so every time I tweak the model a bit and try to understand how the accuracy get affected.\n\nNow when I fitted a Network without Scaling the data( i.e without dividing by 255), I was getting Train_accuracy of 0.94 and Val_accuracy of 0.97, while when I fit the same network with scaling data, I was getting Train_accuracy of 0.979 and Val_accuracy of 0.970.\n\nSo I wanna understand why scaling affects the model so much while every features are in same range (0-255).\n", "votes": 4 }, { "id": 1146953, "postDate": "2021-01-10T07:54:33.893Z", "content": "Neural network uses the weighted sum of inputs value at each node and the resultant value is passed on to an activation function. Here, the weighted sum of inputs can be larger if the pixel values (input values) are in the range 0 - 255, which can affect the model to get trained in an effective way.
\nTo elaborate this, during back propagation, the weights are updated by estimating the slope (gradient) of loss function, if value is 255, then slope of sigmoid activation function used is close to 0, hence the weights wont get updated. This makes the model training ineffective.
\nSo, using raw pixel values of images in the neural network can tend to distort the model reaching the optimal solution. This can be avoided by scaling the pixel values.
\nGenerally, this is done in 3 different ways:
\nBased on the accuracy, one can find choosing any one of the above methods.
\nHope this helps.
", "rawMarkdown": "Neural network uses the weighted sum of inputs value at each node and the resultant value is passed on to an activation function. Here, the weighted sum of inputs can be larger if the pixel values (input values) are in the range 0 - 255, which can affect the model to get trained in an effective way. \n\nTo elaborate this, during back propagation, the weights are updated by estimating the slope (gradient) of loss function, if value is 255, then slope of sigmoid activation function used is close to 0, hence the weights wont get updated. This makes the model training ineffective. \n\nSo, using raw pixel values of images in the neural network can tend to distort the model reaching the optimal solution. This can be avoided by scaling the pixel values.\n\nGenerally, this is done in 3 different ways:\n1. Standard scaling (Z score) - pixel values are scaled to mean 0 and standard deviation 1\n2. Normalisation (MinMax) - pixel values are scaled in range of 0 to 1\n3. Centering - calculate the mean pixel value and subtract each pixel value from mean\n\nBased on the accuracy, one can find choosing any one of the above methods.\n\nHope this helps.", "votes": 4, "replies": [ { "id": 1148475, "postDate": "2021-01-11T06:50:53.443Z", "content": "Thanks for replying. It is really helpful.
", "rawMarkdown": "Thanks for replying. It is really helpful." } ] }, { "id": 1145641, "postDate": "2021-01-09T09:19:46.733Z", "content": "Normalizing the data is one of the most important steps in almost every Machine Learning algorithm. Normalizing the data generally speeds up learning and leads to faster convergence. In addition, in some algorithms its mandatory to normalize the data in order not to be biased by other features. For example, in the case of KNN, if all the features doesn't have the same scale the algorithm will be biased with the feature with biggest values.
\nIn the case of neural networks, differences in the scales across input variables may increase the difficulty of the problem being modeled. An example of this is that large input values (e.g. a spread of hundreds or thousands of units) can result in a model that learns large weight values. A model with large weight values is often unstable, meaning that it may suffer from poor performance during learning and sensitivity to input values resulting in higher generalization error.
\nBecause neural networks are usually trained via backpropagation with the chain rule, you may suffer also the exploding gradient problem because the derivatives of the activation functions used can take on larger values.
\nHope it helps!
", "rawMarkdown": "Normalizing the data is one of the most important steps in almost every Machine Learning algorithm. Normalizing the data generally speeds up learning and leads to faster convergence. In addition, in some algorithms its mandatory to normalize the data in order not to be biased by other features. For example, in the case of KNN, if all the features doesn't have the same scale the algorithm will be biased with the feature with biggest values.\n\nIn the case of neural networks, differences in the scales across input variables may increase the difficulty of the problem being modeled. An example of this is that large input values (e.g. a spread of hundreds or thousands of units) can result in a model that learns large weight values. A model with large weight values is often unstable, meaning that it may suffer from poor performance during learning and sensitivity to input values resulting in higher generalization error.\n\nBecause neural networks are usually trained via backpropagation with the chain rule, you may suffer also the exploding gradient problem because the derivatives of the activation functions used can take on larger values.\n\nHope it helps!", "votes": 3, "replies": [ { "id": 1145849, "postDate": "2021-01-09T11:26:31.257Z", "content": "yeah got it. Thanks😀
", "rawMarkdown": "yeah got it. Thanks😀" }, { "id": 1146432, "postDate": "2021-01-09T19:13:42.027Z", "content": "Thanks 😊
", "rawMarkdown": "Thanks 😊 " }, { "id": 1148480, "postDate": "2021-01-11T07:00:52.913Z", "content": "You're welcome!!
", "rawMarkdown": "You're welcome!!" } ] }, { "id": 1146783, "postDate": "2021-01-10T03:41:59.053Z", "content": "@hpoddar2810 while @fedesoriano already said it, normalization stops larger weights from building up in the network layers. Larger weights are more sensitive to noise in the input data (i.e. errant pixels or noise on the image will be picked up more by the network when using input data that hasn't been normalized). In effect, normalizing between 0 and 1 has the effect of smoothing the network.
", "rawMarkdown": "@hpoddar2810 while @fedesoriano already said it, normalization stops larger weights from building up in the network layers. Larger weights are more sensitive to noise in the input data (i.e. errant pixels or noise on the image will be picked up more by the network when using input data that hasn't been normalized). In effect, normalizing between 0 and 1 has the effect of smoothing the network.", "votes": 1 }, { "id": 2518822, "postDate": "2023-11-09T15:08:10.330Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 1146953, "author_name": "Bala Baskar", "author_url": "", "post_date": "2021-01-10T07:54:33.893000", "content": "Neural network uses the weighted sum of inputs value at each node and the resultant value is passed on to an activation function. Here, the weighted sum of inputs can be larger if the pixel values (input values) are in the range 0 - 255, which can affect the model to get trained in an effective way.
\nTo elaborate this, during back propagation, the weights are updated by estimating the slope (gradient) of loss function, if value is 255, then slope of sigmoid activation function used is close to 0, hence the weights wont get updated. This makes the model training ineffective.
\nSo, using raw pixel values of images in the neural network can tend to distort the model reaching the optimal solution. This can be avoided by scaling the pixel values.
\nGenerally, this is done in 3 different ways:
\nBased on the accuracy, one can find choosing any one of the above methods.
\nHope this helps.
", "votes": 4, "replies": [ { "id": 1148475, "author_name": "Harsh Poddar", "author_url": "", "post_date": "2021-01-11T06:50:53.443000", "content": "Thanks for replying. It is really helpful.
", "votes": 0, "replies": [] } ] }, { "id": 1145641, "author_name": "fedesoriano", "author_url": "", "post_date": "2021-01-09T09:19:46.733000", "content": "Normalizing the data is one of the most important steps in almost every Machine Learning algorithm. Normalizing the data generally speeds up learning and leads to faster convergence. In addition, in some algorithms its mandatory to normalize the data in order not to be biased by other features. For example, in the case of KNN, if all the features doesn't have the same scale the algorithm will be biased with the feature with biggest values.
\nIn the case of neural networks, differences in the scales across input variables may increase the difficulty of the problem being modeled. An example of this is that large input values (e.g. a spread of hundreds or thousands of units) can result in a model that learns large weight values. A model with large weight values is often unstable, meaning that it may suffer from poor performance during learning and sensitivity to input values resulting in higher generalization error.
\nBecause neural networks are usually trained via backpropagation with the chain rule, you may suffer also the exploding gradient problem because the derivatives of the activation functions used can take on larger values.
\nHope it helps!
", "votes": 3, "replies": [ { "id": 1145849, "author_name": "Harsh Poddar", "author_url": "", "post_date": "2021-01-09T11:26:31.257000", "content": "yeah got it. Thanks😀
", "votes": 0, "replies": [] }, { "id": 1146432, "author_name": "Miloud Belarebia", "author_url": "", "post_date": "2021-01-09T19:13:42.027000", "content": "Thanks 😊
", "votes": 0, "replies": [] }, { "id": 1148480, "author_name": "fedesoriano", "author_url": "", "post_date": "2021-01-11T07:00:52.913000", "content": "You're welcome!!
", "votes": 0, "replies": [] } ] }, { "id": 1146783, "author_name": "Craig Thomas", "author_url": "", "post_date": "2021-01-10T03:41:59.053000", "content": "@hpoddar2810 while @fedesoriano already said it, normalization stops larger weights from building up in the network layers. Larger weights are more sensitive to noise in the input data (i.e. errant pixels or noise on the image will be picked up more by the network when using input data that hasn't been normalized). In effect, normalizing between 0 and 1 has the effect of smoothing the network.
", "votes": 1, "replies": [] }, { "id": 2518822, "author_name": "", "author_url": "", "post_date": "2023-11-09T15:08:10.330000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1145469": "Hello everyone, this is my first project on neural networks and so every time I tweak the model a bit and try to understand how the accuracy get affected.\n\nNow when I fitted a Network without Scaling the data( i.e without dividing by 255), I was getting Train_accuracy of 0.94 and Val_accuracy of 0.97, while when I fit the same network with scaling data, I was getting Train_accuracy of 0.979 and Val_accuracy of 0.970.\n\nSo I wanna understand why scaling affects the model so much while every features are in same range (0-255).\n", "1146953": "Neural network uses the weighted sum of inputs value at each node and the resultant value is passed on to an activation function. Here, the weighted sum of inputs can be larger if the pixel values (input values) are in the range 0 - 255, which can affect the model to get trained in an effective way. \n\nTo elaborate this, during back propagation, the weights are updated by estimating the slope (gradient) of loss function, if value is 255, then slope of sigmoid activation function used is close to 0, hence the weights wont get updated. This makes the model training ineffective. \n\nSo, using raw pixel values of images in the neural network can tend to distort the model reaching the optimal solution. This can be avoided by scaling the pixel values.\n\nGenerally, this is done in 3 different ways:\n1. Standard scaling (Z score) - pixel values are scaled to mean 0 and standard deviation 1\n2. Normalisation (MinMax) - pixel values are scaled in range of 0 to 1\n3. Centering - calculate the mean pixel value and subtract each pixel value from mean\n\nBased on the accuracy, one can find choosing any one of the above methods.\n\nHope this helps.", "1145641": "Normalizing the data is one of the most important steps in almost every Machine Learning algorithm. Normalizing the data generally speeds up learning and leads to faster convergence. In addition, in some algorithms its mandatory to normalize the data in order not to be biased by other features. For example, in the case of KNN, if all the features doesn't have the same scale the algorithm will be biased with the feature with biggest values.\n\nIn the case of neural networks, differences in the scales across input variables may increase the difficulty of the problem being modeled. An example of this is that large input values (e.g. a spread of hundreds or thousands of units) can result in a model that learns large weight values. A model with large weight values is often unstable, meaning that it may suffer from poor performance during learning and sensitivity to input values resulting in higher generalization error.\n\nBecause neural networks are usually trained via backpropagation with the chain rule, you may suffer also the exploding gradient problem because the derivatives of the activation functions used can take on larger values.\n\nHope it helps!", "1146783": "@hpoddar2810 while @fedesoriano already said it, normalization stops larger weights from building up in the network layers. Larger weights are more sensitive to noise in the input data (i.e. errant pixels or noise on the image will be picked up more by the network when using input data that hasn't been normalized). In effect, normalizing between 0 and 1 has the effect of smoothing the network.", "2518822": "" } }