Even though all the images in the MNIST dataset are centered, with a similar scale, and face up with no rotations, they have a significant handwriting variation that puzzles me how a linear model achieves such a high classification accuracy.
\n\nAs far as I am able to visualize, given the significant handwriting variation, the digits should be linearly inseparable in a 784 dimensional space, i.e., there should be a little complex (though not very complex) non-linear boundary that separates the different digits. Similar to the well-cited 'XOR' example where positive and negative classes can not be separated by any linear classifier, it seems baffling to me how logistic regression produces such a high accuracy with entirely linear features (no polynomial features).
\n\nAs an example, given any pixel in the image, different handwritten variations of the digits '3' and '5' can make that pixel illuminated or not. Therefore, with a set of learned weights, each pixel can make a digit look as a '3' as well as a '5'. Only with a combination of pixel values should it be possible to say whether a digit is a '3' or a '5'. This is true for most of the digit pairs. So, how is logistic regression, which blindly weighs its decision based on all pixel values independently (without any inter-pixel dependencies), able to achieve such high accuracies.
\n\nI know that I am wrong somewhere or am just over-estimating the variation in the images. However, it would be great if someone could help me with an intuition on how the digits are 'almost' linearly separable.
", "messages": [ { "id": 1222990, "postDate": "2021-03-02T11:03:52.063Z", "content": "By a simple linear regression, I end up with 25% accuracy.
\nBy a second degree polynominal regression, I project maybe 80-85%, but it is moot due to >240 000 dof.
\nA third degree ought to solve it with 400-500 pixels (also nonsense due to >15 000 000 dof).
", "rawMarkdown": "By a simple linear regression, I end up with 25% accuracy.\n\nBy a second degree polynominal regression, I project maybe 80-85%, but it is moot due to >240 000 dof.\n\nA third degree ought to solve it with 400-500 pixels (also nonsense due to >15 000 000 dof)." }, { "id": 652521, "postDate": "2019-10-19T01:35:49.787Z", "content": "That's an interesting question. I started in deep learning just 1 year ago but I realized that any algorithm can give you \"good\" results if you change your data a little bit. For instance, if you want to use images as inputs for a linear model or logistic regression you probably going to flatten them to vectors. So, you give away all the structure on the digits 28x28 file to feed your algorithm a simple vector of 784 bits. You lose that information in exchange for some degree of accuracy. But in the big picture that is quite limiting: is that really useful in real problems? I'd say no.\nIf try simpler algorithms for complex tasks (e.g. image recognition, natural language processing etc) you might get accuracies in the 90% range; but you probably won't get past that. For instance, I work with drug discovery and it's common to use linear models to predict the interaction of a drug with a protein targets. When you read more recent papers the results show that this kind of models aren't getting any better.
", "rawMarkdown": "That's an interesting question. I started in deep learning just 1 year ago but I realized that any algorithm can give you \"good\" results if you change your data a little bit. For instance, if you want to use images as inputs for a linear model or logistic regression you probably going to flatten them to vectors. So, you give away all the structure on the digits 28x28 file to feed your algorithm a simple vector of 784 bits. You lose that information in exchange for some degree of accuracy. But in the big picture that is quite limiting: is that really useful in real problems? I'd say no.\nIf try simpler algorithms for complex tasks (e.g. image recognition, natural language processing etc) you might get accuracies in the 90% range; but you probably won't get past that. For instance, I work with drug discovery and it's common to use linear models to predict the interaction of a drug with a protein targets. When you read more recent papers the results show that this kind of models aren't getting any better. " }, { "id": 624259, "postDate": "2019-09-11T21:43:58.313Z", "content": "Even though all the images in the MNIST dataset are centered, with a similar scale, and face up with no rotations, they have a significant handwriting variation that puzzles me how a linear model achieves such a high classification accuracy.
\n\nAs far as I am able to visualize, given the significant handwriting variation, the digits should be linearly inseparable in a 784 dimensional space, i.e., there should be a little complex (though not very complex) non-linear boundary that separates the different digits. Similar to the well-cited 'XOR' example where positive and negative classes can not be separated by any linear classifier, it seems baffling to me how logistic regression produces such a high accuracy with entirely linear features (no polynomial features).
\n\nAs an example, given any pixel in the image, different handwritten variations of the digits '3' and '5' can make that pixel illuminated or not. Therefore, with a set of learned weights, each pixel can make a digit look as a '3' as well as a '5'. Only with a combination of pixel values should it be possible to say whether a digit is a '3' or a '5'. This is true for most of the digit pairs. So, how is logistic regression, which blindly weighs its decision based on all pixel values independently (without any inter-pixel dependencies), able to achieve such high accuracies.
\n\nI know that I am wrong somewhere or am just over-estimating the variation in the images. However, it would be great if someone could help me with an intuition on how the digits are 'almost' linearly separable.
", "rawMarkdown": "Even though all the images in the MNIST dataset are centered, with a similar scale, and face up with no rotations, they have a significant handwriting variation that puzzles me how a linear model achieves such a high classification accuracy.\n\nAs far as I am able to visualize, given the significant handwriting variation, the digits should be linearly inseparable in a 784 dimensional space, i.e., there should be a little complex (though not very complex) non-linear boundary that separates the different digits. Similar to the well-cited 'XOR' example where positive and negative classes can not be separated by any linear classifier, it seems baffling to me how logistic regression produces such a high accuracy with entirely linear features (no polynomial features).\n\nAs an example, given any pixel in the image, different handwritten variations of the digits '3' and '5' can make that pixel illuminated or not. Therefore, with a set of learned weights, each pixel can make a digit look as a '3' as well as a '5'. Only with a combination of pixel values should it be possible to say whether a digit is a '3' or a '5'. This is true for most of the digit pairs. So, how is logistic regression, which blindly weighs its decision based on all pixel values independently (without any inter-pixel dependencies), able to achieve such high accuracies.\n\nI know that I am wrong somewhere or am just over-estimating the variation in the images. However, it would be great if someone could help me with an intuition on how the digits are 'almost' linearly separable.\n\n" } ], "comments": [ { "id": 1222990, "author_name": "Tord Malmgren", "author_url": "", "post_date": "2021-03-02T11:03:52.063000", "content": "By a simple linear regression, I end up with 25% accuracy.
\nBy a second degree polynominal regression, I project maybe 80-85%, but it is moot due to >240 000 dof.
\nA third degree ought to solve it with 400-500 pixels (also nonsense due to >15 000 000 dof).
", "votes": 0, "replies": [] }, { "id": 652521, "author_name": "Marcos Santana", "author_url": "", "post_date": "2019-10-19T01:35:49.787000", "content": "That's an interesting question. I started in deep learning just 1 year ago but I realized that any algorithm can give you \"good\" results if you change your data a little bit. For instance, if you want to use images as inputs for a linear model or logistic regression you probably going to flatten them to vectors. So, you give away all the structure on the digits 28x28 file to feed your algorithm a simple vector of 784 bits. You lose that information in exchange for some degree of accuracy. But in the big picture that is quite limiting: is that really useful in real problems? I'd say no.\nIf try simpler algorithms for complex tasks (e.g. image recognition, natural language processing etc) you might get accuracies in the 90% range; but you probably won't get past that. For instance, I work with drug discovery and it's common to use linear models to predict the interaction of a drug with a protein targets. When you read more recent papers the results show that this kind of models aren't getting any better.
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1222990": "By a simple linear regression, I end up with 25% accuracy.\n\nBy a second degree polynominal regression, I project maybe 80-85%, but it is moot due to >240 000 dof.\n\nA third degree ought to solve it with 400-500 pixels (also nonsense due to >15 000 000 dof).", "652521": "That's an interesting question. I started in deep learning just 1 year ago but I realized that any algorithm can give you \"good\" results if you change your data a little bit. For instance, if you want to use images as inputs for a linear model or logistic regression you probably going to flatten them to vectors. So, you give away all the structure on the digits 28x28 file to feed your algorithm a simple vector of 784 bits. You lose that information in exchange for some degree of accuracy. But in the big picture that is quite limiting: is that really useful in real problems? I'd say no.\nIf try simpler algorithms for complex tasks (e.g. image recognition, natural language processing etc) you might get accuracies in the 90% range; but you probably won't get past that. For instance, I work with drug discovery and it's common to use linear models to predict the interaction of a drug with a protein targets. When you read more recent papers the results show that this kind of models aren't getting any better. ", "624259": "Even though all the images in the MNIST dataset are centered, with a similar scale, and face up with no rotations, they have a significant handwriting variation that puzzles me how a linear model achieves such a high classification accuracy.\n\nAs far as I am able to visualize, given the significant handwriting variation, the digits should be linearly inseparable in a 784 dimensional space, i.e., there should be a little complex (though not very complex) non-linear boundary that separates the different digits. Similar to the well-cited 'XOR' example where positive and negative classes can not be separated by any linear classifier, it seems baffling to me how logistic regression produces such a high accuracy with entirely linear features (no polynomial features).\n\nAs an example, given any pixel in the image, different handwritten variations of the digits '3' and '5' can make that pixel illuminated or not. Therefore, with a set of learned weights, each pixel can make a digit look as a '3' as well as a '5'. Only with a combination of pixel values should it be possible to say whether a digit is a '3' or a '5'. This is true for most of the digit pairs. So, how is logistic regression, which blindly weighs its decision based on all pixel values independently (without any inter-pixel dependencies), able to achieve such high accuracies.\n\nI know that I am wrong somewhere or am just over-estimating the variation in the images. However, it would be great if someone could help me with an intuition on how the digits are 'almost' linearly separable.\n\n" } }