In the MNIST competition, a decent CNN with around 15K params can be used to achieve an accuracy > 90%
\nCan this performance be achieved via a fully connected network, assuming data is not a limitation.
\nTentatively, What size (number of neurons * number of layers) of fully connected network would it require to achieve >90% accuracy (if at all it is fundamentally possible)
", "messages": [ { "id": 1637726, "postDate": "2022-01-04T06:35:26.643Z", "content": "In the MNIST competition, a decent CNN with around 15K params can be used to achieve an accuracy > 90%
\nCan this performance be achieved via a fully connected network, assuming data is not a limitation.
\nTentatively, What size (number of neurons * number of layers) of fully connected network would it require to achieve >90% accuracy (if at all it is fundamentally possible)
", "rawMarkdown": "In the MNIST competition, a decent CNN with around 15K params can be used to achieve an accuracy > 90%\n\nCan this performance be achieved via a fully connected network, assuming data is not a limitation.\n\nTentatively, What size (number of neurons * number of layers) of fully connected network would it require to achieve >90% accuracy (if at all it is fundamentally possible)", "votes": 2 }, { "id": 1640777, "postDate": "2022-01-06T19:56:54.433Z", "content": "I am unable to break past 97% training accuracy using the FCN below. (Although we can easily reach >99.3% using a CNN, but I want to test the performance of the best possible FCN for this problem)
\nn = 128\ninputs = tf.keras.layers.Input(shape=(784))\nbn_layer = tf.keras.layers.BatchNormalization()(inputs)\ndense_layer_new_2 = tf.keras.layers.Dense(n, activation='sigmoid')(bn_layer)\ndp1_layer = tf.keras.layers.Dropout(0.2)(dense_layer_new_2)\ndense_layer_new_3 = tf.keras.layers.Dense(n, activation='sigmoid')(dp1_layer)\nbn_layer2 = tf.keras.layers.BatchNormalization()(dense_layer_new_3)\ndense_layer_new_4 = tf.keras.layers.Dense(n, activation='sigmoid')(bn_layer2)\ndp3_layer = tf.keras.layers.Dropout(0.2)(dense_layer_new_4)\ndense_layer_new_5 = tf.keras.layers.Dense(n, activation='sigmoid')(dp3_layer)\nbn_layer3 = tf.keras.layers.BatchNormalization()(dense_layer_new_5)\ndense_layer_new_7 = tf.keras.layers.Dense(n, activation='sigmoid')(bn_layer3)\noutputs = tf.keras.layers.Dense(10, activation='softmax')(dense_layer_new_7)\n\nmodel_fcn = tf.keras.Model(inputs=inputs, outputs=outputs)\nmodel_fcn.compile(optimizer='adam', loss=loss, metrics=[\"accuracy\"])\nmodel_fcn.summary()\nCan someone guide me what should I try next to achieve >99% accuracy:
\nAny resource which can teach about these tricks.
", "rawMarkdown": "I am unable to break past 97% training accuracy using the FCN below. (Although we can easily reach >99.3% using a CNN, but I want to test the performance of the best possible FCN for this problem)\n\n```\nn = 128\ninputs = tf.keras.layers.Input(shape=(784))\nbn_layer = tf.keras.layers.BatchNormalization()(inputs)\ndense_layer_new_2 = tf.keras.layers.Dense(n, activation='sigmoid')(bn_layer)\ndp1_layer = tf.keras.layers.Dropout(0.2)(dense_layer_new_2)\ndense_layer_new_3 = tf.keras.layers.Dense(n, activation='sigmoid')(dp1_layer)\nbn_layer2 = tf.keras.layers.BatchNormalization()(dense_layer_new_3)\ndense_layer_new_4 = tf.keras.layers.Dense(n, activation='sigmoid')(bn_layer2)\ndp3_layer = tf.keras.layers.Dropout(0.2)(dense_layer_new_4)\ndense_layer_new_5 = tf.keras.layers.Dense(n, activation='sigmoid')(dp3_layer)\nbn_layer3 = tf.keras.layers.BatchNormalization()(dense_layer_new_5)\ndense_layer_new_7 = tf.keras.layers.Dense(n, activation='sigmoid')(bn_layer3)\noutputs = tf.keras.layers.Dense(10, activation='softmax')(dense_layer_new_7)\n\nmodel_fcn = tf.keras.Model(inputs=inputs, outputs=outputs)\nmodel_fcn.compile(optimizer='adam', loss=loss, metrics=[\"accuracy\"])\nmodel_fcn.summary()\n```\n\n Can someone guide me what should I try next to achieve >99% accuracy:\n1. Change network architecture...more layers, neurons per layer\n2. Different activations, optimizers, learning rate schedulers, etc.\n3. Different regularizations like batch normalizations, L2 regularizations, drop outs\n4. Any other trick?\n\nAny resource which can teach about these tricks.", "votes": 1 }, { "id": 1638394, "postDate": "2022-01-04T17:25:57.157Z", "content": "You can get >90% accuracy with very few neurons actually. I seem to remember watching a video by 3b1b who used 2 layers with just 16 neurons in each and this achieved something like a 96% accuracy. Try seeing how low you can go..
", "rawMarkdown": "You can get >90% accuracy with very few neurons actually. I seem to remember watching a video by 3b1b who used 2 layers with just 16 neurons in each and this achieved something like a 96% accuracy. Try seeing how low you can go..", "votes": 1 }, { "id": 1646273, "postDate": "2022-01-11T15:42:58.530Z", "content": "Hey, Jatin. That is an interesting question.
\nCNNs take advantage of translation equivariance. So maybe a FCN could catch up by using data augmentation. If you can't be translation equivariant, train with translated samples explicitly. You can do that with ImageDataGenerator from Keras. This notebook also has an example specific for MNIST.
\nHope it helps, and please, show us your results when they're available :)
\nEDIT: btw, just noticed that you're using sigmoid activations. Did you try ReLU? Also, you don't need to choose between dropout or batchnorm on each layer, you can do both (just be sure to put dropout after batchnorm).
", "rawMarkdown": "Hey, Jatin. That is an interesting question. \n\nCNNs take advantage of translation equivariance. So maybe a FCN could catch up by using data augmentation. If you can't be translation equivariant, train with translated samples explicitly. You can do that with [ImageDataGenerator](https://www.tensorflow.org/api_docs/python/tf/keras/preprocessing/image/ImageDataGenerator) from Keras. [This notebook](https://www.kaggle.com/cdeotte/25-million-images-0-99757-mnist/notebook) also has an example specific for MNIST. \n\nHope it helps, and please, show us your results when they're available :)\n\nEDIT: btw, just noticed that you're using sigmoid activations. Did you try ReLU? Also, you don't need to choose between dropout or batchnorm on each layer, you can do both (just be sure to put dropout after batchnorm)." } ], "comments": [ { "id": 1640777, "author_name": "Jatin Patni", "author_url": "", "post_date": "2022-01-06T19:56:54.433000", "content": "I am unable to break past 97% training accuracy using the FCN below. (Although we can easily reach >99.3% using a CNN, but I want to test the performance of the best possible FCN for this problem)
\nn = 128\ninputs = tf.keras.layers.Input(shape=(784))\nbn_layer = tf.keras.layers.BatchNormalization()(inputs)\ndense_layer_new_2 = tf.keras.layers.Dense(n, activation='sigmoid')(bn_layer)\ndp1_layer = tf.keras.layers.Dropout(0.2)(dense_layer_new_2)\ndense_layer_new_3 = tf.keras.layers.Dense(n, activation='sigmoid')(dp1_layer)\nbn_layer2 = tf.keras.layers.BatchNormalization()(dense_layer_new_3)\ndense_layer_new_4 = tf.keras.layers.Dense(n, activation='sigmoid')(bn_layer2)\ndp3_layer = tf.keras.layers.Dropout(0.2)(dense_layer_new_4)\ndense_layer_new_5 = tf.keras.layers.Dense(n, activation='sigmoid')(dp3_layer)\nbn_layer3 = tf.keras.layers.BatchNormalization()(dense_layer_new_5)\ndense_layer_new_7 = tf.keras.layers.Dense(n, activation='sigmoid')(bn_layer3)\noutputs = tf.keras.layers.Dense(10, activation='softmax')(dense_layer_new_7)\n\nmodel_fcn = tf.keras.Model(inputs=inputs, outputs=outputs)\nmodel_fcn.compile(optimizer='adam', loss=loss, metrics=[\"accuracy\"])\nmodel_fcn.summary()\nCan someone guide me what should I try next to achieve >99% accuracy:
\nAny resource which can teach about these tricks.
", "votes": 1, "replies": [] }, { "id": 1638394, "author_name": "Samuel Cortinhas", "author_url": "", "post_date": "2022-01-04T17:25:57.157000", "content": "You can get >90% accuracy with very few neurons actually. I seem to remember watching a video by 3b1b who used 2 layers with just 16 neurons in each and this achieved something like a 96% accuracy. Try seeing how low you can go..
", "votes": 1, "replies": [] }, { "id": 1646273, "author_name": "Rafael C.P.", "author_url": "", "post_date": "2022-01-11T15:42:58.530000", "content": "Hey, Jatin. That is an interesting question.
\nCNNs take advantage of translation equivariance. So maybe a FCN could catch up by using data augmentation. If you can't be translation equivariant, train with translated samples explicitly. You can do that with ImageDataGenerator from Keras. This notebook also has an example specific for MNIST.
\nHope it helps, and please, show us your results when they're available :)
\nEDIT: btw, just noticed that you're using sigmoid activations. Did you try ReLU? Also, you don't need to choose between dropout or batchnorm on each layer, you can do both (just be sure to put dropout after batchnorm).
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1637726": "In the MNIST competition, a decent CNN with around 15K params can be used to achieve an accuracy > 90%\n\nCan this performance be achieved via a fully connected network, assuming data is not a limitation.\n\nTentatively, What size (number of neurons * number of layers) of fully connected network would it require to achieve >90% accuracy (if at all it is fundamentally possible)", "1640777": "I am unable to break past 97% training accuracy using the FCN below. (Although we can easily reach >99.3% using a CNN, but I want to test the performance of the best possible FCN for this problem)\n\n```\nn = 128\ninputs = tf.keras.layers.Input(shape=(784))\nbn_layer = tf.keras.layers.BatchNormalization()(inputs)\ndense_layer_new_2 = tf.keras.layers.Dense(n, activation='sigmoid')(bn_layer)\ndp1_layer = tf.keras.layers.Dropout(0.2)(dense_layer_new_2)\ndense_layer_new_3 = tf.keras.layers.Dense(n, activation='sigmoid')(dp1_layer)\nbn_layer2 = tf.keras.layers.BatchNormalization()(dense_layer_new_3)\ndense_layer_new_4 = tf.keras.layers.Dense(n, activation='sigmoid')(bn_layer2)\ndp3_layer = tf.keras.layers.Dropout(0.2)(dense_layer_new_4)\ndense_layer_new_5 = tf.keras.layers.Dense(n, activation='sigmoid')(dp3_layer)\nbn_layer3 = tf.keras.layers.BatchNormalization()(dense_layer_new_5)\ndense_layer_new_7 = tf.keras.layers.Dense(n, activation='sigmoid')(bn_layer3)\noutputs = tf.keras.layers.Dense(10, activation='softmax')(dense_layer_new_7)\n\nmodel_fcn = tf.keras.Model(inputs=inputs, outputs=outputs)\nmodel_fcn.compile(optimizer='adam', loss=loss, metrics=[\"accuracy\"])\nmodel_fcn.summary()\n```\n\n Can someone guide me what should I try next to achieve >99% accuracy:\n1. Change network architecture...more layers, neurons per layer\n2. Different activations, optimizers, learning rate schedulers, etc.\n3. Different regularizations like batch normalizations, L2 regularizations, drop outs\n4. Any other trick?\n\nAny resource which can teach about these tricks.", "1638394": "You can get >90% accuracy with very few neurons actually. I seem to remember watching a video by 3b1b who used 2 layers with just 16 neurons in each and this achieved something like a 96% accuracy. Try seeing how low you can go..", "1646273": "Hey, Jatin. That is an interesting question. \n\nCNNs take advantage of translation equivariance. So maybe a FCN could catch up by using data augmentation. If you can't be translation equivariant, train with translated samples explicitly. You can do that with [ImageDataGenerator](https://www.tensorflow.org/api_docs/python/tf/keras/preprocessing/image/ImageDataGenerator) from Keras. [This notebook](https://www.kaggle.com/cdeotte/25-million-images-0-99757-mnist/notebook) also has an example specific for MNIST. \n\nHope it helps, and please, show us your results when they're available :)\n\nEDIT: btw, just noticed that you're using sigmoid activations. Did you try ReLU? Also, you don't need to choose between dropout or batchnorm on each layer, you can do both (just be sure to put dropout after batchnorm)." } }