I just want to share a simple convolutional neural network that achieves a good result (90%+ accuracy on test data) in around 100-125 epochs. I can't provide a fixed seed for reproducibility because i'm not able to make it to work with tensorflow (if anyone knows a fix, i would be glad to hear it!)
\n\nimport keras\nfrom keras.models import Sequential\nfrom keras.utils import np_utils\nfrom keras.preprocessing.image import ImageDataGenerator\nfrom keras.layers import Dense, Activation, Flatten, Dropout, BatchNormalization\nfrom keras.layers import Conv2D, MaxPooling2D\nfrom keras.datasets import cifar10\nfrom keras import regularizers, optimizers\nimport numpy as np\n\n\n(x_train, y_train), (x_test, y_test) = cifar10.load_data()\nx_train = x_train.astype('float32')\nx_test = x_test.astype('float32')\n\n#z-score\nmean = np.mean(x_train,axis=(0,1,2,3))\nstd = np.std(x_train,axis=(0,1,2,3))\nx_train = (x_train-mean)/(std+1e-7)\nx_test = (x_test-mean)/(std+1e-7)\n\nnum_classes = 10\ny_train = np_utils.to_categorical(y_train,num_classes)\ny_test = np_utils.to_categorical(y_test,num_classes)\n\nbaseMapNum = 32\nweight_decay = 1e-4\nmodel = Sequential()\nmodel.add(Conv2D(baseMapNum, (3,3), padding='same', kernel_regularizer=regularizers.l2(weight_decay), input_shape=x_train.shape[1:]))\nmodel.add(Activation('relu'))\nmodel.add(BatchNormalization())\nmodel.add(Conv2D(baseMapNum, (3,3), padding='same', kernel_regularizer=regularizers.l2(weight_decay)))\nmodel.add(Activation('relu'))\nmodel.add(BatchNormalization())\nmodel.add(MaxPooling2D(pool_size=(2,2)))\nmodel.add(Dropout(0.2))\n\nmodel.add(Conv2D(2*baseMapNum, (3,3), padding='same', kernel_regularizer=regularizers.l2(weight_decay)))\nmodel.add(Activation('relu'))\nmodel.add(BatchNormalization())\nmodel.add(Conv2D(2*baseMapNum, (3,3), padding='same', kernel_regularizer=regularizers.l2(weight_decay)))\nmodel.add(Activation('relu'))\nmodel.add(BatchNormalization())\nmodel.add(MaxPooling2D(pool_size=(2,2)))\nmodel.add(Dropout(0.3))\n\nmodel.add(Conv2D(4*baseMapNum, (3,3), padding='same', kernel_regularizer=regularizers.l2(weight_decay)))\nmodel.add(Activation('relu'))\nmodel.add(BatchNormalization())\nmodel.add(Conv2D(4*baseMapNum, (3,3), padding='same', kernel_regularizer=regularizers.l2(weight_decay)))\nmodel.add(Activation('relu'))\nmodel.add(BatchNormalization())\nmodel.add(MaxPooling2D(pool_size=(2,2)))\nmodel.add(Dropout(0.4))\n\nmodel.add(Flatten())\nmodel.add(Dense(num_classes, activation='softmax'))\n\nmodel.summary()\n\n#data augmentation\ndatagen = ImageDataGenerator(\n featurewise_center=False,\n samplewise_center=False,\n featurewise_std_normalization=False,\n samplewise_std_normalization=False,\n zca_whitening=False,\n rotation_range=15,\n width_shift_range=0.1,\n height_shift_range=0.1,\n horizontal_flip=True,\n vertical_flip=False\n )\ndatagen.fit(x_train)\n\n#training\nbatch_size = 64\nepochs=25\nopt_rms = keras.optimizers.rmsprop(lr=0.001,decay=1e-6)\nmodel.compile(loss='categorical_crossentropy',\n optimizer=opt_rms,\n metrics=['accuracy'])\nmodel.fit_generator(datagen.flow(x_train, y_train, batch_size=batch_size),steps_per_epoch=x_train.shape[0] // batch_size,epochs=3*epochs,verbose=1,validation_data=(x_test,y_test))\nmodel.save_weights('cifar10_normal_rms_ep75.h5')\n\nopt_rms = keras.optimizers.rmsprop(lr=0.0005,decay=1e-6)\nmodel.compile(loss='categorical_crossentropy',\n optimizer=opt_rms,\n metrics=['accuracy'])\nmodel.fit_generator(datagen.flow(x_train, y_train, batch_size=batch_size),steps_per_epoch=x_train.shape[0] // batch_size,epochs=epochs,verbose=1,validation_data=(x_test,y_test))\nmodel.save_weights('cifar10_normal_rms_ep100.h5')\n\nopt_rms = keras.optimizers.rmsprop(lr=0.0003,decay=1e-6)\nmodel.compile(loss='categorical_crossentropy',\n optimizer=opt_rms,\n metrics=['accuracy'])\nmodel.fit_generator(datagen.flow(x_train, y_train, batch_size=batch_size),steps_per_epoch=x_train.shape[0] // batch_size,epochs=epochs,verbose=1,validation_data=(x_test,y_test))\nmodel.save_weights('cifar10_normal_rms_ep125.h5')\n\n#testing - no kaggle eval\nscores = model.evaluate(x_test, y_test, batch_size=128, verbose=1)\nprint('\\nTest result: %.3f loss: %.3f' % (scores[1]*100,scores[0]))\nI just want to share a simple convolutional neural network that achieves a good result (90%+ accuracy on test data) in around 100-125 epochs. I can't provide a fixed seed for reproducibility because i'm not able to make it to work with tensorflow (if anyone knows a fix, i would be glad to hear it!)
\n\nimport keras\nfrom keras.models import Sequential\nfrom keras.utils import np_utils\nfrom keras.preprocessing.image import ImageDataGenerator\nfrom keras.layers import Dense, Activation, Flatten, Dropout, BatchNormalization\nfrom keras.layers import Conv2D, MaxPooling2D\nfrom keras.datasets import cifar10\nfrom keras import regularizers, optimizers\nimport numpy as np\n\n\n(x_train, y_train), (x_test, y_test) = cifar10.load_data()\nx_train = x_train.astype('float32')\nx_test = x_test.astype('float32')\n\n#z-score\nmean = np.mean(x_train,axis=(0,1,2,3))\nstd = np.std(x_train,axis=(0,1,2,3))\nx_train = (x_train-mean)/(std+1e-7)\nx_test = (x_test-mean)/(std+1e-7)\n\nnum_classes = 10\ny_train = np_utils.to_categorical(y_train,num_classes)\ny_test = np_utils.to_categorical(y_test,num_classes)\n\nbaseMapNum = 32\nweight_decay = 1e-4\nmodel = Sequential()\nmodel.add(Conv2D(baseMapNum, (3,3), padding='same', kernel_regularizer=regularizers.l2(weight_decay), input_shape=x_train.shape[1:]))\nmodel.add(Activation('relu'))\nmodel.add(BatchNormalization())\nmodel.add(Conv2D(baseMapNum, (3,3), padding='same', kernel_regularizer=regularizers.l2(weight_decay)))\nmodel.add(Activation('relu'))\nmodel.add(BatchNormalization())\nmodel.add(MaxPooling2D(pool_size=(2,2)))\nmodel.add(Dropout(0.2))\n\nmodel.add(Conv2D(2*baseMapNum, (3,3), padding='same', kernel_regularizer=regularizers.l2(weight_decay)))\nmodel.add(Activation('relu'))\nmodel.add(BatchNormalization())\nmodel.add(Conv2D(2*baseMapNum, (3,3), padding='same', kernel_regularizer=regularizers.l2(weight_decay)))\nmodel.add(Activation('relu'))\nmodel.add(BatchNormalization())\nmodel.add(MaxPooling2D(pool_size=(2,2)))\nmodel.add(Dropout(0.3))\n\nmodel.add(Conv2D(4*baseMapNum, (3,3), padding='same', kernel_regularizer=regularizers.l2(weight_decay)))\nmodel.add(Activation('relu'))\nmodel.add(BatchNormalization())\nmodel.add(Conv2D(4*baseMapNum, (3,3), padding='same', kernel_regularizer=regularizers.l2(weight_decay)))\nmodel.add(Activation('relu'))\nmodel.add(BatchNormalization())\nmodel.add(MaxPooling2D(pool_size=(2,2)))\nmodel.add(Dropout(0.4))\n\nmodel.add(Flatten())\nmodel.add(Dense(num_classes, activation='softmax'))\n\nmodel.summary()\n\n#data augmentation\ndatagen = ImageDataGenerator(\n featurewise_center=False,\n samplewise_center=False,\n featurewise_std_normalization=False,\n samplewise_std_normalization=False,\n zca_whitening=False,\n rotation_range=15,\n width_shift_range=0.1,\n height_shift_range=0.1,\n horizontal_flip=True,\n vertical_flip=False\n )\ndatagen.fit(x_train)\n\n#training\nbatch_size = 64\nepochs=25\nopt_rms = keras.optimizers.rmsprop(lr=0.001,decay=1e-6)\nmodel.compile(loss='categorical_crossentropy',\n optimizer=opt_rms,\n metrics=['accuracy'])\nmodel.fit_generator(datagen.flow(x_train, y_train, batch_size=batch_size),steps_per_epoch=x_train.shape[0] // batch_size,epochs=3*epochs,verbose=1,validation_data=(x_test,y_test))\nmodel.save_weights('cifar10_normal_rms_ep75.h5')\n\nopt_rms = keras.optimizers.rmsprop(lr=0.0005,decay=1e-6)\nmodel.compile(loss='categorical_crossentropy',\n optimizer=opt_rms,\n metrics=['accuracy'])\nmodel.fit_generator(datagen.flow(x_train, y_train, batch_size=batch_size),steps_per_epoch=x_train.shape[0] // batch_size,epochs=epochs,verbose=1,validation_data=(x_test,y_test))\nmodel.save_weights('cifar10_normal_rms_ep100.h5')\n\nopt_rms = keras.optimizers.rmsprop(lr=0.0003,decay=1e-6)\nmodel.compile(loss='categorical_crossentropy',\n optimizer=opt_rms,\n metrics=['accuracy'])\nmodel.fit_generator(datagen.flow(x_train, y_train, batch_size=batch_size),steps_per_epoch=x_train.shape[0] // batch_size,epochs=epochs,verbose=1,validation_data=(x_test,y_test))\nmodel.save_weights('cifar10_normal_rms_ep125.h5')\n\n#testing - no kaggle eval\nscores = model.evaluate(x_test, y_test, batch_size=128, verbose=1)\nprint('\\nTest result: %.3f loss: %.3f' % (scores[1]*100,scores[0]))\nIt works like a charm. \nI wonder how much effort you need to put into finding the right CNN design and correct hyper parameter. ?\nSeems like data augmentation and regularization is the main key in this architecture.
", "rawMarkdown": "It works like a charm. \nI wonder how much effort you need to put into finding the right CNN design and correct hyper parameter. ?\nSeems like data augmentation and regularization is the main key in this architecture.", "votes": null }, { "id": "247365", "postDate": "11/22/2017 22:32:24", "content": "Hmm, had to remove the batch_noormalization layers bc it didn't work on my gpu (not the best one in the market...) got 89% with ReLU after the 125 epochs and removing the kerneñ_initializer too...
", "rawMarkdown": "Hmm, had to remove the batch_noormalization layers bc it didn't work on my gpu (not the best one in the market...) got 89% with ReLU after the 125 epochs and removing the kerneñ_initializer too...", "votes": null }, { "id": "248642", "postDate": "11/26/2017 17:10:43", "content": "How much accuracy did you get? pls urgent
", "rawMarkdown": "How much accuracy did you get? pls urgent", "votes": null }, { "id": "248651", "postDate": "11/26/2017 17:30:02", "content": "I got just above 90% accuracy. Running over few times, u can say 90%+.\nI ran on nvidia geforce GTX 1080
", "rawMarkdown": "I got just above 90% accuracy. Running over few times, u can say 90%+.\nI ran on nvidia geforce GTX 1080", "votes": null }, { "id": "248734", "postDate": "11/26/2017 21:22:03", "content": "Okay okay above 90%... but how much is that? For example i got 91.20 for RELU activation function with 250 epochs (keeping the rule of dividing by 2 the learning rate every 25 epochs...), I also reached the same accuracy ~91.25% for the ELU activation function but I saw that ELU learned much faster than RELU.
", "rawMarkdown": "Okay okay above 90%... but how much is that? For example i got 91.20 for RELU activation function with 250 epochs (keeping the rule of dividing by 2 the learning rate every 25 epochs...), I also reached the same accuracy ~91.25% for the ELU activation function but I saw that ELU learned much faster than RELU.", "votes": null }, { "id": "248736", "postDate": "11/26/2017 21:23:00", "content": "I want to see how sensitive the network is to different activation functions and how much do the random weight initialization affects it..
", "rawMarkdown": "I want to see how sensitive the network is to different activation functions and how much do the random weight initialization affects it..", "votes": null }, { "id": "248847", "postDate": "11/27/2017 04:54:00", "content": "I am on work now. I will tell you exactly how much i got because i do not remember the previous results.\nWill also run with 'elu' and let you know the results. Performing it for 125 epoch and changing the learning rate after 75 epochs and then after 25 epochs dividing by 2. (exactly the code you have given here)
", "rawMarkdown": "I am on work now. I will tell you exactly how much i got because i do not remember the previous results.\nWill also run with 'elu' and let you know the results. Performing it for 125 epoch and changing the learning rate after 75 epochs and then after 25 epochs dividing by 2. (exactly the code you have given here)", "votes": null }, { "id": "248865", "postDate": "11/27/2017 05:51:14", "content": "With 125 epoch as per above code, with \"relu\" activation function, validation accuracy reached 89.24. \nSame code when ran with \"elu\" activation function, validation accuracy reached 90.30.
\n\nIt is more or less similar. It also depends on random initialization. It may go to 91% if the starting random positions are good enough. Also, Have not tested for 250 epochs, may be with 250 epochs it will reach 91%.
", "rawMarkdown": "With 125 epoch as per above code, with \"relu\" activation function, validation accuracy reached 89.24. \nSame code when ran with \"elu\" activation function, validation accuracy reached 90.30.\n\nIt is more or less similar. It also depends on random initialization. It may go to 91% if the starting random positions are good enough. Also, Have not tested for 250 epochs, may be with 250 epochs it will reach 91%.", "votes": null }, { "id": "253886", "postDate": "12/05/2017 19:46:51", "content": "How did you design your architecture? would you please answer as soon as possible? Thanks.
", "rawMarkdown": "How did you design your architecture? would you please answer as soon as possible? Thanks.", "votes": null }, { "id": "253912", "postDate": "12/05/2017 21:09:13", "content": "conv-relu-batchnorm is a common triplet. Pooling reduces signal and makes the model more robust against spatial invariance, so it can not be used very much, just the exact amount of maxpooling will make the model work fine. We need a fully connected layer at the end to classify the images to their classes.\nThe combination of L2 + batchnorm will act as a regularizer, preventing overfitting and keeping the weights small so that the model is able to generalize pretty well. \nThis architecture is quite effective if we measure the quotient between params, complexity, time to train... and accuracy.
", "rawMarkdown": "conv-relu-batchnorm is a common triplet. Pooling reduces signal and makes the model more robust against spatial invariance, so it can not be used very much, just the exact amount of maxpooling will make the model work fine. We need a fully connected layer at the end to classify the images to their classes.\nThe combination of L2 + batchnorm will act as a regularizer, preventing overfitting and keeping the weights small so that the model is able to generalize pretty well. \nThis architecture is quite effective if we measure the quotient between params, complexity, time to train... and accuracy.", "votes": null }, { "id": "297811", "postDate": "03/18/2018 01:39:47", "content": "I was on a plane ride to Honolulu (6hrs) and gave this a shot. I was able to get 89...something with a much smaller neural net and only 75% of training data. The validation data was the test data set. I basically tossed out 25% of training data. I had my personal laptop and training was taking a while so I wanted to build something that wouldn't kill all my battery life. I hit 87% at 40 epochs and killed it off once it hit 89. I had 4 convolutions of 64,64 and 32,32. My intention was to run it for 200 epochs but I killed it off because I wanted to run something else. Thus, I believe the model can be further simplified; I'm not convinced that you need a big model. I also didn't use any padding.
", "rawMarkdown": "I was on a plane ride to Honolulu (6hrs) and gave this a shot. I was able to get 89...something with a much smaller neural net and only 75% of training data. The validation data was the test data set. I basically tossed out 25% of training data. I had my personal laptop and training was taking a while so I wanted to build something that wouldn't kill all my battery life. I hit 87% at 40 epochs and killed it off once it hit 89. I had 4 convolutions of 64,64 and 32,32. My intention was to run it for 200 epochs but I killed it off because I wanted to run something else. Thus, I believe the model can be further simplified; I'm not convinced that you need a big model. I also didn't use any padding.", "votes": null }, { "id": "308290", "postDate": "04/03/2018 09:26:55", "content": "Isn't that batch normalization to be applied before RELU ?
", "rawMarkdown": "Isn't that batch normalization to be applied before RELU ?", "votes": null }, { "id": "330952", "postDate": "05/20/2018 03:59:04", "content": "Yes Roshan, batch normalization is more often done before the activation function. Since the goal here is to normalize the weights and bias of the next layer, some do it after the activation function as well.
", "rawMarkdown": "Yes Roshan, batch normalization is more often done before the activation function. Since the goal here is to normalize the weights and bias of the next layer, some do it after the activation function as well.", "votes": null }, { "id": "372643", "postDate": "08/20/2018 01:23:29", "content": "Could you show your code here.
", "rawMarkdown": "Could you show your code here.", "votes": null }, { "id": "387666", "postDate": "09/15/2018 11:42:52", "content": "Does the size of the images affect the accuracy or depth of our model ? Can we increase the size or it wont be necessary ?
", "rawMarkdown": "Does the size of the images affect the accuracy or depth of our model ? Can we increase the size or it wont be necessary ?", "votes": null }, { "id": "412724", "postDate": "10/30/2018 17:07:24", "content": "Could someone state how much he's getting at 5, 10, 20, 50 epochs?\nI'm trying to reproduce this network but can't get more than 80% in a short time (10-20 epochs) I'm probably missing smthing.
", "rawMarkdown": "Could someone state how much he's getting at 5, 10, 20, 50 epochs?\nI'm trying to reproduce this network but can't get more than 80% in a short time (10-20 epochs) I'm probably missing smthing.", "votes": null }, { "id": "419397", "postDate": "11/11/2018 22:39:10", "content": "Did it myself.\n20epoch : 80% (10min gtx1050)\n40epoch : 85%\netc.\nL2 reg + data augmentation (random_cropping in resnet paper) seems to be really efficient on cifar10 because of the \"small\" dataset
", "rawMarkdown": "Did it myself.\n20epoch : 80% (10min gtx1050)\n40epoch : 85%\netc.\nL2 reg + data augmentation (random_cropping in resnet paper) seems to be really efficient on cifar10 because of the \"small\" dataset", "votes": null }, { "id": "424025", "postDate": "11/19/2018 12:57:13", "content": "I can also confirm that RMSProp seems much efficient than Adam, for this kind of neural network (it needs to be verified for a resnet). One should try to examine why.
", "rawMarkdown": "I can also confirm that RMSProp seems much efficient than Adam, for this kind of neural network (it needs to be verified for a resnet). One should try to examine why.", "votes": null }, { "id": "581562", "postDate": "07/22/2019 05:26:22", "content": "Has anyone tried it without the data augmentation?\nAlso, does anyone know the amount of parameters this model needs to be trained?
", "rawMarkdown": "Has anyone tried it without the data augmentation?\nAlso, does anyone know the amount of parameters this model needs to be trained?", "votes": null }, { "id": "588909", "postDate": "07/31/2019 08:07:07", "content": "It's your choice. Nowadays we apply batch normalization after activation as its faster. Activation forwards only positive weights to the next layer. If we do batch normalization after activation we normalize only positive weights.
", "rawMarkdown": "It's your choice. Nowadays we apply batch normalization after activation as its faster. Activation forwards only positive weights to the next layer. If we do batch normalization after activation we normalize only positive weights.", "votes": null } ], "comments": [ { "id": 244978, "author_name": "abhijeet3922", "author_url": "", "post_date": "11/17/2017 11:01:54", "content": "It works like a charm. \nI wonder how much effort you need to put into finding the right CNN design and correct hyper parameter. ?\nSeems like data augmentation and regularization is the main key in this architecture.
", "votes": null, "replies": [] }, { "id": 247365, "author_name": "ericalcaide9834", "author_url": "", "post_date": "11/22/2017 22:32:24", "content": "Hmm, had to remove the batch_noormalization layers bc it didn't work on my gpu (not the best one in the market...) got 89% with ReLU after the 125 epochs and removing the kerneñ_initializer too...
", "votes": null, "replies": [] }, { "id": 248642, "author_name": "ericalcaide9834", "author_url": "", "post_date": "11/26/2017 17:10:43", "content": "How much accuracy did you get? pls urgent
", "votes": null, "replies": [ { "id": 248651, "author_name": "abhijeet3922", "author_url": "", "post_date": "11/26/2017 17:30:02", "content": "I got just above 90% accuracy. Running over few times, u can say 90%+.\nI ran on nvidia geforce GTX 1080
", "votes": null, "replies": [] }, { "id": 248734, "author_name": "ericalcaide9834", "author_url": "", "post_date": "11/26/2017 21:22:03", "content": "Okay okay above 90%... but how much is that? For example i got 91.20 for RELU activation function with 250 epochs (keeping the rule of dividing by 2 the learning rate every 25 epochs...), I also reached the same accuracy ~91.25% for the ELU activation function but I saw that ELU learned much faster than RELU.
", "votes": null, "replies": [] }, { "id": 248736, "author_name": "ericalcaide9834", "author_url": "", "post_date": "11/26/2017 21:23:00", "content": "I want to see how sensitive the network is to different activation functions and how much do the random weight initialization affects it..
", "votes": null, "replies": [] }, { "id": 248865, "author_name": "abhijeet3922", "author_url": "", "post_date": "11/27/2017 05:51:14", "content": "With 125 epoch as per above code, with \"relu\" activation function, validation accuracy reached 89.24. \nSame code when ran with \"elu\" activation function, validation accuracy reached 90.30.
\n\nIt is more or less similar. It also depends on random initialization. It may go to 91% if the starting random positions are good enough. Also, Have not tested for 250 epochs, may be with 250 epochs it will reach 91%.
", "votes": null, "replies": [] } ] }, { "id": 248847, "author_name": "abhijeet3922", "author_url": "", "post_date": "11/27/2017 04:54:00", "content": "I am on work now. I will tell you exactly how much i got because i do not remember the previous results.\nWill also run with 'elu' and let you know the results. Performing it for 125 epoch and changing the learning rate after 75 epochs and then after 25 epochs dividing by 2. (exactly the code you have given here)
", "votes": null, "replies": [] }, { "id": 253886, "author_name": "eliesl", "author_url": "", "post_date": "12/05/2017 19:46:51", "content": "How did you design your architecture? would you please answer as soon as possible? Thanks.
", "votes": null, "replies": [ { "id": 253912, "author_name": "ericalcaide9834", "author_url": "", "post_date": "12/05/2017 21:09:13", "content": "conv-relu-batchnorm is a common triplet. Pooling reduces signal and makes the model more robust against spatial invariance, so it can not be used very much, just the exact amount of maxpooling will make the model work fine. We need a fully connected layer at the end to classify the images to their classes.\nThe combination of L2 + batchnorm will act as a regularizer, preventing overfitting and keeping the weights small so that the model is able to generalize pretty well. \nThis architecture is quite effective if we measure the quotient between params, complexity, time to train... and accuracy.
", "votes": null, "replies": [] } ] }, { "id": 297811, "author_name": "johnnyboycurtis", "author_url": "", "post_date": "03/18/2018 01:39:47", "content": "I was on a plane ride to Honolulu (6hrs) and gave this a shot. I was able to get 89...something with a much smaller neural net and only 75% of training data. The validation data was the test data set. I basically tossed out 25% of training data. I had my personal laptop and training was taking a while so I wanted to build something that wouldn't kill all my battery life. I hit 87% at 40 epochs and killed it off once it hit 89. I had 4 convolutions of 64,64 and 32,32. My intention was to run it for 200 epochs but I killed it off because I wanted to run something else. Thus, I believe the model can be further simplified; I'm not convinced that you need a big model. I also didn't use any padding.
", "votes": null, "replies": [ { "id": 372643, "author_name": "shangweichen", "author_url": "", "post_date": "08/20/2018 01:23:29", "content": "Could you show your code here.
", "votes": null, "replies": [] } ] }, { "id": 308290, "author_name": "roshangopalakrishnan", "author_url": "", "post_date": "04/03/2018 09:26:55", "content": "Isn't that batch normalization to be applied before RELU ?
", "votes": null, "replies": [ { "id": 330952, "author_name": "ravitee", "author_url": "", "post_date": "05/20/2018 03:59:04", "content": "Yes Roshan, batch normalization is more often done before the activation function. Since the goal here is to normalize the weights and bias of the next layer, some do it after the activation function as well.
", "votes": null, "replies": [] }, { "id": 588909, "author_name": "joyjeni", "author_url": "", "post_date": "07/31/2019 08:07:07", "content": "It's your choice. Nowadays we apply batch normalization after activation as its faster. Activation forwards only positive weights to the next layer. If we do batch normalization after activation we normalize only positive weights.
", "votes": null, "replies": [] } ] }, { "id": 387666, "author_name": "ibrahimsherify", "author_url": "", "post_date": "09/15/2018 11:42:52", "content": "Does the size of the images affect the accuracy or depth of our model ? Can we increase the size or it wont be necessary ?
", "votes": null, "replies": [] }, { "id": 412724, "author_name": "", "author_url": "", "post_date": "10/30/2018 17:07:24", "content": "Could someone state how much he's getting at 5, 10, 20, 50 epochs?\nI'm trying to reproduce this network but can't get more than 80% in a short time (10-20 epochs) I'm probably missing smthing.
", "votes": null, "replies": [ { "id": 419397, "author_name": "", "author_url": "", "post_date": "11/11/2018 22:39:10", "content": "Did it myself.\n20epoch : 80% (10min gtx1050)\n40epoch : 85%\netc.\nL2 reg + data augmentation (random_cropping in resnet paper) seems to be really efficient on cifar10 because of the \"small\" dataset
", "votes": null, "replies": [] }, { "id": 424025, "author_name": "", "author_url": "", "post_date": "11/19/2018 12:57:13", "content": "I can also confirm that RMSProp seems much efficient than Adam, for this kind of neural network (it needs to be verified for a resnet). One should try to examine why.
", "votes": null, "replies": [] } ] }, { "id": 581562, "author_name": "lpintoc", "author_url": "", "post_date": "07/22/2019 05:26:22", "content": "Has anyone tried it without the data augmentation?\nAlso, does anyone know the amount of parameters this model needs to be trained?
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "225695": "I just want to share a simple convolutional neural network that achieves a good result (90%+ accuracy on test data) in around 100-125 epochs. I can't provide a fixed seed for reproducibility because i'm not able to make it to work with tensorflow (if anyone knows a fix, i would be glad to hear it!)\n\n import keras\n from keras.models import Sequential\n from keras.utils import np_utils\n from keras.preprocessing.image import ImageDataGenerator\n from keras.layers import Dense, Activation, Flatten, Dropout, BatchNormalization\n from keras.layers import Conv2D, MaxPooling2D\n from keras.datasets import cifar10\n from keras import regularizers, optimizers\n import numpy as np\n \n \n (x_train, y_train), (x_test, y_test) = cifar10.load_data()\n x_train = x_train.astype('float32')\n x_test = x_test.astype('float32')\n \n #z-score\n mean = np.mean(x_train,axis=(0,1,2,3))\n std = np.std(x_train,axis=(0,1,2,3))\n x_train = (x_train-mean)/(std+1e-7)\n x_test = (x_test-mean)/(std+1e-7)\n \n num_classes = 10\n y_train = np_utils.to_categorical(y_train,num_classes)\n y_test = np_utils.to_categorical(y_test,num_classes)\n \n baseMapNum = 32\n weight_decay = 1e-4\n model = Sequential()\n model.add(Conv2D(baseMapNum, (3,3), padding='same', kernel_regularizer=regularizers.l2(weight_decay), input_shape=x_train.shape[1:]))\n model.add(Activation('relu'))\n model.add(BatchNormalization())\n model.add(Conv2D(baseMapNum, (3,3), padding='same', kernel_regularizer=regularizers.l2(weight_decay)))\n model.add(Activation('relu'))\n model.add(BatchNormalization())\n model.add(MaxPooling2D(pool_size=(2,2)))\n model.add(Dropout(0.2))\n \n model.add(Conv2D(2*baseMapNum, (3,3), padding='same', kernel_regularizer=regularizers.l2(weight_decay)))\n model.add(Activation('relu'))\n model.add(BatchNormalization())\n model.add(Conv2D(2*baseMapNum, (3,3), padding='same', kernel_regularizer=regularizers.l2(weight_decay)))\n model.add(Activation('relu'))\n model.add(BatchNormalization())\n model.add(MaxPooling2D(pool_size=(2,2)))\n model.add(Dropout(0.3))\n \n model.add(Conv2D(4*baseMapNum, (3,3), padding='same', kernel_regularizer=regularizers.l2(weight_decay)))\n model.add(Activation('relu'))\n model.add(BatchNormalization())\n model.add(Conv2D(4*baseMapNum, (3,3), padding='same', kernel_regularizer=regularizers.l2(weight_decay)))\n model.add(Activation('relu'))\n model.add(BatchNormalization())\n model.add(MaxPooling2D(pool_size=(2,2)))\n model.add(Dropout(0.4))\n \n model.add(Flatten())\n model.add(Dense(num_classes, activation='softmax'))\n \n model.summary()\n \n #data augmentation\n datagen = ImageDataGenerator(\n \tfeaturewise_center=False,\n \tsamplewise_center=False,\n \tfeaturewise_std_normalization=False,\n \tsamplewise_std_normalization=False,\n \tzca_whitening=False,\n \trotation_range=15,\n \twidth_shift_range=0.1,\n \theight_shift_range=0.1,\n \thorizontal_flip=True,\n \tvertical_flip=False\n \t)\n datagen.fit(x_train)\n \n #training\n batch_size = 64\n epochs=25\n opt_rms = keras.optimizers.rmsprop(lr=0.001,decay=1e-6)\n model.compile(loss='categorical_crossentropy',\n \t\toptimizer=opt_rms,\n \t\tmetrics=['accuracy'])\n model.fit_generator(datagen.flow(x_train, y_train, batch_size=batch_size),steps_per_epoch=x_train.shape[0] // batch_size,epochs=3*epochs,verbose=1,validation_data=(x_test,y_test))\n model.save_weights('cifar10_normal_rms_ep75.h5')\n \n opt_rms = keras.optimizers.rmsprop(lr=0.0005,decay=1e-6)\n model.compile(loss='categorical_crossentropy',\n \t\toptimizer=opt_rms,\n \t\tmetrics=['accuracy'])\n model.fit_generator(datagen.flow(x_train, y_train, batch_size=batch_size),steps_per_epoch=x_train.shape[0] // batch_size,epochs=epochs,verbose=1,validation_data=(x_test,y_test))\n model.save_weights('cifar10_normal_rms_ep100.h5')\n \n opt_rms = keras.optimizers.rmsprop(lr=0.0003,decay=1e-6)\n model.compile(loss='categorical_crossentropy',\n \t\toptimizer=opt_rms,\n \t\tmetrics=['accuracy'])\n model.fit_generator(datagen.flow(x_train, y_train, batch_size=batch_size),steps_per_epoch=x_train.shape[0] // batch_size,epochs=epochs,verbose=1,validation_data=(x_test,y_test))\n model.save_weights('cifar10_normal_rms_ep125.h5')\n \n #testing - no kaggle eval\n scores = model.evaluate(x_test, y_test, batch_size=128, verbose=1)\n print('\\nTest result: %.3f loss: %.3f' % (scores[1]*100,scores[0]))", "244978": "It works like a charm. \nI wonder how much effort you need to put into finding the right CNN design and correct hyper parameter. ?\nSeems like data augmentation and regularization is the main key in this architecture.", "247365": "Hmm, had to remove the batch_noormalization layers bc it didn't work on my gpu (not the best one in the market...) got 89% with ReLU after the 125 epochs and removing the kerneñ_initializer too...", "248642": "How much accuracy did you get? pls urgent", "248651": "I got just above 90% accuracy. Running over few times, u can say 90%+.\nI ran on nvidia geforce GTX 1080", "248734": "Okay okay above 90%... but how much is that? For example i got 91.20 for RELU activation function with 250 epochs (keeping the rule of dividing by 2 the learning rate every 25 epochs...), I also reached the same accuracy ~91.25% for the ELU activation function but I saw that ELU learned much faster than RELU.", "248736": "I want to see how sensitive the network is to different activation functions and how much do the random weight initialization affects it..", "248847": "I am on work now. I will tell you exactly how much i got because i do not remember the previous results.\nWill also run with 'elu' and let you know the results. Performing it for 125 epoch and changing the learning rate after 75 epochs and then after 25 epochs dividing by 2. (exactly the code you have given here)", "248865": "With 125 epoch as per above code, with \"relu\" activation function, validation accuracy reached 89.24. \nSame code when ran with \"elu\" activation function, validation accuracy reached 90.30.\n\nIt is more or less similar. It also depends on random initialization. It may go to 91% if the starting random positions are good enough. Also, Have not tested for 250 epochs, may be with 250 epochs it will reach 91%.", "253886": "How did you design your architecture? would you please answer as soon as possible? Thanks.", "253912": "conv-relu-batchnorm is a common triplet. Pooling reduces signal and makes the model more robust against spatial invariance, so it can not be used very much, just the exact amount of maxpooling will make the model work fine. We need a fully connected layer at the end to classify the images to their classes.\nThe combination of L2 + batchnorm will act as a regularizer, preventing overfitting and keeping the weights small so that the model is able to generalize pretty well. \nThis architecture is quite effective if we measure the quotient between params, complexity, time to train... and accuracy.", "297811": "I was on a plane ride to Honolulu (6hrs) and gave this a shot. I was able to get 89...something with a much smaller neural net and only 75% of training data. The validation data was the test data set. I basically tossed out 25% of training data. I had my personal laptop and training was taking a while so I wanted to build something that wouldn't kill all my battery life. I hit 87% at 40 epochs and killed it off once it hit 89. I had 4 convolutions of 64,64 and 32,32. My intention was to run it for 200 epochs but I killed it off because I wanted to run something else. Thus, I believe the model can be further simplified; I'm not convinced that you need a big model. I also didn't use any padding.", "308290": "Isn't that batch normalization to be applied before RELU ?", "330952": "Yes Roshan, batch normalization is more often done before the activation function. Since the goal here is to normalize the weights and bias of the next layer, some do it after the activation function as well.", "372643": "Could you show your code here.", "387666": "Does the size of the images affect the accuracy or depth of our model ? Can we increase the size or it wont be necessary ?", "412724": "Could someone state how much he's getting at 5, 10, 20, 50 epochs?\nI'm trying to reproduce this network but can't get more than 80% in a short time (10-20 epochs) I'm probably missing smthing.", "419397": "Did it myself.\n20epoch : 80% (10min gtx1050)\n40epoch : 85%\netc.\nL2 reg + data augmentation (random_cropping in resnet paper) seems to be really efficient on cifar10 because of the \"small\" dataset", "424025": "I can also confirm that RMSProp seems much efficient than Adam, for this kind of neural network (it needs to be verified for a resnet). One should try to examine why.", "581562": "Has anyone tried it without the data augmentation?\nAlso, does anyone know the amount of parameters this model needs to be trained?", "588909": "It's your choice. Nowadays we apply batch normalization after activation as its faster. Activation forwards only positive weights to the next layer. If we do batch normalization after activation we normalize only positive weights." }, "source": "meta" }