I am curious if anyone is using neural networks and what kind of results they are getting. I tried 3 layer NN with 150 neurons on the hidden layer, it ran about 1 hr and gave me around 87.52%. If anyone tried NN, i would appreciate any help on how to go\r\n about it.
", "messages": [ { "id": 2958874, "postDate": "2024-08-14T12:53:06.667Z", "content": "I was able to get to 99.5% accuracy on public leaderboard using Convolutional Neural Nets.
\n\nThis was jointly down to:
\n\n1) borrowing from Daniel's wonderful blog on how to implement CNN in Lasagne on Images (link below)
\n\n2) Synthesizing More Training Data for such data-hungry algorithm using Image Processing libraries in Python: \n - Rotate left and right\n - zoom\n - Stretch vertically & horizontally, \n - Elastic deformations. \nThe latter is particularly interesting, in that in a broad sense it allows us to create more variants of legit hand-writing than originally available in training.
\n\n3) reducing overfit using concepts of Early Convergence and Dropouts
\n\nFinal submission was a simple average-based ensemble of Top 5 CNNets.
\n\n", "rawMarkdown": "I was able to get to 99.5% accuracy on public leaderboard using Convolutional Neural Nets. \r\n\r\nThis was jointly down to:\r\n\r\n1) borrowing from Daniel's wonderful blog on how to implement CNN in Lasagne on Images (link below)\r\n\r\n2) Synthesizing More Training Data for such data-hungry algorithm using Image Processing libraries in Python: \r\n - Rotate left and right\r\n - zoom\r\n - Stretch vertically & horizontally, \r\n - Elastic deformations. \r\nThe latter is particularly interesting, in that in a broad sense it allows us to create more variants of legit hand-writing than originally available in training.\r\n\r\n3) reducing overfit using concepts of Early Convergence and Dropouts\r\n\r\n\r\nFinal submission was a simple average-based ensemble of Top 5 CNNets.\r\n\r\nhttp://danielnouri.org/notes/2014/12/17/using-convolutional-neural-nets-to-detect-facial-keypoints-tutorial/\r\n", "votes": 4 }, { "id": 21412, "postDate": "2013-03-19T12:17:25.053Z", "content": "By far the coolest and mightiest approach right now is Deep Learning. See http://deeplearning.net/tutorial/, but it's quite a lot to learn if you only know about normal neural networks. For an easier way I\r\n would read up on Auto Encoders, Denoising Auto Encoders and Stacked Auto Encoders (basically do the above tutorial, just skip everything on Restricted Boltzmann Machines and Deep Belief Networks). You can get great results with just these techniques and they\r\n are easy to understand and implement. You also don't have to follow their python implementation, just reading through the theory parts should prepare you for doing it yourself. I just started and am at 94.6% with lots of room to optimize.
", "votes": 9 }, { "id": 65775, "postDate": "2015-03-09T13:29:43.090Z", "content": "I got 96% with this code, no feature engineering and lazy parameter tuning. So I think that it could do a lot better.
\nI used this competition to try and really understand how basic neural networks work so I wrote it using only numpy and in a way that should make it very readable and understandable. Sort of a hybrid between notes and function. I wrote two blog posts here and here that might be useful for people starting out. Hopefully all the information in there is accurate ;-). I'm still new to neural nets as well so I would appreciate some feedback from all the smart people here.
", "votes": 3 }, { "id": 65277, "postDate": "2015-03-02T18:44:38.613Z", "content": "I got 0.98886 on the test data set using a 8000 node, 3 layer neural net and some of the image techniques talked about in this thread. I explain my solution thoroughly here w/ code included: http://nicklocascio.com/neural-net-mnist-kaggle/.
", "votes": 3 }, { "id": 39515, "postDate": "2014-02-26T14:13:53.563Z", "content": "A dbn that will give a bit over 99% on the public ldb
\nimport numpy as np
import pandas as pd
from nolearn.dbn import DBN
from scipy.ndimage import convolve
import scipy.ndimage as nd
###############################################################################
# Setting updef nudge_dataset(X, Y):
"""
This produces a dataset 5 times bigger than the original one,
by moving the 8x8 images in X around by 1px to left, right, down, up
"""
direction_vectors = [
[[0, 1, 0],
[0, 0, 0],
[0, 0, 0]],[[0, 0, 0],
[1, 0, 0],
[0, 0, 0]],[[0, 0, 0],
[0, 0, 1],
[0, 0, 0]],[[0, 0, 0],
[0, 0, 0],
[0, 1, 0]]]shift = lambda x, w: convolve(x.reshape((28, 28)), mode='constant',
weights=w).ravel()
X = np.concatenate([X] +
[np.apply_along_axis(shift, 1, X, vector)
for vector in direction_vectors])
Y = np.concatenate([Y for _ in range(5)], axis=0)
return X, Ydef rotate_dataset(X,Y):
XX = np.zeros(X_train.shape)
for index in range(X.shape[0]):
angle = np.random.randint(-7,7)
XX[index,:] = nd.rotate(np.reshape(X[index,:],((28,28))),angle,reshape=False).ravel()
X = np.vstack((X,XX))
Y = np.hstack((Y,Y))
return X, Y# Load Data
mnist = pd.read_csv("train.csv")
y_train = mnist['label'].values
X_train = mnist.loc[:,'pixel0':].values
X_test = pd.read_csv("test.csv").valuesX_train = np.asarray(X_train / 255.0, 'float32')
X_test = np.asarray(X_test / 255.0, 'float32')
X_train, y_train = nudge_dataset(X_train, y_train)
X_train, y_train = rotate_dataset(X_train, y_train)clf = DBN(
[X_train.shape[1], 350, 10],
learn_rates=0.3,
learn_rate_decays=0.95,
learn_rates_pretrain=0.005,
epochs=120,
verbose=1,
)clf.fit(X_train, y_train)subm = pd.read_csv("knn_benchmark.csv")
subm.Label = clf.predict(X_test)
subm.to_csv("result.csv", index_label='ImageId', col=['Label'], index=False)
Woot! My very first hand-coded perceptron network scored 88% - same as LeCun reports for 1-layer without preprocessing. Not an impressive score, perhaps, but nice to stride in the footsteps of the masters. Hand-coded allowed me to parallelize the classification ensemble (one per digit).
\nNext ... trying my hand at "Convolutional net LeNet-1 subsampling to 16x16 pixels"?
", "votes": 2 }, { "id": 25117, "postDate": "2013-06-04T01:55:46.597Z", "content": "@Yogesh, Are you using a separate holdout validation set and watching the error rate on that? If you're just running your network for set amounts of iterations you're kind of shooting in the dark for what your accuracy will be. I just ran mine and it worked\r\n quite well (60th place at 0.986) so it looks like it carries over pretty well to Kaggle's data. This is using dropout and L2 hidden unit weight vector constraining which is good for about 1% more on MNIST so standard nets should still be able to get 0.97-0.975.\r\n All the hyper parameters are identical to what's used in the dropout paper I linked earlier.
\r\n@Eugene, what form of dimensionality reduction are you doing? Are you just rescaling the input down to 14x14? That kind of dimensionality reduction almost always hurts accuracy in the long run (may be made up in the short term for faster computation speeds)\r\n since your loosing information the network could use. You could try a technique like PCA dimensionality reduction and training a network on that (I think the general rule of thumb is use as many dimensions as necessary to keep 99% of variance).
\r\n
\r\nWhat is the runtime like for your NN? For reference, I'm using 2 hidden layers with 800 hidden units each and it takes about 20 minutes to get that 0.986, it's actually about the same speed as Scikit-Learns Random Forest (it get's to an accuracy of 0.967 in\r\n about 2 minutes). I am using a GPU accelerated version built on top of gnumpy though, which'll be about 10x faster than running it on the CPU.
\r\n
\r\nIf I understand your question correctly you're wondering about how different algorithms process the input to get their predictions and why there are such significant differences. In general things split into "low" and "high" dimensional algorithms. Things like\r\n SVM, NNs, and RF are good for high dimensional data where as lower dimensional data (less than 100 inputs) are typically the domain of linear models, I think. Specifically with Random Forest versus Neural Networks, (I don't know much about Random Forest to\r\n be honest) I think Random Forest relies on training a bunch of weak learners on random subsets of the data and features (via bagging and boosting I think) and combining them together into a larger model. Essentially it breaks the larger problem down into a\r\n bunch of smaller problems and tries to solve those and then get a coherent answer out of them. Traditional Neural Networks, are a method of automatically finding features in the input that end up being good predictors/discriminators of the output. For this\r\n digit task it'll learn features (a hidden unit is a feature detector to some degree) like the vertical bar of a 7 or the middle horiztonal part of a 3 and the output layer will learn how to weight those features such that it's more likely to predict a 7 when\r\n that hidden unit for the vertical bar of the 7 has a high activation for instance. This is all done via backpropagating the errors it makes on the networks and correcting the weights so it's less likely to make those errors. I started by reading the user guide/overview\r\n documentation on Scikit-Learn's website and when I was interested in specific models I looked for Coursera's or Udacity courses that covered them.
Is the output using softmax/cross entropy to explicitly force the NN to learn a probability distribution over the classes? Are you using momentum and minibatch learning? If not, those would be good things to add. Bumping up the hidden units a bit more and\r\n adding a second hidden layer may help as well but with the current speed of your network your going to start getting into 24 hour + training times...
\r\nDid you write the neural net yourself or are you using a library? What langauge? You can definitely get faster than 16 hours for 500 HUs.
\r\nFor overfitting you could look into implementing a recent technique like dropout. http://arxiv.org/pdf/1207.0580.pdf
", "votes": 3 }, { "id": 24788, "postDate": "2013-05-25T02:34:25.013Z", "content": "[quote=Yogesh Bhalerao;24766]
\r\nThanks Alec, I am now running with 250 hidden units and 1000 iterations. But will now work on understanding the accuracy with each iterations.
\r\nSince I am new to ML and NN, I am going to have to first understand what softmax/cross entrophy means (Need to find time to study Optimization Theory). My approach is the same as described in the NN topic of the Standford ML course (Andrew Ng) i.e., backpropagation+regularization+Conjugate\r\n Gradient
\r\n[/quote]
\r\nSoftmax/cross entropy isn't too bad. It's is a way of forcing the outputs of the NN to explicitly model a probability distribution over the classes/output units. Here's a video explaining it https://class.coursera.org/neuralnets-2012-001/lecture/47.\r\n (That's the video I learned it from and the wikipedia page). The error derivatives are real simple too so it's not hard to tack on to a more standard network architecture.
\r\nI haven't looked at the Stanford ML course's NN lectures yet, but they might have set it up that way without making it explicit that that was what they were doing.
", "votes": 3 }, { "id": 21384, "postDate": "2013-03-18T13:29:16.037Z", "content": "I am curious if anyone is using neural networks and what kind of results they are getting. I tried 3 layer NN with 150 neurons on the hidden layer, it ran about 1 hr and gave me around 87.52%. If anyone tried NN, i would appreciate any help on how to go\r\n about it.
", "votes": 3 }, { "id": 62636, "postDate": "2015-01-12T08:34:41.157Z", "content": "I tried this and it works great. However, the rotate function contains a loop that will exponentially increase the size of the Y array (i.e. it doubles it's size every loop and it goes for 42000 loops! so doing the math that is 2^42000 = not in this lifetime).
\nReplace the rotate_dataset function below with this alternative which actually works and note it only requires the X array because it's not rotating anything in Y.
\ndef rotate_dataset(X):
XX = np.zeros(X.shape)
for index in range(X.shape[0]):
angle = np.random.randint(-7,7)
XX[index,:] = nd.rotate(np.reshape(X[index,:],((28,28))),angle,reshape=False).ravel()
return XX
[quote=Rafael;39515]
\nA dbn that will give a bit over 99% on the public ldb
\nimport numpy as np
import pandas as pd
from nolearn.dbn import DBN
from scipy.ndimage import convolve
import scipy.ndimage as nd
###############################################################################
# Setting up
def nudge_dataset(X, Y):
"""
This produces a dataset 5 times bigger than the original one,
by moving the 8x8 images in X around by 1px to left, right, down, up
"""
direction_vectors = [
[[0, 1, 0],
[0, 0, 0],
[0, 0, 0]],[[0, 0, 0],
[1, 0, 0],
[0, 0, 0]],[[0, 0, 0],
[0, 0, 1],
[0, 0, 0]],[[0, 0, 0],
[0, 0, 0],
[0, 1, 0]]]
shift = lambda x, w: convolve(x.reshape((28, 28)), mode='constant',
weights=w).ravel()
X = np.concatenate([X] +
[np.apply_along_axis(shift, 1, X, vector)
for vector in direction_vectors])
Y = np.concatenate([Y for _ in range(5)], axis=0)
return X, Y
def rotate_dataset(X,Y):
XX = np.zeros(X_train.shape)
for index in range(X.shape[0]):
angle = np.random.randint(-7,7)
XX[index,:] = nd.rotate(np.reshape(X[index,:],((28,28))),angle,reshape=False).ravel()
X = np.vstack((X,XX))
Y = np.hstack((Y,Y))
return X, Y
# Load Data
mnist = pd.read_csv("train.csv")
y_train = mnist['label'].values
X_train = mnist.loc[:,'pixel0':].values
X_test = pd.read_csv("test.csv").valuesX_train = np.asarray(X_train / 255.0, 'float32')
X_test = np.asarray(X_test / 255.0, 'float32')
X_train, y_train = nudge_dataset(X_train, y_train)
X_train, y_train = rotate_dataset(X_train, y_train)clf = DBN(
[X_train.shape[1], 350, 10],
learn_rates=0.3,
learn_rate_decays=0.95,
learn_rates_pretrain=0.005,
epochs=120,
verbose=1,
)clf.fit(X_train, y_train)subm = pd.read_csv("knn_benchmark.csv")
subm.Label = clf.predict(X_test)
subm.to_csv("result.csv", index_label='ImageId', col=['Label'], index=False)
[/quote]
", "votes": 1 }, { "id": 24844, "postDate": "2013-05-27T15:31:40.067Z", "content": "[quote=KL Tah;24842]
\r\npeople here are posting things like "my accuracy on the test data set is 92%" but how are you actually able to know the accuracy of the test set without already knowing the correct answers? just curious...
\r\n[/quote]
\r\nSubmit it =)
", "votes": 2 }, { "id": 40024, "postDate": "2014-03-08T18:06:05.767Z", "content": "Has anyone plotted learning curves to see if the training and cross validation set are converging, and to get an idea of whether the problem is bias or variance?
", "votes": 1 }, { "id": 28525, "postDate": "2013-08-10T16:49:41.883Z", "content": "This was my experience with neural nets. I also took Ng's class but I made a more general implementation of the algorithm in python using numpy (not a fan of theano).
\nI was able to get about 95% accuracy training for a few minutes using mini batches of size 50.
\nMinibatches are the key for training the nets, doing full batch is crazy.
\nI don't have a GPU so only CPU training.
\nThen I read a paper of Hinton and he mentions that for MNIST the nets often are trained for days or even weeks. So I let my net training for a full night and I was able to get 97-98% accuracy.
\nAlso if you go to Lecun's MNIST page (http://yann.lecun.com/exdb/mnist/) you can see benchmarks for different architectures. If you are using only backprop I recommend using only one hidden layer. I used 300 hidden units.
\nSo my advice is: use mini-batches to overcome your memory issues and give more time to the nets.
\nAnd another tip: save often the weights of the net, that way you can start from that point. I saved every 10k iterations or so. So in case you ran out of memory of whatever you didn't lose your time.
\nGood luck!
", "votes": 1 }, { "id": 28472, "postDate": "2013-08-09T05:22:12.210Z", "content": "yeah, Tim, using the original MNIST data is like being given the answers ahead of time. Don't do that
", "votes": 1 }, { "id": 28422, "postDate": "2013-08-08T09:29:25.760Z", "content": "I just figured out that there is a very big difference in that extra data from the original MNIST set: So if you use the original MNIST data your test error will be much lower. The best I achieved with the kaggle data was 1.9 % test error with an ensemble of 30 neural networks. If I use an ensemble of 10 neural nets that were trained on the original MNIST set then I go down to 0.5 % test error on the kaggle test set (a single net has about 1.6% to 1.7% test error on the original MNIST set) – and this is not even fully optimized yet!
\nI found dropout to be very helpful as regularization. Also sparse initialization improves training speed and it seems to help a bit with generalization. I found momentum and also Nesterov momentum not very helpful — I guess these will be more helpful in training deep neural networks or on other data sets. Oddly enough I found 8000 hidden units much better than 800 hidden units; together with dropout the network does not overfit and generalizes well. But it takes a lot of time to train such a net without a GPU: To train 8000 hidden units to 1.2 % cross validation error takes a bit more than 10 minutes on my theano implementation – a good numpy implementation of 8000 hidden units should not take longer than 15 hours to train.
", "votes": 1 }, { "id": 25762, "postDate": "2013-06-16T19:19:54.283Z", "content": "I am also a newbie (well a relative newbie as I last coded ML algorithms in grad school), and also taking Andrew Ng's Coursera ML class. I used a modified version of NN from one of the class exercises (I added normalization and PCA). With 350 HUs (1 hidden\r\n layer) and 1k iterations I was able to get 100% on my train set, 98% on my test set, and 98.31% on the Kaggle test set upon submitting the answer.
\r\nThis took several hours to run (though PCA certainly improved it) as I was using Octave.
\r\nThank you everyone for the useful links. I liked the various docs at http://ufldl.stanford.edu/wiki/index.php/UFLDL_Tutorial as well.\r\n
\r\n", "votes": 1 }, { "id": 25641, "postDate": "2013-06-14T19:40:47.917Z", "content": "@TravelingDesi, reading through the last few pages should give you a good overview of various methods to try. Also there's quite a few different ways to do regularization of Neural Networks, some of which are mentioned in the last few pages as well.
\r\nWeirdly enough, everyone who seems to be running Ng's networks seems to be getting scores that are very similar to what you can get with Logistic Regression ~ 92.5% http://deeplearning.net/tutorial/logreg.html#logreg
", "votes": 1 }, { "id": 24846, "postDate": "2013-05-27T16:49:50.227Z", "content": "[quote=KL Tah;24842]
\r\npeople here are posting things like "my accuracy on the test data set is 92%" but how are you actually able to know the accuracy of the test set without already knowing the correct answers? just curious...
\r\n[/quote]
\r\n\r\nMNIST (the dataset this competition is based off of) is publicly available (train and test) and a great test dataset for various ML algorithms so that's one way of knowing it. The other is to just split out part of the train set to use as a validation set\r\n to test against. Scores won't carry over directly to the Kaggle test set, but they should be in the ballpark.
", "votes": 1 }, { "id": 24767, "postDate": "2013-05-24T22:25:37.693Z", "content": "[quote=Yogesh Bhalerao;24746]
\r\nI was able to get 85% accuracy on the score, using a single hidden layer with 25 activation units, regularization and conjugate gradient descent for 50 iterations. Any suggestions on what I should be trying next to improve my accuracy? Note: I am new to\r\n machine learning and NN
\r\n[/quote]
\r\nI've got 90% accuracy with reduced dimensionality (thanks vivekn for the idea: instead of 784 input I've got only 176) and my NN had just 18 neurons in hidden layer.
\r\nNow I'm going to plot learning curves and see if it's giving anything useful. With 176 input values it's much simpler to debug my NN.
\r\nYogesh, I'd say plotting these guys is more helpful than trying something different (or learning new stuff). I believe it's worth to understand some sort of analysis or debugging techniques before going further. That actually was exactly what Andrew Ng suggested\r\n =)
", "votes": 1 }, { "id": 24763, "postDate": "2013-05-24T17:15:04.797Z", "content": "Increasing the number of hidden units is a good step. Most papers with MNIST like a minimum of 500 hidden units, per layer and some go up to 2000. Also consider plotting accuracy as a function of iterations, you may simply need to run it longer to get more\r\n accurate.
\r\nAre you using softmax/cross entropy output? That usually helps get a little better too.
", "votes": 1 }, { "id": 24519, "postDate": "2013-05-20T13:15:04.213Z", "content": "[quote=vivekn;24510]
\r\n\r\nI divided the image into 2x2 cells, for each 2x2 cell I took the maximum of the four intensities and used it as the intensity for the output image. I think a more better way of downsampling would be to apply a Gaussian blur and then take one of the four\r\n pixels. But this was faster and easy to implement.
\r\n[/quote]
\r\nVivekn, that's worth to try. Thanks a lot for the idea! I believe such a dimensionality reduction is possible here, because we're working we pictures, but have you tried any other algorithms to reduce dimensionality that could potentially be applied to every\r\n dataset?
\r\nDo you think it's useful if I turn pictures from grayscale to black-white?. Would it improve computing speed or simplify model? How that would influence overall performance of algorithm? By algorithm I mean anything available, not only NN.
", "votes": 1 }, { "id": 24510, "postDate": "2013-05-20T07:57:46.523Z", "content": "[quote=fathersson;24501]
\r\n[quote=vivekn;24489]
\r\nI downsampled the image to 14 x 14 and then applied some basic thresholding and then fed it to the network.
\r\n[/quote]
\r\nDo I get it correct that you basically get center of each image of 28x28 size and fed it to the network?
\r\n[/quote]
\r\n\r\nI divided the image into 2x2 cells, for each 2x2 cell I took the maximum of the four intensities and used it as the intensity for the output image. I think a more better way of downsampling would be to apply a Gaussian blur and then take one of the four\r\n pixels. But this was faster and easy to implement.
\r\n", "votes": 1 }, { "id": 21614, "postDate": "2013-03-25T07:50:59.310Z", "content": "With bit of heuristics and debugging i am getting more than 98.5% with simple ANN. Now i am curious what convolutional NN would do.
", "votes": 1 }, { "id": 21415, "postDate": "2013-03-19T13:57:44.347Z", "content": "I tried to increase my neurons to 500 in the hidden layer to see the results. But its taking forever to complete this. It has already taken about 11 hrs and its still going on.
", "votes": 1 }, { "id": 98353, "postDate": "2015-11-06T18:28:46.597Z", "content": "Here is a notebook I wrote with example python code for a neural network and a convolutional nn. I get >0.96 accuracy in 500 seconds for 1000 nodes in the hidden layer in 15 epochs. Let me know if you have any questions.
", "rawMarkdown": "Here is a [notebook I wrote with example python][1] code for a neural network and a convolutional nn. I get >0.96 accuracy in 500 seconds for 1000 nodes in the hidden layer in 15 epochs. Let me know if you have any questions.\r\n\r\n\r\n [1]: https://www.kaggle.com/kobakhit/digit-recognizer/digit-recognizer-in-python-using-cnn" }, { "id": 81873, "postDate": "2015-06-14T19:16:16.477Z", "content": "%Implementing back propagation on NN with 401 units in input, 25 in hidden and 10 output
\nd1= zeros(size(Theta1));
d2= zeros(size(Theta2));
ynew=zeros(num_labels,m);
help1=eye(num_labels);
for i=1:m
ynew(:,i)=help1(:,y(i,1));
end
for t=1:m
na1=X(t,:);
nz2=Theta1*na1';
na2=sigmoid(nz2);
na2=[1;na2];
nz3=Theta2*(na2);
na3=sigmoid(nz3);
delta3=na3-ynew(:,t);
delta2=(Theta2)' * (delta3 ) ;
delta2=delta2(2:end,:);
delta2=delta2.*sigmoidGradient(nz2);
d1=d1+delta2* na1;
d2=d2+delta3* na2';
Theta1_grad=Theta1_grad+(1/m)*d1;
Theta2_grad=Theta2_grad+(1/m)*d2;
end
Theta1_temp = [zeros(size(Theta1,1),1) Theta1(:,2:end)];
Theta2_temp = [zeros(size(Theta2,1),1) Theta2(:,2:end)];
Theta1_grad = Theta1_grad+ (lambda/m )* Theta1_temp;
Theta2_grad = Theta2_grad + (lambda/m) * Theta2_temp ;
what is wrong with the above implementation of Back propagation? I am new to ML and stuck at BP, getting accuracy of just 10% on my original training set ..?Any help
" }, { "id": 72791, "postDate": "2015-04-21T01:20:25.863Z", "content": "Mostly as a learning exercise for myself, I have written a Python neural network library that wraps some simple C code using the CBLAS library for performance. I've gotten it to be pretty fast - I was able to score 97.9% on the test set training a neural net with 1 hidden layer of 400 nodes for just 5 minutes (30 iterations over the training set) on my Core i5 MacBook Pro. Anyway, I thought other people might find it interesting or useful and I would love to get feedback on it. You can get the code here: https://github.com/tburmeister/pyneural
\nThe library is still very much a work in progress, and in particular I don't have a good install script yet, but it shouldn't be too hard to get running. Once it's installed, it should be pretty simple to use.
\nEdit: update for those interested - I'm now up to over 99% accuracy on the test set, with a training time of roughly 30 minutes. I have added an IPython notebook with my training code to my git repo.
" }, { "id": 65281, "postDate": "2015-03-02T19:30:22.497Z", "content": "@Nick -
\nHow does your score scale with node size?
\nI got 0.98871 with hidden_layer_sizes = (1568,200)
\n(But that includes adding some synthetic training examples.)
\nEDIT: Just saw you had the synthetic data too. Thanks for the blog writeup. Very nice.
" }, { "id": 64096, "postDate": "2015-02-12T07:01:26.397Z", "content": "I used a simple two-layer neural network with 500 hidden units and conjugate gradient with scipy.minimize. It took me 13 hours to get the result and i got 100% accuracy on the train set and 95.8% on the test set. Afterwards, I normalized the train data (by dividing each element by 255 and subtracting row mean), and I also add regularization. Now I got 98.3% on the test set, which is really amazing. I would like to try convolutional neural network next.
\nby the way, has anybody found the MNIST dataset in Kaggle is different from the MNIST dataset in http://yann.lecun.com/exdb/mnist/? I used the trained weight to predict the MNIST dataset in yann website, which gives 9% accuracy. Then I opened both csv files and found that kaggle seems to have done some modifications to the dataset, but i am not sure.
" }, { "id": 62775, "postDate": "2015-01-16T03:29:37.327Z", "content": "[quote=@$#!$#;21384]
\nI am curious if anyone is using neural networks and what kind of results they are getting. I tried 3 layer NN with 150 neurons on the hidden layer, it ran about 1 hr and gave me around 87.52%. If anyone tried NN, i would appreciate any help on how to go about it.
\n[/quote]
\nI'm not participating in this, but thought you might be interested in looking at this:
\nSimon, W. E., and J. R. Carter, “Generalizing From a Small Set of Training Exemplars for Handwritten Digit Recognition,” Proceedings of Applications of Artificial Neural Networks II, SPIE, 1991
" }, { "id": 59433, "postDate": "2014-12-03T02:25:29.077Z", "content": "[quote=Alexei Vasilkov;59429]
\n[quote=inversion;59428]
\nJust started playing, python code ported from matlab (from Dr. Ng's class), plus using the nudge/rotate script found in this thread.
\n1-hidden layer, size = 200, lambda=5, 500 epochs, LB: 0.98629
\n[/quote]
\nHave you ported optimization function as well and trained it without setting learning rate?
\nAlso how long did it take to train for 500 epochs?
\n[/quote]
\nI'm using the scipy.minimize function (with CG). My learning rate was very small. Run time: 187 minutes.
" }, { "id": 59429, "postDate": "2014-12-03T01:26:46.977Z", "content": "[quote=inversion;59428]
\nJust started playing, python code ported from matlab (from Dr. Ng's class), plus using the nudge/rotate script found in this thread.
\n1-hidden layer, size = 200, lambda=5, 500 epochs, LB: 0.98629
\n[/quote]
\nHave you ported optimization function as well and trained it without setting learning rate?
\nAlso how long did it take to train for 500 epochs?
" }, { "id": 59428, "postDate": "2014-12-03T00:17:20.430Z", "content": "Just started playing, python code ported from matlab (from Dr. Ng's class), plus using the nudge/rotate script found in this thread.
\n1-hidden layer, size = 200, lambda=5, 500 epochs, LB: 0.98629
" }, { "id": 59014, "postDate": "2014-11-26T16:27:22.423Z", "content": "99.2% using a custom theano 2 layer convolutional neural net (plus 1 hidden layer) with max pooling, dropout, and momentum.
" }, { "id": 58828, "postDate": "2014-11-24T23:07:51.853Z", "content": "[quote=hden;58509]
\nSimple 2-layer convolutional neuron network using node.js give you about 0.97286.
\n[/quote]
\nWow.
\nI use simple input-hidden-output network changing learning rate by hand, lol. So far, the best result is 0.94243 with 300 units in one hidden layer. I used gradient descent without softmax, momentum or whatever else.(Don't have time now to upgrade my code with those, but have time to run it again and again and try different hidden layer size, stupid me) So can anybody say what is max result I can achieve using this kind of a beginner network? Also I have a question.. I've tried to make two hidden layers but error rate is not improving and in the end learning becomes very slow and don't go behind 0.9. Is it normal behavior? I guess error gets pretty small after descending from first hidden layer to input, is it right? It's just that I'm not sure in my algorithm. It works but what if it has some bugs and it holds it from being even better.
" }, { "id": 58509, "postDate": "2014-11-20T10:18:56.540Z", "content": "Simple 2-layer convolutional neuron network using node.js give you about 0.97286.
" }, { "id": 58285, "postDate": "2014-11-18T07:27:13.080Z", "content": "Hi
\nDoes any one know how to work with the MNIST dataset on H20? (0xdata), using Conv Neural Nets. I am seeing bits and pieces on the internet where it seems it can be done, but not any working code.
\nAlso, does any one have experience with GraphLab http://graphlab.com/products/create/docs/graphlab.toolkits.deeplearning.html#example-1-digit-recognition-on-mnist-data
\nRegards
" }, { "id": 56886, "postDate": "2014-10-28T06:16:42.427Z", "content": "96.8% straight out of the box with @hexadata h20. Impressed with tool. 3 hidden layers 200, 100, 200, 10 fold cross validation on training set. I did very little, just testing h20. 1 hour to train on Mac Air. 2 hours wall clock time.
\nTo generate submission from h20 export, only "programming" I did:
\ncreate header, cut out 1st column, remove quotes, add line #'s
cat header > submission
cut -f1 -d, h2oexport | tr -d '"' | nl -v0 -nrn -s, | tr -d ' ' | tail -28000 >> submission
Hi guys,
I read some procedures your have done, but it would be very different if applying SOM? I could also set it up with a few hidden layers?
I am beginner in Neural Network and Machine leraning.
With 25 hidden nodes, I could take 93.5% (on cross-validation). I had to play with the learning rate and iteration number though.
" }, { "id": 30018, "postDate": "2013-09-02T21:42:33.367Z", "content": "I followed the following paper, with a few small modification, and I got 98.9%.
\n\n
arxiv.org/pdf/1204.3968v1.pdf
\nFor those not interested in follow the link, it uses a convolutional neural network with LP pooling (not max).
" }, { "id": 29233, "postDate": "2013-08-22T13:15:52.783Z", "content": "I tried pretraining an Autoencoder with 400 units and then training random forests on the transformed data (400-dimensional). Got me up to .9620. I'm now fiddling around with learning rates to see if I can get better representations with the AE's.
\nGood luck!
" }, { "id": 28591, "postDate": "2013-08-12T15:55:32.317Z", "content": "Also, using a "committee" or multiple mlps helps a lot. using a committee of 3 deep convolutional networks improved my score by a little over .1%
" }, { "id": 28540, "postDate": "2013-08-11T03:48:07.087Z", "content": "Daniel, if you're just using a simple MLP, try 2 layers with 400-800 HUs each. results are best if the # of HUs per layer is reasonably constant. Try rectified linear units as your activation instead of a hyperbolic tangent. Dropout or dropconnect help against overfitting (much more so than L1 or L2 regularization) when you have a model with high complexity
" }, { "id": 28532, "postDate": "2013-08-10T21:09:10.483Z", "content": "I tried one HL and then went crazy and tried like 10 HL (with less hidden units, like 30 or so) and I found difficulties training the deep net using only gradient descent; doing some research I learned that on those cases pre-training before gradient decent is the solution but is a more complex approach than vanilla MLP. Maybe I just need to try 2 HL like some benchmarks on Lecun's page.
\nI think the problem is that I am using mini-batches and the advance optimization methods (from scipy) do not work as expected. I believe that with a simpler data set the optimization methods using full batch on a deep network will work as expected. An approach could be using larger batches maybe that way the advance optimizations give better results but I still need to test that.
" }, { "id": 28531, "postDate": "2013-08-10T20:48:18.793Z", "content": "Alessandro,
\nI am sorry I did not reply earlier - was on vacation and then busy with work. Anyway.. My cost function is the same as the one we used in the class (ie no modifications). I did post some more details on what I changed compared to code in the class on my blog at http://analyticsthoughts.blogspot.com/2013/08/from-machine-learning-coursera-to.html.
\nAlec, Jared, Daniel, Tim, Mihailo all provide good advice and data. Personally, I would have tried mini-batch as well, combined with larger number of HUs - but unfortunately I need to move on to other stuff.
\n" }, { "id": 28528, "postDate": "2013-08-10T17:48:21.227Z", "content": "
[quote=Daniel Rodriguez;28525]
\nThis was my experience with neural nets. I also took Ng's class but I made a more general implementation of the algorithm in python using numpy (not a fan of theano).
\nI was able to get about 95% accuracy training for a few minutes using mini batches of size 50.
\nMinibatches are the key for training the nets, doing full batch is crazy.
\nI don't have a GPU so only CPU training.
\nThen I read a paper of Hinton and he mentions that for MNIST the nets often are trained for days or even weeks. So I let my net training for a full night and I was able to get 97-98% accuracy.
\nAlso if you go to Lecun's MNIST page (http://yann.lecun.com/exdb/mnist/) you can see benchmarks for different architectures. If you are using only backprop I recommend using only one hidden layer. I used 300 hidden units.
\nSo my advice is: use mini-batches to overcome your memory issues and give more time to the nets.
\nAnd another tip: save often the weights of the net, that way you can start from that point. I saved every 10k iterations or so. So in case you ran out of memory of whatever you didn't lose your time.
\nGood luck!
\n[/quote]
\n\n
using multiple layers helps over just one if your model is a vanilla MLP.
" }, { "id": 28477, "postDate": "2013-08-09T06:56:16.347Z", "content": "[quote=Mihailo;27446]
\nHi Alessandro,
\nHave you tried mini-batch approach?
\nAlso, I think that implementation from Prof. Hinton's class on coursera transposes input matrix (I'm not absolutely sure about this, I'd need to check). Might be an issue with Octave that it better supports lots of columns. Prof Ng's data set was smaller so maybe that's why it wasn't an issue at the time. I vaguely remember trying this set with both (albeit modified) examples and that I had to do transpose of the input matrix. That was a good few months now so don't take my word for it.
\n[/quote]
\nno I did not.. I'm really kind of stuck with this!
\nUnfortunately I didn't have time last year to follow Hilton's class (I did Ng's instead)..still hoping for a re-run of that, even though I could watch the video by myself, being a class is such much better!
" }, { "id": 28474, "postDate": "2013-08-09T05:35:32.590Z", "content": "Well, that's embarrassing :( – at least I now know there is something wrong with my net.
" }, { "id": 28473, "postDate": "2013-08-09T05:23:50.127Z", "content": "I'm using a neural network with 2 convolutional layers followed by a dense layer of 200 units with dropconnect. i didn't spend much time trying to optimize the hyperparameters
" }, { "id": 28460, "postDate": "2013-08-08T21:14:10.873Z", "content": "@Tim, I think using the original MNIST data would be considered "something fishy" for this competition. You are supposed to use the provided Kaggle data, otherwise you are likely training your networks on some of the kaggle "test" digits.
Also I've found GPU vs. CPU comparisons are non-linear for the amount of hidden units. The more complex the network, the faster a GPU is compared to the CPU as normally as GPUs tend to be limited by other factors than raw flops (such as a bandwidth) when you're training smaller networks. CPUs slow to a crawl doing dot products and the like on enormous matrices.
8000 hidden units on a regular CPU is probably significantly more than 15 hours.
Hi Alessandro,
\nHave you tried mini-batch approach?
\nAlso, I think that implementation from Prof. Hinton's class on coursera transposes input matrix (I'm not absolutely sure about this, I'd need to check). Might be an issue with Octave that it better supports lots of columns. Prof Ng's data set was smaller so maybe that's why it wasn't an issue at the time. I vaguely remember trying this set with both (albeit modified) examples and that I had to do transpose of the input matrix. That was a good few months now so don't take my word for it.
" }, { "id": 27325, "postDate": "2013-07-17T05:02:35.690Z", "content": "unfortunately octave keep crashing and it's really get annoing.. getting go further than 30k examples after a quite not really conservative PCA reduction :|
\r\nany chance we can see your implementation for the cost function? I've tried clearing, saving to disk and loading (binary files are quite quick) but still.. no luck -.-
" }, { "id": 27261, "postDate": "2013-07-16T03:35:01.927Z", "content": "\r\nAlessandro,
\r\n\r\nI was running this on a laptop, and frequently ran into similar problems. My solution was to
\r\n(1) aggressively clean up variables (and more specifically, large matrices) (that is call "clear" from your Octave code wherever you can),
\r\n(2) not run other programs at the time, and
\r\n(3) use PCA (this also helped with performance)
\r\n\r\nEven then, I ran into some limits on training set size and NN size.
\r\n\r\nMy system specs are: Lenovo T430 with Intel Core i5-3320M Processor, 2.60 GHz, with 8GB RAM, running 64-bit Windows 7 and Octave 3.2.4. With that, I could run with a training set of 49,000 rows x 784 pixels (I\r\n also generated my own data - hence the large training set). Or to clarify, I only used the data posted for this competition on Kaggle (plus what I generated), and not the entire MNIST data set.
\r\n\r\nI also ran into similar errors with the number of NN nodes (I was only able to go up to 400 or so nodes with a single layer).
\r\n\r\nSo in summary, I suggest you try to reduce the training set and size of the NN first just to see if you can make it work. Then, you can try to increase the sizes to see where the limit is for your system. Hope this helps.
\r\n\r\nIf you are still stuck, let me know and I can share the code (though it's not pretty). I don't want to post it here as 95% of it is from the class.
\r\n\r\n" }, { "id": 27225, "postDate": "2013-07-15T11:07:14.403Z", "content": "hey GundarsK - I was trying to use the ex4 from the NG class too but I always got the following error while calculating the cost
\r\n"error: memory exhausted or requested size too large for range of Octave's index type -- trying to return to prompt"
\r\nhow did you get around this?
" }, { "id": 25646, "postDate": "2013-06-14T20:12:35.897Z" }, { "id": 25609, "postDate": "2013-06-14T13:00:23.207Z", "content": "I am Prof Ng student in coursera as well. I used the Neural Network we studied in the class and tweaked it slightly (784 input NEurons, 100 hidden NEurons and 10 output). I also trained for 100 iterations.
\r\n\r\nNeedless to say the training was running all night and I got a 92.5% accuracy on my test set. Thats great for my first attempt, but now I dont know what to change the NN to make it better (besides the obvious changing # of hidden layers, # of Neurons etc\r\n etc)
\r\n\r\nInterestingly I also ran regularization (over many values of lambda) and selecting the one that gave the most accuracy on my cross validation set. The smallest lambda (least regularization) was not the most accurate on cross validation.
\r\n\r\nThanks for any pointers. I will go through this thread to find out what others are doing as well.
" }, { "id": 25139, "postDate": "2013-06-04T21:32:58.713Z", "content": "When you make your submission do you train the model on on all the data or are you still only using the 60% training? That's eating into a lot of your possible data if you're not. Use the test/validation to figure out your parameters and then add a bit more\r\n on to the training time to account for the more data (there's fancier ways to estimate when to stop using log-likelihood stuff I think but I haven't played around with that). Your using conjugate gradient right? Have you tried using just simple gradient descent\r\n with momentum?
\r\n
\r\nYeah, Octave is a great protyping tool but not good for production. Originally I used python and numpy that was fully vectorized (forward pass and backprop on the full mini-batch as dot products/matrix multiplies) which is about 7.5-15x slower (around 500-1000\r\n digits per second / 10 mini-batches), then I ported it to run on the GPU (I have a GTX 660 TI to run it on which is about $250-300 but I'm a gamer so I justified it that way :P ) using\r\nTijmen Tieleman's gnumpy, http://www.cs.toronto.edu/~tijmen/gnumpy.html which is good for about 7.5K digits per second.
@Alec: I split my training set into three data sets. 60% Training, 20% Cross-validation and 20% Test. Also what tool are you using? I am using Octave 3.2 and it is super slow.
" }, { "id": 25107, "postDate": "2013-06-03T22:04:15.350Z", "content": "I hope someone could help me a little.
\r\nI've been trying to reduce dimensionality (from 28x28 to 14x14) and then train my NN with two hidden layers with 120 neurons in each! I even assume that my NN could overfit such an input data. And I've got only 92.7% - not really much I believe, especially\r\n with complex NN. It's kind of hard to continue - I'm not sure about where to go exactly, so I chose to try some other algorithms.
\r\nAnd I've tried random forest. With n = 1000 it immediately gives much better result - 96.7%
\r\nThe question here is tricky. I assume that given input should have some sort of properties that such an input data could be processed by different learning methods with different performance. For example random forest on such an input gives better result\r\n with less time. There are may be some other interesting details. Like if we have particular features or properties in input data set, then it's probably better to utilize particular precompute and then particular methods.
\r\nCould you, please, point me out on such a sources (books, articles, smth), where I can get some description on my question. I'm not sure they even exists, but still...
" }, { "id": 25106, "postDate": "2013-06-03T21:43:21.313Z", "content": "Hmm... The error rates I've been quoting are on full MNIST, which is 60k examples, so they won't compare directly but that still seems too low. I've been meaning to run a network on Kaggle's version for a while, so I'll do that and see how it does and get\r\n back to you.
" }, { "id": 25061, "postDate": "2013-06-02T19:16:45.143Z", "content": "[quote=Alec Radford;24788]
\r\n[quote=Yogesh Bhalerao;24766]
\r\nThanks Alec, I am now running with 250 hidden units and 1000 iterations. But will now work on understanding the accuracy with each iterations.
\r\nSince I am new to ML and NN, I am going to have to first understand what softmax/cross entrophy means (Need to find time to study Optimization Theory). My approach is the same as described in the NN topic of the Standford ML course (Andrew Ng) i.e., backpropagation+regularization+Conjugate\r\n Gradient
\r\n[/quote]
\r\nSoftmax/cross entropy isn't too bad. It's is a way of forcing the outputs of the NN to explicitly model a probability distribution over the classes/output units. Here's a video explaining it https://class.coursera.org/neuralnets-2012-001/lecture/47.\r\n (That's the video I learned it from and the wikipedia page). The error derivatives are real simple too so it's not hard to tack on to a more standard network architecture.
\r\nI haven't looked at the Stanford ML course's NN lectures yet, but they might have set it up that way without making it explicit that that was what they were doing.
\r\n[/quote]
\r\nI am using cross entropy. After bumping up the hidden units to 250 with 1000 iterations got an accuracy around 95%. Still a long way to go.
" }, { "id": 24948, "postDate": "2013-05-30T16:47:10.397Z", "content": "I used simple NN with 500 neurons in 1 hidden layer architecture. The NN "learn" in 16h, with 96.84% in testing data (and 100% in training data, i am thinking about over-fitting problem), and stuck there. I am trying to improve it...
" }, { "id": 24852, "postDate": "2013-05-28T00:53:29.547Z", "content": "I was computing the numerical gradient analytically and then comparing those results with my NN to determine if the gradient descent was working as expected. This was one debugging technique suggested by Andrew. But I think it seems that in my earlier attempt\r\n I used far many less activation units (25).
\r\n" }, { "id": 24842, "postDate": "2013-05-27T14:47:41.273Z", "content": "people here are posting things like "my accuracy on the test data set is 92%" but how are you actually able to know the accuracy of the test set without already knowing the correct answers? just curious...
" }, { "id": 24766, "postDate": "2013-05-24T21:44:27.637Z", "content": "Thanks Alec, I am now running with 250 hidden units and 1000 iterations. But will now work on understanding the accuracy with each iterations.
\r\nSince I am new to ML and NN, I am going to have to first understand what softmax/cross entrophy means (Need to find time to study Optimization Theory). My approach is the same as described in the NN topic of the Standford ML course (Andrew Ng) i.e., backpropagation+regularization+Conjugate\r\n Gradient
\r\n" }, { "id": 24746, "postDate": "2013-05-24T00:37:11.363Z", "content": "I was able to get 85% accuracy on the score, using a single hidden layer with 25 activation units, regularization and conjugate gradient descent for 50 iterations. Any suggestions on what I should be trying next to improve my accuracy? Note: I am new to\r\n machine learning and NN
" }, { "id": 24584, "postDate": "2013-05-21T09:04:32.217Z", "content": "[quote=Eugene;24519]
\r\nVivekn, that's worth to try. Thanks a lot for the idea! I believe such a dimensionality reduction is possible here, because we're working we pictures, but have you tried any other algorithms to reduce dimensionality that could potentially be applied to every\r\n dataset?
\r\nDo you think it's useful if I turn pictures from grayscale to black-white?. Would it improve computing speed or simplify model? How that would influence overall performance of algorithm? By algorithm I mean anything available, not only NN.
\r\n[/quote]
\r\nConverting to black and white helps reduce some noise in the data and works well for this dataset, though I am not sure how well it would work on other image datasets. There will not be much of a difference in terms of speed, since you still have the same\r\n number of weights. I didn't try other methods of dimensionality reduction, though there have been some posts in the forum that PCA doesn't work quite well. I wanted try out autoencoders for this, will do when I get some time.
" }, { "id": 24501, "postDate": "2013-05-19T20:38:36.077Z", "content": "[quote=vivekn;24489]
\r\nI downsampled the image to 14 x 14 and then applied some basic thresholding and then fed it to the network.
\r\n[/quote]
\r\nDo I get it correct that you basically get center of each image of 28x28 size and fed it to the network?
" }, { "id": 24489, "postDate": "2013-05-19T15:22:23.467Z", "content": "I am hitting a plateau at around 95.4% on the test set used for the leaderboard, though I am getting around 98.8% on a validation set I separated from the training data. I've experimented with different things - convolutional nets, L2 regularization, momentum\r\n etc with not much success. The best result I got on the validation set (98.8%) was using a simple 3 layer network with 500 hidden units trained using mini batch gradient descent for 20 minutes (20 epochs). I downsampled the image to 14 x 14 and then applied\r\n some basic thresholding and then fed it to the network.
\r\nBut the strange thing is I could get 95% accuracy on the leaderboard with only 50 units, albeit with lower accuracy on my validation set.
" }, { "id": 24415, "postDate": "2013-05-17T11:22:30.130Z", "content": "@fathersson: Yeah i did complete the NN with one hidden layer and 500 neurons. I meant NN by simple ANN. Well i used octave. But when i actually tried my NN on the test data i got around 92% not 98.5(my test and train data got mixed up). Yeah for me it took\r\n around 21hrs to complete with one hidden layer with 500 neurons. After that i couldn't work on it. Maybe this weekend i'll look into convolutional way of solving it.
" }, { "id": 24400, "postDate": "2013-05-16T22:15:45.597Z", "content": "[quote=@$#!$#;21614]
\r\nWith bit of heuristics and debugging i am getting more than 98.5% with simple ANN. Now i am curious what convolutional NN would do.
\r\n[/quote]
\r\n@$#!$#, have you been able to finish NN with hidden=500? It took for me three days and looks like it will simply takes at least same. I'm using R (not revolution) and its neuralnet package. Haven't tried others, yet.
\r\nWhat do you mean by 'simple ANN'? It seems that it's just another spelling for same NN like 'Artificial NN'. What tool did you use?
" }, { "id": 23132, "postDate": "2013-04-19T03:46:41.127Z", "content": "Guys, I am trying to use neuralnet for this problem, but R gives me some error that I do not understand. Your help is appreciated. Here is the code:
\r\nlibrary(neuralnet)
\r\n
\r\n
\r\n
\r\n
\r\n
mydata <- data.frame(labels, train)\r\n
nn <- neuralnet(labels ~ train, mydata, hidden=10, threshold=.01)\r\n\r\n
The error says:\r\nError in neurons[[i]] %*% weights[[i]] : non-conformable arguments
\r\nthanks for share
", "rawMarkdown": "thanks for share" } ], "comments": [ { "id": 2958874, "author_name": "sethpointaverage", "author_url": "", "post_date": "2024-08-14T12:53:06.667000", "content": "I was able to get to 99.5% accuracy on public leaderboard using Convolutional Neural Nets.
\n\nThis was jointly down to:
\n\n1) borrowing from Daniel's wonderful blog on how to implement CNN in Lasagne on Images (link below)
\n\n2) Synthesizing More Training Data for such data-hungry algorithm using Image Processing libraries in Python: \n - Rotate left and right\n - zoom\n - Stretch vertically & horizontally, \n - Elastic deformations. \nThe latter is particularly interesting, in that in a broad sense it allows us to create more variants of legit hand-writing than originally available in training.
\n\n3) reducing overfit using concepts of Early Convergence and Dropouts
\n\nFinal submission was a simple average-based ensemble of Top 5 CNNets.
\n\n", "votes": 4, "replies": [] }, { "id": 21412, "author_name": "Max Schumacher", "author_url": "", "post_date": "2013-03-19T12:17:25.053000", "content": "By far the coolest and mightiest approach right now is Deep Learning. See http://deeplearning.net/tutorial/, but it's quite a lot to learn if you only know about normal neural networks. For an easier way I\r\n would read up on Auto Encoders, Denoising Auto Encoders and Stacked Auto Encoders (basically do the above tutorial, just skip everything on Restricted Boltzmann Machines and Deep Belief Networks). You can get great results with just these techniques and they\r\n are easy to understand and implement. You also don't have to follow their python implementation, just reading through the theory parts should prepare you for doing it yourself. I just started and am at 94.6% with lots of room to optimize.
", "votes": 9, "replies": [] }, { "id": 65775, "author_name": "Florian Muellerklein", "author_url": "", "post_date": "2015-03-09T13:29:43.090000", "content": "I got 96% with this code, no feature engineering and lazy parameter tuning. So I think that it could do a lot better.
\nI used this competition to try and really understand how basic neural networks work so I wrote it using only numpy and in a way that should make it very readable and understandable. Sort of a hybrid between notes and function. I wrote two blog posts here and here that might be useful for people starting out. Hopefully all the information in there is accurate ;-). I'm still new to neural nets as well so I would appreciate some feedback from all the smart people here.
", "votes": 3, "replies": [] }, { "id": 65277, "author_name": "Nick Locascio", "author_url": "", "post_date": "2015-03-02T18:44:38.613000", "content": "I got 0.98886 on the test data set using a 8000 node, 3 layer neural net and some of the image techniques talked about in this thread. I explain my solution thoroughly here w/ code included: http://nicklocascio.com/neural-net-mnist-kaggle/.
", "votes": 3, "replies": [] }, { "id": 39515, "author_name": "Ilyas Potamitis", "author_url": "", "post_date": "2014-02-26T14:13:53.563000", "content": "A dbn that will give a bit over 99% on the public ldb
\nimport numpy as np
import pandas as pd
from nolearn.dbn import DBN
from scipy.ndimage import convolve
import scipy.ndimage as nd
###############################################################################
# Setting updef nudge_dataset(X, Y):
"""
This produces a dataset 5 times bigger than the original one,
by moving the 8x8 images in X around by 1px to left, right, down, up
"""
direction_vectors = [
[[0, 1, 0],
[0, 0, 0],
[0, 0, 0]],[[0, 0, 0],
[1, 0, 0],
[0, 0, 0]],[[0, 0, 0],
[0, 0, 1],
[0, 0, 0]],[[0, 0, 0],
[0, 0, 0],
[0, 1, 0]]]shift = lambda x, w: convolve(x.reshape((28, 28)), mode='constant',
weights=w).ravel()
X = np.concatenate([X] +
[np.apply_along_axis(shift, 1, X, vector)
for vector in direction_vectors])
Y = np.concatenate([Y for _ in range(5)], axis=0)
return X, Ydef rotate_dataset(X,Y):
XX = np.zeros(X_train.shape)
for index in range(X.shape[0]):
angle = np.random.randint(-7,7)
XX[index,:] = nd.rotate(np.reshape(X[index,:],((28,28))),angle,reshape=False).ravel()
X = np.vstack((X,XX))
Y = np.hstack((Y,Y))
return X, Y# Load Data
mnist = pd.read_csv("train.csv")
y_train = mnist['label'].values
X_train = mnist.loc[:,'pixel0':].values
X_test = pd.read_csv("test.csv").valuesX_train = np.asarray(X_train / 255.0, 'float32')
X_test = np.asarray(X_test / 255.0, 'float32')
X_train, y_train = nudge_dataset(X_train, y_train)
X_train, y_train = rotate_dataset(X_train, y_train)clf = DBN(
[X_train.shape[1], 350, 10],
learn_rates=0.3,
learn_rate_decays=0.95,
learn_rates_pretrain=0.005,
epochs=120,
verbose=1,
)clf.fit(X_train, y_train)subm = pd.read_csv("knn_benchmark.csv")
subm.Label = clf.predict(X_test)
subm.to_csv("result.csv", index_label='ImageId', col=['Label'], index=False)
Woot! My very first hand-coded perceptron network scored 88% - same as LeCun reports for 1-layer without preprocessing. Not an impressive score, perhaps, but nice to stride in the footsteps of the masters. Hand-coded allowed me to parallelize the classification ensemble (one per digit).
\nNext ... trying my hand at "Convolutional net LeNet-1 subsampling to 16x16 pixels"?
", "votes": 2, "replies": [] }, { "id": 25117, "author_name": "Alec Radford", "author_url": "", "post_date": "2013-06-04T01:55:46.597000", "content": "@Yogesh, Are you using a separate holdout validation set and watching the error rate on that? If you're just running your network for set amounts of iterations you're kind of shooting in the dark for what your accuracy will be. I just ran mine and it worked\r\n quite well (60th place at 0.986) so it looks like it carries over pretty well to Kaggle's data. This is using dropout and L2 hidden unit weight vector constraining which is good for about 1% more on MNIST so standard nets should still be able to get 0.97-0.975.\r\n All the hyper parameters are identical to what's used in the dropout paper I linked earlier.
\r\n@Eugene, what form of dimensionality reduction are you doing? Are you just rescaling the input down to 14x14? That kind of dimensionality reduction almost always hurts accuracy in the long run (may be made up in the short term for faster computation speeds)\r\n since your loosing information the network could use. You could try a technique like PCA dimensionality reduction and training a network on that (I think the general rule of thumb is use as many dimensions as necessary to keep 99% of variance).
\r\n
\r\nWhat is the runtime like for your NN? For reference, I'm using 2 hidden layers with 800 hidden units each and it takes about 20 minutes to get that 0.986, it's actually about the same speed as Scikit-Learns Random Forest (it get's to an accuracy of 0.967 in\r\n about 2 minutes). I am using a GPU accelerated version built on top of gnumpy though, which'll be about 10x faster than running it on the CPU.
\r\n
\r\nIf I understand your question correctly you're wondering about how different algorithms process the input to get their predictions and why there are such significant differences. In general things split into "low" and "high" dimensional algorithms. Things like\r\n SVM, NNs, and RF are good for high dimensional data where as lower dimensional data (less than 100 inputs) are typically the domain of linear models, I think. Specifically with Random Forest versus Neural Networks, (I don't know much about Random Forest to\r\n be honest) I think Random Forest relies on training a bunch of weak learners on random subsets of the data and features (via bagging and boosting I think) and combining them together into a larger model. Essentially it breaks the larger problem down into a\r\n bunch of smaller problems and tries to solve those and then get a coherent answer out of them. Traditional Neural Networks, are a method of automatically finding features in the input that end up being good predictors/discriminators of the output. For this\r\n digit task it'll learn features (a hidden unit is a feature detector to some degree) like the vertical bar of a 7 or the middle horiztonal part of a 3 and the output layer will learn how to weight those features such that it's more likely to predict a 7 when\r\n that hidden unit for the vertical bar of the 7 has a high activation for instance. This is all done via backpropagating the errors it makes on the networks and correcting the weights so it's less likely to make those errors. I started by reading the user guide/overview\r\n documentation on Scikit-Learn's website and when I was interested in specific models I looked for Coursera's or Udacity courses that covered them.
Is the output using softmax/cross entropy to explicitly force the NN to learn a probability distribution over the classes? Are you using momentum and minibatch learning? If not, those would be good things to add. Bumping up the hidden units a bit more and\r\n adding a second hidden layer may help as well but with the current speed of your network your going to start getting into 24 hour + training times...
\r\nDid you write the neural net yourself or are you using a library? What langauge? You can definitely get faster than 16 hours for 500 HUs.
\r\nFor overfitting you could look into implementing a recent technique like dropout. http://arxiv.org/pdf/1207.0580.pdf
", "votes": 3, "replies": [] }, { "id": 24788, "author_name": "Alec Radford", "author_url": "", "post_date": "2013-05-25T02:34:25.013000", "content": "[quote=Yogesh Bhalerao;24766]
\r\nThanks Alec, I am now running with 250 hidden units and 1000 iterations. But will now work on understanding the accuracy with each iterations.
\r\nSince I am new to ML and NN, I am going to have to first understand what softmax/cross entrophy means (Need to find time to study Optimization Theory). My approach is the same as described in the NN topic of the Standford ML course (Andrew Ng) i.e., backpropagation+regularization+Conjugate\r\n Gradient
\r\n[/quote]
\r\nSoftmax/cross entropy isn't too bad. It's is a way of forcing the outputs of the NN to explicitly model a probability distribution over the classes/output units. Here's a video explaining it https://class.coursera.org/neuralnets-2012-001/lecture/47.\r\n (That's the video I learned it from and the wikipedia page). The error derivatives are real simple too so it's not hard to tack on to a more standard network architecture.
\r\nI haven't looked at the Stanford ML course's NN lectures yet, but they might have set it up that way without making it explicit that that was what they were doing.
", "votes": 3, "replies": [] }, { "id": 62636, "author_name": "Andrew T. Crooks", "author_url": "", "post_date": "2015-01-12T08:34:41.157000", "content": "I tried this and it works great. However, the rotate function contains a loop that will exponentially increase the size of the Y array (i.e. it doubles it's size every loop and it goes for 42000 loops! so doing the math that is 2^42000 = not in this lifetime).
\nReplace the rotate_dataset function below with this alternative which actually works and note it only requires the X array because it's not rotating anything in Y.
\ndef rotate_dataset(X):
XX = np.zeros(X.shape)
for index in range(X.shape[0]):
angle = np.random.randint(-7,7)
XX[index,:] = nd.rotate(np.reshape(X[index,:],((28,28))),angle,reshape=False).ravel()
return XX
[quote=Rafael;39515]
\nA dbn that will give a bit over 99% on the public ldb
\nimport numpy as np
import pandas as pd
from nolearn.dbn import DBN
from scipy.ndimage import convolve
import scipy.ndimage as nd
###############################################################################
# Setting up
def nudge_dataset(X, Y):
"""
This produces a dataset 5 times bigger than the original one,
by moving the 8x8 images in X around by 1px to left, right, down, up
"""
direction_vectors = [
[[0, 1, 0],
[0, 0, 0],
[0, 0, 0]],[[0, 0, 0],
[1, 0, 0],
[0, 0, 0]],[[0, 0, 0],
[0, 0, 1],
[0, 0, 0]],[[0, 0, 0],
[0, 0, 0],
[0, 1, 0]]]
shift = lambda x, w: convolve(x.reshape((28, 28)), mode='constant',
weights=w).ravel()
X = np.concatenate([X] +
[np.apply_along_axis(shift, 1, X, vector)
for vector in direction_vectors])
Y = np.concatenate([Y for _ in range(5)], axis=0)
return X, Y
def rotate_dataset(X,Y):
XX = np.zeros(X_train.shape)
for index in range(X.shape[0]):
angle = np.random.randint(-7,7)
XX[index,:] = nd.rotate(np.reshape(X[index,:],((28,28))),angle,reshape=False).ravel()
X = np.vstack((X,XX))
Y = np.hstack((Y,Y))
return X, Y
# Load Data
mnist = pd.read_csv("train.csv")
y_train = mnist['label'].values
X_train = mnist.loc[:,'pixel0':].values
X_test = pd.read_csv("test.csv").valuesX_train = np.asarray(X_train / 255.0, 'float32')
X_test = np.asarray(X_test / 255.0, 'float32')
X_train, y_train = nudge_dataset(X_train, y_train)
X_train, y_train = rotate_dataset(X_train, y_train)clf = DBN(
[X_train.shape[1], 350, 10],
learn_rates=0.3,
learn_rate_decays=0.95,
learn_rates_pretrain=0.005,
epochs=120,
verbose=1,
)clf.fit(X_train, y_train)subm = pd.read_csv("knn_benchmark.csv")
subm.Label = clf.predict(X_test)
subm.to_csv("result.csv", index_label='ImageId', col=['Label'], index=False)
[/quote]
", "votes": 1, "replies": [] }, { "id": 24844, "author_name": "Eugene", "author_url": "", "post_date": "2013-05-27T15:31:40.067000", "content": "[quote=KL Tah;24842]
\r\npeople here are posting things like "my accuracy on the test data set is 92%" but how are you actually able to know the accuracy of the test set without already knowing the correct answers? just curious...
\r\n[/quote]
\r\nSubmit it =)
", "votes": 2, "replies": [] }, { "id": 40024, "author_name": "John Y", "author_url": "", "post_date": "2014-03-08T18:06:05.767000", "content": "Has anyone plotted learning curves to see if the training and cross validation set are converging, and to get an idea of whether the problem is bias or variance?
", "votes": 1, "replies": [] }, { "id": 28525, "author_name": "Daniel Rodriguez", "author_url": "", "post_date": "2013-08-10T16:49:41.883000", "content": "This was my experience with neural nets. I also took Ng's class but I made a more general implementation of the algorithm in python using numpy (not a fan of theano).
\nI was able to get about 95% accuracy training for a few minutes using mini batches of size 50.
\nMinibatches are the key for training the nets, doing full batch is crazy.
\nI don't have a GPU so only CPU training.
\nThen I read a paper of Hinton and he mentions that for MNIST the nets often are trained for days or even weeks. So I let my net training for a full night and I was able to get 97-98% accuracy.
\nAlso if you go to Lecun's MNIST page (http://yann.lecun.com/exdb/mnist/) you can see benchmarks for different architectures. If you are using only backprop I recommend using only one hidden layer. I used 300 hidden units.
\nSo my advice is: use mini-batches to overcome your memory issues and give more time to the nets.
\nAnd another tip: save often the weights of the net, that way you can start from that point. I saved every 10k iterations or so. So in case you ran out of memory of whatever you didn't lose your time.
\nGood luck!
", "votes": 1, "replies": [] }, { "id": 28472, "author_name": "Jared Huling", "author_url": "", "post_date": "2013-08-09T05:22:12.210000", "content": "yeah, Tim, using the original MNIST data is like being given the answers ahead of time. Don't do that
", "votes": 1, "replies": [] }, { "id": 28422, "author_name": "Tim Dettmers", "author_url": "", "post_date": "2013-08-08T09:29:25.760000", "content": "I just figured out that there is a very big difference in that extra data from the original MNIST set: So if you use the original MNIST data your test error will be much lower. The best I achieved with the kaggle data was 1.9 % test error with an ensemble of 30 neural networks. If I use an ensemble of 10 neural nets that were trained on the original MNIST set then I go down to 0.5 % test error on the kaggle test set (a single net has about 1.6% to 1.7% test error on the original MNIST set) – and this is not even fully optimized yet!
\nI found dropout to be very helpful as regularization. Also sparse initialization improves training speed and it seems to help a bit with generalization. I found momentum and also Nesterov momentum not very helpful — I guess these will be more helpful in training deep neural networks or on other data sets. Oddly enough I found 8000 hidden units much better than 800 hidden units; together with dropout the network does not overfit and generalizes well. But it takes a lot of time to train such a net without a GPU: To train 8000 hidden units to 1.2 % cross validation error takes a bit more than 10 minutes on my theano implementation – a good numpy implementation of 8000 hidden units should not take longer than 15 hours to train.
", "votes": 1, "replies": [] }, { "id": 25762, "author_name": "GundarsK", "author_url": "", "post_date": "2013-06-16T19:19:54.283000", "content": "I am also a newbie (well a relative newbie as I last coded ML algorithms in grad school), and also taking Andrew Ng's Coursera ML class. I used a modified version of NN from one of the class exercises (I added normalization and PCA). With 350 HUs (1 hidden\r\n layer) and 1k iterations I was able to get 100% on my train set, 98% on my test set, and 98.31% on the Kaggle test set upon submitting the answer.
\r\nThis took several hours to run (though PCA certainly improved it) as I was using Octave.
\r\nThank you everyone for the useful links. I liked the various docs at http://ufldl.stanford.edu/wiki/index.php/UFLDL_Tutorial as well.\r\n
\r\n", "votes": 1, "replies": [] }, { "id": 25641, "author_name": "Alec Radford", "author_url": "", "post_date": "2013-06-14T19:40:47.917000", "content": "@TravelingDesi, reading through the last few pages should give you a good overview of various methods to try. Also there's quite a few different ways to do regularization of Neural Networks, some of which are mentioned in the last few pages as well.
\r\nWeirdly enough, everyone who seems to be running Ng's networks seems to be getting scores that are very similar to what you can get with Logistic Regression ~ 92.5% http://deeplearning.net/tutorial/logreg.html#logreg
", "votes": 1, "replies": [] }, { "id": 24846, "author_name": "Alec Radford", "author_url": "", "post_date": "2013-05-27T16:49:50.227000", "content": "[quote=KL Tah;24842]
\r\npeople here are posting things like "my accuracy on the test data set is 92%" but how are you actually able to know the accuracy of the test set without already knowing the correct answers? just curious...
\r\n[/quote]
\r\n\r\nMNIST (the dataset this competition is based off of) is publicly available (train and test) and a great test dataset for various ML algorithms so that's one way of knowing it. The other is to just split out part of the train set to use as a validation set\r\n to test against. Scores won't carry over directly to the Kaggle test set, but they should be in the ballpark.
", "votes": 1, "replies": [] }, { "id": 24767, "author_name": "Eugene", "author_url": "", "post_date": "2013-05-24T22:25:37.693000", "content": "[quote=Yogesh Bhalerao;24746]
\r\nI was able to get 85% accuracy on the score, using a single hidden layer with 25 activation units, regularization and conjugate gradient descent for 50 iterations. Any suggestions on what I should be trying next to improve my accuracy? Note: I am new to\r\n machine learning and NN
\r\n[/quote]
\r\nI've got 90% accuracy with reduced dimensionality (thanks vivekn for the idea: instead of 784 input I've got only 176) and my NN had just 18 neurons in hidden layer.
\r\nNow I'm going to plot learning curves and see if it's giving anything useful. With 176 input values it's much simpler to debug my NN.
\r\nYogesh, I'd say plotting these guys is more helpful than trying something different (or learning new stuff). I believe it's worth to understand some sort of analysis or debugging techniques before going further. That actually was exactly what Andrew Ng suggested\r\n =)
", "votes": 1, "replies": [] }, { "id": 24763, "author_name": "Alec Radford", "author_url": "", "post_date": "2013-05-24T17:15:04.797000", "content": "Increasing the number of hidden units is a good step. Most papers with MNIST like a minimum of 500 hidden units, per layer and some go up to 2000. Also consider plotting accuracy as a function of iterations, you may simply need to run it longer to get more\r\n accurate.
\r\nAre you using softmax/cross entropy output? That usually helps get a little better too.
", "votes": 1, "replies": [] }, { "id": 24519, "author_name": "Eugene", "author_url": "", "post_date": "2013-05-20T13:15:04.213000", "content": "[quote=vivekn;24510]
\r\n\r\nI divided the image into 2x2 cells, for each 2x2 cell I took the maximum of the four intensities and used it as the intensity for the output image. I think a more better way of downsampling would be to apply a Gaussian blur and then take one of the four\r\n pixels. But this was faster and easy to implement.
\r\n[/quote]
\r\nVivekn, that's worth to try. Thanks a lot for the idea! I believe such a dimensionality reduction is possible here, because we're working we pictures, but have you tried any other algorithms to reduce dimensionality that could potentially be applied to every\r\n dataset?
\r\nDo you think it's useful if I turn pictures from grayscale to black-white?. Would it improve computing speed or simplify model? How that would influence overall performance of algorithm? By algorithm I mean anything available, not only NN.
", "votes": 1, "replies": [] }, { "id": 24510, "author_name": "vivekn", "author_url": "", "post_date": "2013-05-20T07:57:46.523000", "content": "[quote=fathersson;24501]
\r\n[quote=vivekn;24489]
\r\nI downsampled the image to 14 x 14 and then applied some basic thresholding and then fed it to the network.
\r\n[/quote]
\r\nDo I get it correct that you basically get center of each image of 28x28 size and fed it to the network?
\r\n[/quote]
\r\n\r\nI divided the image into 2x2 cells, for each 2x2 cell I took the maximum of the four intensities and used it as the intensity for the output image. I think a more better way of downsampling would be to apply a Gaussian blur and then take one of the four\r\n pixels. But this was faster and easy to implement.
\r\n", "votes": 1, "replies": [] }, { "id": 21614, "author_name": "@$#!$#", "author_url": "", "post_date": "2013-03-25T07:50:59.310000", "content": "With bit of heuristics and debugging i am getting more than 98.5% with simple ANN. Now i am curious what convolutional NN would do.
", "votes": 1, "replies": [] }, { "id": 21415, "author_name": "@$#!$#", "author_url": "", "post_date": "2013-03-19T13:57:44.347000", "content": "I tried to increase my neurons to 500 in the hidden layer to see the results. But its taking forever to complete this. It has already taken about 11 hrs and its still going on.
", "votes": 1, "replies": [] }, { "id": 98353, "author_name": "Koba", "author_url": "", "post_date": "2015-11-06T18:28:46.597000", "content": "Here is a notebook I wrote with example python code for a neural network and a convolutional nn. I get >0.96 accuracy in 500 seconds for 1000 nodes in the hidden layer in 15 epochs. Let me know if you have any questions.
", "votes": 0, "replies": [] }, { "id": 81873, "author_name": "@Akansha", "author_url": "", "post_date": "2015-06-14T19:16:16.477000", "content": "%Implementing back propagation on NN with 401 units in input, 25 in hidden and 10 output
\nd1= zeros(size(Theta1));
d2= zeros(size(Theta2));
ynew=zeros(num_labels,m);
help1=eye(num_labels);
for i=1:m
ynew(:,i)=help1(:,y(i,1));
end
for t=1:m
na1=X(t,:);
nz2=Theta1*na1';
na2=sigmoid(nz2);
na2=[1;na2];
nz3=Theta2*(na2);
na3=sigmoid(nz3);
delta3=na3-ynew(:,t);
delta2=(Theta2)' * (delta3 ) ;
delta2=delta2(2:end,:);
delta2=delta2.*sigmoidGradient(nz2);
d1=d1+delta2* na1;
d2=d2+delta3* na2';
Theta1_grad=Theta1_grad+(1/m)*d1;
Theta2_grad=Theta2_grad+(1/m)*d2;
end
Theta1_temp = [zeros(size(Theta1,1),1) Theta1(:,2:end)];
Theta2_temp = [zeros(size(Theta2,1),1) Theta2(:,2:end)];
Theta1_grad = Theta1_grad+ (lambda/m )* Theta1_temp;
Theta2_grad = Theta2_grad + (lambda/m) * Theta2_temp ;
what is wrong with the above implementation of Back propagation? I am new to ML and stuck at BP, getting accuracy of just 10% on my original training set ..?Any help
", "votes": 0, "replies": [] }, { "id": 72791, "author_name": "tburm", "author_url": "", "post_date": "2015-04-21T01:20:25.863000", "content": "Mostly as a learning exercise for myself, I have written a Python neural network library that wraps some simple C code using the CBLAS library for performance. I've gotten it to be pretty fast - I was able to score 97.9% on the test set training a neural net with 1 hidden layer of 400 nodes for just 5 minutes (30 iterations over the training set) on my Core i5 MacBook Pro. Anyway, I thought other people might find it interesting or useful and I would love to get feedback on it. You can get the code here: https://github.com/tburmeister/pyneural
\nThe library is still very much a work in progress, and in particular I don't have a good install script yet, but it shouldn't be too hard to get running. Once it's installed, it should be pretty simple to use.
\nEdit: update for those interested - I'm now up to over 99% accuracy on the test set, with a training time of roughly 30 minutes. I have added an IPython notebook with my training code to my git repo.
", "votes": 0, "replies": [] }, { "id": 65281, "author_name": "inversion", "author_url": "", "post_date": "2015-03-02T19:30:22.497000", "content": "@Nick -
\nHow does your score scale with node size?
\nI got 0.98871 with hidden_layer_sizes = (1568,200)
\n(But that includes adding some synthetic training examples.)
\nEDIT: Just saw you had the synthetic data too. Thanks for the blog writeup. Very nice.
", "votes": 0, "replies": [] }, { "id": 64096, "author_name": "", "author_url": "", "post_date": "2015-02-12T07:01:26.397000", "content": "", "votes": 0, "replies": [] }, { "id": 62775, "author_name": "", "author_url": "", "post_date": "2015-01-16T03:29:37.327000", "content": "", "votes": 0, "replies": [] }, { "id": 59433, "author_name": "", "author_url": "", "post_date": "2014-12-03T02:25:29.077000", "content": "", "votes": 0, "replies": [] }, { "id": 59429, "author_name": "", "author_url": "", "post_date": "2014-12-03T01:26:46.977000", "content": "", "votes": 0, "replies": [] }, { "id": 59428, "author_name": "", "author_url": "", "post_date": "2014-12-03T00:17:20.430000", "content": "", "votes": 0, "replies": [] }, { "id": 59014, "author_name": "", "author_url": "", "post_date": "2014-11-26T16:27:22.423000", "content": "", "votes": 0, "replies": [] }, { "id": 58828, "author_name": "", "author_url": "", "post_date": "2014-11-24T23:07:51.853000", "content": "", "votes": 0, "replies": [] }, { "id": 58509, "author_name": "", "author_url": "", "post_date": "2014-11-20T10:18:56.540000", "content": "", "votes": 0, "replies": [] }, { "id": 58285, "author_name": "", "author_url": "", "post_date": "2014-11-18T07:27:13.080000", "content": "", "votes": 0, "replies": [] }, { "id": 56886, "author_name": "", "author_url": "", "post_date": "2014-10-28T06:16:42.427000", "content": "", "votes": 0, "replies": [] }, { "id": 54232, "author_name": "", "author_url": "", "post_date": "2014-09-18T10:54:46.550000", "content": "", "votes": 0, "replies": [] }, { "id": 39488, "author_name": "", "author_url": "", "post_date": "2014-02-26T04:08:22.083000", "content": "", "votes": 0, "replies": [] }, { "id": 30018, "author_name": "", "author_url": "", "post_date": "2013-09-02T21:42:33.367000", "content": "", "votes": 0, "replies": [] }, { "id": 29233, "author_name": "", "author_url": "", "post_date": "2013-08-22T13:15:52.783000", "content": "", "votes": 0, "replies": [] }, { "id": 28591, "author_name": "", "author_url": "", "post_date": "2013-08-12T15:55:32.317000", "content": "", "votes": 0, "replies": [] }, { "id": 28540, "author_name": "", "author_url": "", "post_date": "2013-08-11T03:48:07.087000", "content": "", "votes": 0, "replies": [] }, { "id": 28532, "author_name": "", "author_url": "", "post_date": "2013-08-10T21:09:10.483000", "content": "", "votes": 0, "replies": [] }, { "id": 28531, "author_name": "", "author_url": "", "post_date": "2013-08-10T20:48:18.793000", "content": "", "votes": 0, "replies": [] }, { "id": 28528, "author_name": "", "author_url": "", "post_date": "2013-08-10T17:48:21.227000", "content": "", "votes": 0, "replies": [] }, { "id": 28477, "author_name": "", "author_url": "", "post_date": "2013-08-09T06:56:16.347000", "content": "", "votes": 0, "replies": [] }, { "id": 28474, "author_name": "", "author_url": "", "post_date": "2013-08-09T05:35:32.590000", "content": "", "votes": 0, "replies": [] }, { "id": 28473, "author_name": "", "author_url": "", "post_date": "2013-08-09T05:23:50.127000", "content": "", "votes": 0, "replies": [] }, { "id": 28460, "author_name": "", "author_url": "", "post_date": "2013-08-08T21:14:10.873000", "content": "", "votes": 0, "replies": [] }, { "id": 27446, "author_name": "", "author_url": "", "post_date": "2013-07-19T14:49:38.503000", "content": "", "votes": 0, "replies": [] }, { "id": 27325, "author_name": "", "author_url": "", "post_date": "2013-07-17T05:02:35.690000", "content": "", "votes": 0, "replies": [] }, { "id": 27261, "author_name": "", "author_url": "", "post_date": "2013-07-16T03:35:01.927000", "content": "", "votes": 0, "replies": [] }, { "id": 27225, "author_name": "", "author_url": "", "post_date": "2013-07-15T11:07:14.403000", "content": "", "votes": 0, "replies": [] }, { "id": 25646, "author_name": "", "author_url": "", "post_date": "2013-06-14T20:12:35.897000", "content": "", "votes": 0, "replies": [] }, { "id": 25609, "author_name": "", "author_url": "", "post_date": "2013-06-14T13:00:23.207000", "content": "", "votes": 0, "replies": [] }, { "id": 25139, "author_name": "", "author_url": "", "post_date": "2013-06-04T21:32:58.713000", "content": "", "votes": 0, "replies": [] }, { "id": 25135, "author_name": "", "author_url": "", "post_date": "2013-06-04T19:50:02.527000", "content": "", "votes": 0, "replies": [] }, { "id": 25107, "author_name": "", "author_url": "", "post_date": "2013-06-03T22:04:15.350000", "content": "", "votes": 0, "replies": [] }, { "id": 25106, "author_name": "", "author_url": "", "post_date": "2013-06-03T21:43:21.313000", "content": "", "votes": 0, "replies": [] }, { "id": 25061, "author_name": "", "author_url": "", "post_date": "2013-06-02T19:16:45.143000", "content": "", "votes": 0, "replies": [] }, { "id": 24948, "author_name": "", "author_url": "", "post_date": "2013-05-30T16:47:10.397000", "content": "", "votes": 0, "replies": [] }, { "id": 24852, "author_name": "", "author_url": "", "post_date": "2013-05-28T00:53:29.547000", "content": "", "votes": 0, "replies": [] }, { "id": 24842, "author_name": "", "author_url": "", "post_date": "2013-05-27T14:47:41.273000", "content": "", "votes": 0, "replies": [] }, { "id": 24766, "author_name": "", "author_url": "", "post_date": "2013-05-24T21:44:27.637000", "content": "", "votes": 0, "replies": [] }, { "id": 24746, "author_name": "", "author_url": "", "post_date": "2013-05-24T00:37:11.363000", "content": "", "votes": 0, "replies": [] }, { "id": 24584, "author_name": "", "author_url": "", "post_date": "2013-05-21T09:04:32.217000", "content": "", "votes": 0, "replies": [] }, { "id": 24501, "author_name": "", "author_url": "", "post_date": "2013-05-19T20:38:36.077000", "content": "", "votes": 0, "replies": [] }, { "id": 24489, "author_name": "", "author_url": "", "post_date": "2013-05-19T15:22:23.467000", "content": "", "votes": 0, "replies": [] }, { "id": 24415, "author_name": "", "author_url": "", "post_date": "2013-05-17T11:22:30.130000", "content": "", "votes": 0, "replies": [] }, { "id": 24400, "author_name": "", "author_url": "", "post_date": "2013-05-16T22:15:45.597000", "content": "", "votes": 0, "replies": [] }, { "id": 23132, "author_name": "", "author_url": "", "post_date": "2013-04-19T03:46:41.127000", "content": "", "votes": 0, "replies": [] }, { "id": 2958799, "author_name": "", "author_url": "", "post_date": "2024-08-14T11:16:48.830000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "2958874": "1. Learn about CNNs\n2. Use data augmentation (rotations, random crops, or libraries like RandAugment)\n3. Do hyperparameter tuning, optimize the size of the neural network, the optimizer learning rate, and the amount of regularization ", "114321": "I was able to get to 99.5% accuracy on public leaderboard using Convolutional Neural Nets. \r\n\r\nThis was jointly down to:\r\n\r\n1) borrowing from Daniel's wonderful blog on how to implement CNN in Lasagne on Images (link below)\r\n\r\n2) Synthesizing More Training Data for such data-hungry algorithm using Image Processing libraries in Python: \r\n - Rotate left and right\r\n - zoom\r\n - Stretch vertically & horizontally, \r\n - Elastic deformations. \r\nThe latter is particularly interesting, in that in a broad sense it allows us to create more variants of legit hand-writing than originally available in training.\r\n\r\n3) reducing overfit using concepts of Early Convergence and Dropouts\r\n\r\n\r\nFinal submission was a simple average-based ensemble of Top 5 CNNets.\r\n\r\nhttp://danielnouri.org/notes/2014/12/17/using-convolutional-neural-nets-to-detect-facial-keypoints-tutorial/\r\n", "21412": "", "65775": "", "65277": "", "39515": "", "55760": "", "25117": "", "24957": "", "24788": "", "21384": "", "62636": "", "24844": "", "40024": "", "28525": "", "28472": "", "28422": "", "25762": "", "25641": "", "24846": "", "24767": "", "24763": "", "24519": "", "24510": "", "21614": "", "21415": "", "98353": "Here is a [notebook I wrote with example python][1] code for a neural network and a convolutional nn. I get >0.96 accuracy in 500 seconds for 1000 nodes in the hidden layer in 15 epochs. Let me know if you have any questions.\r\n\r\n\r\n [1]: https://www.kaggle.com/kobakhit/digit-recognizer/digit-recognizer-in-python-using-cnn", "81873": "", "72791": "", "65281": "", "64096": "", "62775": "", "59433": "", "59429": "", "59428": "", "59014": "", "58828": "", "58509": "", "58285": "", "56886": "", "54232": "", "39488": "", "30018": "", "29233": "", "28591": "", "28540": "", "28532": "", "28531": "", "28528": "", "28477": "", "28474": "", "28473": "", "28460": "", "27446": "", "27325": "", "27261": "", "27225": "", "25646": "", "25609": "", "25139": "", "25135": "", "25107": "", "25106": "", "25061": "", "24948": "", "24852": "", "24842": "", "24766": "", "24746": "", "24584": "", "24501": "", "24489": "", "24415": "", "24400": "", "23132": "", "2958799": "thanks for share" } }