![\"SdA](\"http://i.imgur.com/ZQJDH.png\" "\\"SdA")
I was excited to know what all methods is everyone using in this event. Mostly for preprocessing.:)
", "messages": [ { "id": 15712, "postDate": "2012-10-16T16:29:14.243Z", "content": "My first serious entry was about 97.5% accurate.
\r\nI used PCA to extract features retaining 90% of the variance (goes from 784 to about 80 features) and then used 1-nearest-neighbor.
\r\nI'm now looking at single hidden layer neural nets (100 hidden neurons) and getting 95-96% accuracy. This works better if I don't do dimensionality reduction first.
\r\nI'm going to look at SVMs next. The final classifier will probably just be everything voting on the right answer.
", "votes": 5 }, { "id": 16676, "postDate": "2012-11-07T18:14:44.287Z", "content": "Running the convolutional neural net for 105 epochs gave me the 98.8 score.
\r\n@Matt After you've done pre-training the SdA's you can use drop-out to train your model without having to worry about overfitting.
\r\nI've gotten pretty good results with it so far i.e. ~97% and Geoffrey Hinton have showed very promising results with it!
", "votes": 2 }, { "id": 24407, "postDate": "2013-05-17T03:30:01.283Z", "content": "[quote=fathersson;24401]When you used 800 neurons how much time did it take? It seems that it takes more than 3 days for me for just 500 hidden neurons in just one hidden layer. I'm using R (cran one, not Revolution) and neuralnet package on my Mac with\r\n 16 Gb RAM. Any advice would be appreciated.[/quote]
\r\nIs there the ability to print debug or status info as the training goes for that package to see what's going on? If the learning rate is too high it might hit an instability and just stall out if it has some internal stopping parameters it's using. Another\r\n thing is it may need normalized input between 0 and 1.
\r\nWere there any other parameters you needed to set and if so, what were they? 3 days is way too long. A similarish net with 800 hidden units written in python with numpy (so by no means fast) get's around 90% within a few minutes and near 98% within an hour\r\n or two. I was testing on MNIST itself so I don't know how it'll translate to kaggle scores though.
\r\nIf you want to play more you could try your own basic implementation. I followed this coursera and was quite happy with it https://class.coursera.org/neuralnets-2012-001/class/index
\r\n[quote=fathersson;24401]But may be this kind of problem is not really meant to be solved by NN.[/quote]
\r\nThe state of the art for this problem is currently a type of NN that's at 99.77% so don't give up hope :)
\r\nhttp://arxiv.org/pdf/1202.2745v1.pdf
", "votes": 1 }, { "id": 18807, "postDate": "2012-12-30T08:46:16.397Z", "content": "I am a bit skeptical about reducing features in such a problem (and indeed results are always worse).
\r\nFeature reduction makes sense when the number of observation is less than the number of features. In this case 42000 observations can greatly handle 784 features and solve the problem with higher accuracy.
\r\nI tried SVM without feature reduction (>98%) and with SVD (~80% reduction) that gave me a worse accuracy below 94%.
True, that is to a large degree what is happening when using SdA, but only during the pre-training! The "fine-tuning" training is not regularized in this way.
\r\nIn my opinion if you are not regularizing and are not experiencing overfitting then either you are not training long enough or do not have a big enough network ;)
\r\nWith regards to smaller batches learning faster: usually an epoch is a sweep of all training examples (size N), so if you train with batches of size 1, there will be N weight updates per epoch, whereas if you train with batches of size 100 there will be\r\n N/100 weight updates per epoch. Just something you might want to consider when judging which method learns faster.
\r\nCheers, Rasmus.
", "votes": 1 }, { "id": 16681, "postDate": "2012-11-07T23:21:15.780Z", "content": "Thanks for the info and the reference!
\r\nI believe a similar effect is at work in SdA where a random fraction of inputs in each layer are corrupted (i.e. set to zero) in each training batch. The following figure displays the validation error vs. fine-tuning epoch for a few SdA architectures.\r\n
\r\n
For the larger mini-batches - more averaging of training entries for computing loss gradients - validation error is close to monotonically decreasing. Although smaller batches converge faster so I'm leaning towards those.
", "votes": 1 }, { "id": 16679, "postDate": "2012-11-07T21:21:33.587Z", "content": "Basically you set the output of neurons in hidden layers to zero, with 50% probability, and otherwise train it as you normally would, i.e. SGD. This forces the NN to be very robust and redundant. It can also be seen as a very effecient way of doing modelling\r\n averaging. Take a look at the impressive paper from G.H. http://arxiv.org/abs/1207.0580
\r\nI wanted to verify the method so i trained a 784-800-800-10 FFNN with SGD and dropout for 1000 epochs (took around 11 hours on my laptop) i.e. a huge network trained for a loong time: should lead to overfitting. After each epoch i recorded the validation\r\n error and plotted it. See attached plot.
\r\nIt's really incredible that the validation error is decreasing with no sign of overfitting.
\r\n\r\n
\r\n
\r\n
", "votes": 1 }, { "id": 16672, "postDate": "2012-11-07T16:43:34.277Z", "content": "
Stacked Denoising Autoencoders using DeepLearning/Theano:
\r\nhttp://deeplearning.net/tutorial/SdA.html
\r\nJust adapting their SdA code to load the Kaggle training and test sets and using default parameters will get you into the top 50 on the public leader board!
\r\nTo improve on that, I've been exploring variations of the hyperparameters through bagging. For each set of hyperparameters, I train atleast 5 SdA using a 90/10 train/valid split and then look at convergence of validation error with extent of fine-tuning.\r\n This method has served me well so far, but it takes a lot of computing power. I only have 14 cores at my disposal and no GPUs so I can’t do as much exploration as I’d like. Additionally, I’m concerned the 90/10 split may be too aggressive and it may need\r\n reduced to 80/20.
\r\nI’d strongly recommend everyone checkout Theano and the DeepLearning tutorials. If you have access to GPUs, you’ll be able to try out a lot of models. Their Restricted Boltzmann Machine also looks promising. If I had the computing power, I’d be exploring\r\n these methods also.
", "votes": 1 }, { "id": 15406, "postDate": "2012-10-10T19:05:35.793Z", "content": "50 hidden neurons is way too small for this problem. I used 800 neurons in each of the hidden layers. Maybe less would have worked too. When using large networks and unmodified images as training data, it's important to have a validation set as well, otherwise\r\n networks tend to overfit.
", "votes": 1 }, { "id": 15400, "postDate": "2012-10-10T17:14:50.670Z", "content": "hi amenem, how many units are you using in your NN hidden layer? I've tried with 50 and only got 93% accuracy on the training set
", "votes": 1 }, { "id": 24479, "postDate": "2013-05-18T23:38:16.320Z", "content": "Glad to hear you're getting results now!
\r\nIt's single threaded, I've been thinking about adding multi-threading to it, but I wanted to squeeze the performance I could get out of a single thread first. There's some libraries that offer a sort of insta multithreading for numpy (numexpr) but it relies\r\n on Intel's MKL which isn't free for commerical use. That and I'm not sure if my dropout implementation is working like it should...
" }, { "id": 24476, "postDate": "2013-05-18T21:17:28.573Z", "content": "[quote=Alec Radford;24407]
\r\nIs there the ability to print debug or status info as the training goes for that package to see what's going on? If the learning rate is too high it might hit an instability and just stall out if it has some internal stopping parameters it's using. Another\r\n thing is it may need normalized input between 0 and 1.
\r\nWere there any other parameters you needed to set and if so, what were they? 3 days is way too long. A similarish net with 800 hidden units written in python with numpy (so by no means fast) get's around 90% within a few minutes and near 98% within an hour\r\n or two. I was testing on MNIST itself so I don't know how it'll translate to kaggle scores though.
\r\n[/quote]
\r\nAlec, thanks a lot for valuable advices. There is indeed a way to print out debug info. It turns out that I specified threshold that is too smal and didn't specified max step parameter, so it would indefinitely try best to converge.
\r\nI wouldn't say it takes minutes, but at least I've got what exactly it is doing.
\r\nBTW, does your implementation with NumPy use all cores (if you have several)? Because R is definitely single threaded...
" }, { "id": 24401, "postDate": "2013-05-16T22:23:54.613Z", "content": "[quote=ahans;15406]
\r\n50 hidden neurons is way too small for this problem. I used 800 neurons in each of the hidden layers. Maybe less would have worked too. When using large networks and unmodified images as training data, it's important to have a validation set as well, otherwise\r\n networks tend to overfit.
\r\n[/quote]
\r\nAhans, could you share some light on the problem?
\r\nWhen you used 800 neurons how much time did it take? It seems that it takes more than 3 days for me for just 500 hidden neurons in just one hidden layer. I'm using R (cran one, not Revolution) and neuralnet package on my Mac with 16 Gb RAM. Any advice would\r\n be appreciated.
\r\nI would like to play more with NN, rather than try something else.
\r\nBut may be this kind of problem is not really meant to be solved by NN.
" }, { "id": 18955, "postDate": "2013-01-04T03:11:01.447Z", "content": "[quote=Phillip Chilton Adkins;18811]
\r\nDid you choose your features as part of a cross-validation procedure? Is your 80% test or train? I had a different experience; my cross-val got much better when i limited the number of components. It's not true that you want to limit the number of components\r\n only if if n[/quote]
\r\n\r\n
Yes, I chose the number of features using cross-validation. 80% of the data was used to train and the remaining 20% as test.
" }, { "id": 18921, "postDate": "2013-01-03T02:37:33.273Z", "content": "I've just made my first submission, I used Nearest Centroid with no preprocessing, got 80% accuracy. I sure need to refine that a lot
" }, { "id": 18813, "postDate": "2012-12-30T10:51:07.187Z", "content": "That is certainly one thing i did. I'm currently getting about 95% with it in x-val, but i haven't made a submission yet. I'm going to try to make it better first." }, { "id": 18812, "postDate": "2012-12-30T10:37:11.493Z", "content": "[quote=Phillip Chilton Adkins;18811]
\r\nDid you choose your features as part of a cross-validation procedure? Is your 80% test or train? I had a different experience; my cross-val got much better when i limited the number of components. It's not true that you want to limit the number of components\r\n only if if n[/quote]
\r\n\r\n
\r\n
Yes I chose as cross validation.
\r\nYour point is interesting and makes sense. Only curiosity, what accuracy did you get on these data?
\r\nI guess you used PCA + k-NN.
Correct me if I'm wrong.
" }, { "id": 18811, "postDate": "2012-12-30T10:28:50.747Z", "content": "Did you choose your features as part of a cross-validation procedure? Is your 80% test or train? I had a different experience; my cross-val got much better when i limited the number of components. It's not true that you want to limit the number of components\r\n only if if n<p. depending on how noisy the data is, you might want to reduce the number of components to, say, n/50 (50 data points per component). Higher components may contain useful information but they can also contain noise. Often, looking at the components\r\n can be a good way to decide when they transition from structure to noise." }, { "id": 18809, "postDate": "2012-12-30T10:22:11.843Z", "content": "When you do a PCA, the columns usually come out in order of decreasing variance. Therefore, columns with zero variance will come at the end of the list. Just take a look at the eigenvalues - any zero eigenvalues will correspond to a column with zero variance." }, { "id": 18808, "postDate": "2012-12-30T08:49:10.713Z", "content": "[quote=Phillip Chilton Adkins;18797]
\r\nColumns with zero variance should be pushed into the final eigenspace - as long as you choose anything but the final eigenspace when doing the PCA truncation, those columns will be completely ignored.
\r\n[/quote]
\r\n\r\n
Cool. So you first reorder columns according to their variance, I guess, and work on a slightly reduced matrix.
" }, { "id": 18804, "postDate": "2012-12-30T06:01:03.707Z", "content": "I tried to run a PCA and then a random forest for classification. The number of principle components was chosen by a cross validation method. I trained a random forest on 80% of the data and tested it on the remaining 20%. The in-sample Out of Bag Prediction\r\n Rates and the test prediction rates are plotted below for various numbers of Priniple Components. Somewhere between 40 and 50 Principle Components give the best prediction rate of about a 95.3%.
" }, { "id": 18797, "postDate": "2012-12-29T21:40:31.527Z", "content": "Columns with zero variance should be pushed into the final eigenspace - as long as you choose anything but the final eigenspace when doing the PCA truncation, those columns will be completely ignored.
" }, { "id": 18530, "postDate": "2012-12-24T08:23:14.437Z", "content": "Sean,how did you deal with columns with zero variance?
" }, { "id": 17992, "postDate": "2012-12-17T08:14:09.457Z", "content": "I have used SVM too, it could perform well..
" }, { "id": 17951, "postDate": "2012-12-16T19:51:31.920Z", "content": "eh curious... i was trying the other way... SVM and then a classification method like RF.
\r\nSVM with a radial basis function can be squeezed up to a 97% of accuracy... not bad. I'm struggling a bit for that 3% :)
" }, { "id": 17740, "postDate": "2012-12-12T18:37:23.567Z", "content": "Friends, I have another question on PCA. What data set do you use to fit it? Simply whole train dataset? Or some subset of it?
" }, { "id": 17603, "postDate": "2012-12-10T19:17:23.773Z", "content": "To Sean:
\r\nYeah. I just used PCA library in a wrong way :)
" }, { "id": 17573, "postDate": "2012-12-09T21:11:49.823Z", "content": "2 Ivan - pca took maybe 1 minute on my 8gb windows 7 (64bit) laptop
" }, { "id": 17568, "postDate": "2012-12-09T19:18:45.600Z", "content": "\r\n>> I used PCA to extract features retaining 90% of the variance (goes from 784 to about 80 features) and then used 1-nearest-neighbor.
\r\nHow long did it take you to apply PCA on train data?
" }, { "id": 16812, "postDate": "2012-11-12T10:34:22.950Z", "content": "I used dense SIFT and AOSO-Logitboost[1], which work pretty good.
\r\n[1]: http://icml.cc/discuss/2012/569.html
" }, { "id": 16747, "postDate": "2012-11-09T15:37:53.580Z", "content": "Thanks for the clarification and advice Rasmus.
\r\nI’ll definitely keep these ideas in mind as I further develop this SdA model.
" }, { "id": 16678, "postDate": "2012-11-07T20:45:32.120Z", "content": "[quote=rasmusbergpalm;16676]
\r\n@Matt After you've done pre-training the SdA's you can use drop-out to train your model without having to worry about overfitting.
\r\nI've gotten pretty good results with it so far i.e. ~97% and Geoffrey Hinton have showed very promising results with it!
\r\n[/quote]
\r\nThanks for the advice. Could you please elaborate on what is implied by "drop-out"?
\r\nCurrently, I'm using stochastic gradient descent with single entry mini-batches for the fine-tuning training. Empirically, I've found validation error decreases roughly monotonically as a function of fine-tuning epoch and it appears overfitting isn't an\r\n issue at this stage.
" }, { "id": 16419, "postDate": "2012-10-31T22:16:24.743Z", "content": "I'm using Convolutional Neural Nets using my deep learning toolbox for the training. (https://github.com/rasmusbergpalm/DeepLearnToolbox)\r\n
\r\nWith this network (https://github.com/rasmusbergpalm/DeepLearnToolbox/blob/master/CNN/cnnexamples.m) and 20 epochs i got 97.4. I'll let it run for 100 epochs and see what that gives.\r\n
\r\nYou should disregard my 0.99% accuracy submission for now, as the model was taken from my thesis and was trained on 60.000 randomly selected images of the 70.000 total, i.e. it has trained on the test data provided by kaggle, which explains the good score.\r\n When tested on my test set (which it has not trained on) it got 0.9878 accuracy.
" }, { "id": 15417, "postDate": "2012-10-11T00:44:54.510Z", "content": "Hi \r\nFrans Slothouber..What is the depth of the random forest you are designing and the umber of forests..I have been stuck with 86% accuracy and can't seem to improve on it!
" }, { "id": 15334, "postDate": "2012-10-09T08:43:58.110Z", "content": "hi what accuracy you got ...i used 1-hidden layer NN as well as simple kNN.
\r\nkNN gave much better results with 97 % accuracy while for 3 layer NN it was 95%.
\r\nprobably NN might improve if i use two or more hidden layers.
" }, { "id": 15302, "postDate": "2012-10-08T20:01:42.803Z", "content": "Just submitted a first attempt using a simple 4-layer neural network. I randomly split the training data into 90% training and 10% validation data, used the training data for weight adjustment, then used the weights that produced the best result on the validation\r\n set to predict the values for the competition's test data.
" }, { "id": 13641, "postDate": "2012-08-30T12:18:21.610Z", "content": "I'm trying to evolve a network using an implementation of NEAT(http://en.wikipedia.org/wiki/Neuroevolution_of_augmenting_topologies).
\r\nSo far I haven't spent time on pre-processing\\feature extraction : I've simply jammed the network with all 784 inputs and held my breath. Not surprisingly progress is VERY slow because of this, but I'm curious to see what NEAT can achieve. Last time I checked\r\n the best network accuracy was 0.75 and still improving(slowly).
\r\nI'll give it a bit more time and start thinking about feature extraction in order to reduce network complexity.
\r\nHappy mining.
" }, { "id": 13539, "postDate": "2012-08-27T23:34:14.357Z", "content": "Mainly using random forest and post processing. For the random forest method some entries in the test set are easy, others are difficult. I've been filtering out these difficult entries in the test set and post processing these with different methods. Sort\r\n of a two stage process.
\r\nBeen thinking of using a SVM to predict the wrong guesses of random forest. (But that might turn out to be a Baron Münchhausen pulling himself out of the quicksand by his own boots kind of trick).
\r\nTried to 'improve' the training and test set with feature extraction but without much success.
\r\nAlso looked at the performance characteristics of random forest (see Prospect section). Looks like for random forest expanding the training set might give some improvement.
\r\nMy goal is to see what the maximum is that can be acchieved with the random forest method.
\r\n" }, { "id": 13062, "postDate": "2012-08-12T07:55:04.330Z", "content": "
I was excited to know what all methods is everyone using in this event. Mostly for preprocessing.:)
" } ], "comments": [ { "id": 15712, "author_name": "Chris Taylor", "author_url": "", "post_date": "2012-10-16T16:29:14.243000", "content": "My first serious entry was about 97.5% accurate.
\r\nI used PCA to extract features retaining 90% of the variance (goes from 784 to about 80 features) and then used 1-nearest-neighbor.
\r\nI'm now looking at single hidden layer neural nets (100 hidden neurons) and getting 95-96% accuracy. This works better if I don't do dimensionality reduction first.
\r\nI'm going to look at SVMs next. The final classifier will probably just be everything voting on the right answer.
", "votes": 5, "replies": [] }, { "id": 16676, "author_name": "rasmusbergpalm", "author_url": "", "post_date": "2012-11-07T18:14:44.287000", "content": "Running the convolutional neural net for 105 epochs gave me the 98.8 score.
\r\n@Matt After you've done pre-training the SdA's you can use drop-out to train your model without having to worry about overfitting.
\r\nI've gotten pretty good results with it so far i.e. ~97% and Geoffrey Hinton have showed very promising results with it!
", "votes": 2, "replies": [] }, { "id": 24407, "author_name": "Alec Radford", "author_url": "", "post_date": "2013-05-17T03:30:01.283000", "content": "[quote=fathersson;24401]When you used 800 neurons how much time did it take? It seems that it takes more than 3 days for me for just 500 hidden neurons in just one hidden layer. I'm using R (cran one, not Revolution) and neuralnet package on my Mac with\r\n 16 Gb RAM. Any advice would be appreciated.[/quote]
\r\nIs there the ability to print debug or status info as the training goes for that package to see what's going on? If the learning rate is too high it might hit an instability and just stall out if it has some internal stopping parameters it's using. Another\r\n thing is it may need normalized input between 0 and 1.
\r\nWere there any other parameters you needed to set and if so, what were they? 3 days is way too long. A similarish net with 800 hidden units written in python with numpy (so by no means fast) get's around 90% within a few minutes and near 98% within an hour\r\n or two. I was testing on MNIST itself so I don't know how it'll translate to kaggle scores though.
\r\nIf you want to play more you could try your own basic implementation. I followed this coursera and was quite happy with it https://class.coursera.org/neuralnets-2012-001/class/index
\r\n[quote=fathersson;24401]But may be this kind of problem is not really meant to be solved by NN.[/quote]
\r\nThe state of the art for this problem is currently a type of NN that's at 99.77% so don't give up hope :)
\r\nhttp://arxiv.org/pdf/1202.2745v1.pdf
", "votes": 1, "replies": [] }, { "id": 18807, "author_name": "Piggy ", "author_url": "", "post_date": "2012-12-30T08:46:16.397000", "content": "I am a bit skeptical about reducing features in such a problem (and indeed results are always worse).
\r\nFeature reduction makes sense when the number of observation is less than the number of features. In this case 42000 observations can greatly handle 784 features and solve the problem with higher accuracy.
\r\nI tried SVM without feature reduction (>98%) and with SVD (~80% reduction) that gave me a worse accuracy below 94%.
True, that is to a large degree what is happening when using SdA, but only during the pre-training! The "fine-tuning" training is not regularized in this way.
\r\nIn my opinion if you are not regularizing and are not experiencing overfitting then either you are not training long enough or do not have a big enough network ;)
\r\nWith regards to smaller batches learning faster: usually an epoch is a sweep of all training examples (size N), so if you train with batches of size 1, there will be N weight updates per epoch, whereas if you train with batches of size 100 there will be\r\n N/100 weight updates per epoch. Just something you might want to consider when judging which method learns faster.
\r\nCheers, Rasmus.
", "votes": 1, "replies": [] }, { "id": 16681, "author_name": "Matt Hagy", "author_url": "", "post_date": "2012-11-07T23:21:15.780000", "content": "Thanks for the info and the reference!
\r\nI believe a similar effect is at work in SdA where a random fraction of inputs in each layer are corrupted (i.e. set to zero) in each training batch. The following figure displays the validation error vs. fine-tuning epoch for a few SdA architectures.\r\n
\r\n
For the larger mini-batches - more averaging of training entries for computing loss gradients - validation error is close to monotonically decreasing. Although smaller batches converge faster so I'm leaning towards those.
", "votes": 1, "replies": [] }, { "id": 16679, "author_name": "rasmusbergpalm", "author_url": "", "post_date": "2012-11-07T21:21:33.587000", "content": "Basically you set the output of neurons in hidden layers to zero, with 50% probability, and otherwise train it as you normally would, i.e. SGD. This forces the NN to be very robust and redundant. It can also be seen as a very effecient way of doing modelling\r\n averaging. Take a look at the impressive paper from G.H. http://arxiv.org/abs/1207.0580
\r\nI wanted to verify the method so i trained a 784-800-800-10 FFNN with SGD and dropout for 1000 epochs (took around 11 hours on my laptop) i.e. a huge network trained for a loong time: should lead to overfitting. After each epoch i recorded the validation\r\n error and plotted it. See attached plot.
\r\nIt's really incredible that the validation error is decreasing with no sign of overfitting.
\r\n\r\n
\r\n
\r\n
", "votes": 1, "replies": [] }, { "id": 16672, "author_name": "Matt Hagy", "author_url": "", "post_date": "2012-11-07T16:43:34.277000", "content": "
Stacked Denoising Autoencoders using DeepLearning/Theano:
\r\nhttp://deeplearning.net/tutorial/SdA.html
\r\nJust adapting their SdA code to load the Kaggle training and test sets and using default parameters will get you into the top 50 on the public leader board!
\r\nTo improve on that, I've been exploring variations of the hyperparameters through bagging. For each set of hyperparameters, I train atleast 5 SdA using a 90/10 train/valid split and then look at convergence of validation error with extent of fine-tuning.\r\n This method has served me well so far, but it takes a lot of computing power. I only have 14 cores at my disposal and no GPUs so I can’t do as much exploration as I’d like. Additionally, I’m concerned the 90/10 split may be too aggressive and it may need\r\n reduced to 80/20.
\r\nI’d strongly recommend everyone checkout Theano and the DeepLearning tutorials. If you have access to GPUs, you’ll be able to try out a lot of models. Their Restricted Boltzmann Machine also looks promising. If I had the computing power, I’d be exploring\r\n these methods also.
", "votes": 1, "replies": [] }, { "id": 15406, "author_name": "ahans", "author_url": "", "post_date": "2012-10-10T19:05:35.793000", "content": "50 hidden neurons is way too small for this problem. I used 800 neurons in each of the hidden layers. Maybe less would have worked too. When using large networks and unmodified images as training data, it's important to have a validation set as well, otherwise\r\n networks tend to overfit.
", "votes": 1, "replies": [] }, { "id": 15400, "author_name": "Quantum Leap", "author_url": "", "post_date": "2012-10-10T17:14:50.670000", "content": "hi amenem, how many units are you using in your NN hidden layer? I've tried with 50 and only got 93% accuracy on the training set
", "votes": 1, "replies": [] }, { "id": 24479, "author_name": "Alec Radford", "author_url": "", "post_date": "2013-05-18T23:38:16.320000", "content": "Glad to hear you're getting results now!
\r\nIt's single threaded, I've been thinking about adding multi-threading to it, but I wanted to squeeze the performance I could get out of a single thread first. There's some libraries that offer a sort of insta multithreading for numpy (numexpr) but it relies\r\n on Intel's MKL which isn't free for commerical use. That and I'm not sure if my dropout implementation is working like it should...
", "votes": 0, "replies": [] }, { "id": 24476, "author_name": "Eugene", "author_url": "", "post_date": "2013-05-18T21:17:28.573000", "content": "[quote=Alec Radford;24407]
\r\nIs there the ability to print debug or status info as the training goes for that package to see what's going on? If the learning rate is too high it might hit an instability and just stall out if it has some internal stopping parameters it's using. Another\r\n thing is it may need normalized input between 0 and 1.
\r\nWere there any other parameters you needed to set and if so, what were they? 3 days is way too long. A similarish net with 800 hidden units written in python with numpy (so by no means fast) get's around 90% within a few minutes and near 98% within an hour\r\n or two. I was testing on MNIST itself so I don't know how it'll translate to kaggle scores though.
\r\n[/quote]
\r\nAlec, thanks a lot for valuable advices. There is indeed a way to print out debug info. It turns out that I specified threshold that is too smal and didn't specified max step parameter, so it would indefinitely try best to converge.
\r\nI wouldn't say it takes minutes, but at least I've got what exactly it is doing.
\r\nBTW, does your implementation with NumPy use all cores (if you have several)? Because R is definitely single threaded...
", "votes": 0, "replies": [] }, { "id": 24401, "author_name": "Eugene", "author_url": "", "post_date": "2013-05-16T22:23:54.613000", "content": "[quote=ahans;15406]
\r\n50 hidden neurons is way too small for this problem. I used 800 neurons in each of the hidden layers. Maybe less would have worked too. When using large networks and unmodified images as training data, it's important to have a validation set as well, otherwise\r\n networks tend to overfit.
\r\n[/quote]
\r\nAhans, could you share some light on the problem?
\r\nWhen you used 800 neurons how much time did it take? It seems that it takes more than 3 days for me for just 500 hidden neurons in just one hidden layer. I'm using R (cran one, not Revolution) and neuralnet package on my Mac with 16 Gb RAM. Any advice would\r\n be appreciated.
\r\nI would like to play more with NN, rather than try something else.
\r\nBut may be this kind of problem is not really meant to be solved by NN.
", "votes": 0, "replies": [] }, { "id": 18955, "author_name": "Shan", "author_url": "", "post_date": "2013-01-04T03:11:01.447000", "content": "[quote=Phillip Chilton Adkins;18811]
\r\nDid you choose your features as part of a cross-validation procedure? Is your 80% test or train? I had a different experience; my cross-val got much better when i limited the number of components. It's not true that you want to limit the number of components\r\n only if if n[/quote]
\r\n\r\n
Yes, I chose the number of features using cross-validation. 80% of the data was used to train and the remaining 20% as test.
", "votes": 0, "replies": [] }, { "id": 18921, "author_name": "gr33ndata", "author_url": "", "post_date": "2013-01-03T02:37:33.273000", "content": "I've just made my first submission, I used Nearest Centroid with no preprocessing, got 80% accuracy. I sure need to refine that a lot
", "votes": 0, "replies": [] }, { "id": 18813, "author_name": "Phillip Chilton Adkins", "author_url": "", "post_date": "2012-12-30T10:51:07.187000", "content": "That is certainly one thing i did. I'm currently getting about 95% with it in x-val, but i haven't made a submission yet. I'm going to try to make it better first.", "votes": 0, "replies": [] }, { "id": 18812, "author_name": "Piggy ", "author_url": "", "post_date": "2012-12-30T10:37:11.493000", "content": "[quote=Phillip Chilton Adkins;18811]
\r\nDid you choose your features as part of a cross-validation procedure? Is your 80% test or train? I had a different experience; my cross-val got much better when i limited the number of components. It's not true that you want to limit the number of components\r\n only if if n[/quote]
\r\n\r\n
\r\n
Yes I chose as cross validation.
\r\nYour point is interesting and makes sense. Only curiosity, what accuracy did you get on these data?
\r\nI guess you used PCA + k-NN.
Correct me if I'm wrong.
", "votes": 0, "replies": [] }, { "id": 18811, "author_name": "Phillip Chilton Adkins", "author_url": "", "post_date": "2012-12-30T10:28:50.747000", "content": "Did you choose your features as part of a cross-validation procedure? Is your 80% test or train? I had a different experience; my cross-val got much better when i limited the number of components. It's not true that you want to limit the number of components\r\n only if if n<p. depending on how noisy the data is, you might want to reduce the number of components to, say, n/50 (50 data points per component). Higher components may contain useful information but they can also contain noise. Often, looking at the components\r\n can be a good way to decide when they transition from structure to noise.", "votes": 0, "replies": [] }, { "id": 18809, "author_name": "Phillip Chilton Adkins", "author_url": "", "post_date": "2012-12-30T10:22:11.843000", "content": "When you do a PCA, the columns usually come out in order of decreasing variance. Therefore, columns with zero variance will come at the end of the list. Just take a look at the eigenvalues - any zero eigenvalues will correspond to a column with zero variance.", "votes": 0, "replies": [] }, { "id": 18808, "author_name": "Piggy ", "author_url": "", "post_date": "2012-12-30T08:49:10.713000", "content": "[quote=Phillip Chilton Adkins;18797]
\r\nColumns with zero variance should be pushed into the final eigenspace - as long as you choose anything but the final eigenspace when doing the PCA truncation, those columns will be completely ignored.
\r\n[/quote]
\r\n\r\n
Cool. So you first reorder columns according to their variance, I guess, and work on a slightly reduced matrix.
", "votes": 0, "replies": [] }, { "id": 18804, "author_name": "Shan", "author_url": "", "post_date": "2012-12-30T06:01:03.707000", "content": "I tried to run a PCA and then a random forest for classification. The number of principle components was chosen by a cross validation method. I trained a random forest on 80% of the data and tested it on the remaining 20%. The in-sample Out of Bag Prediction\r\n Rates and the test prediction rates are plotted below for various numbers of Priniple Components. Somewhere between 40 and 50 Principle Components give the best prediction rate of about a 95.3%.
", "votes": 0, "replies": [] }, { "id": 18797, "author_name": "Phillip Chilton Adkins", "author_url": "", "post_date": "2012-12-29T21:40:31.527000", "content": "Columns with zero variance should be pushed into the final eigenspace - as long as you choose anything but the final eigenspace when doing the PCA truncation, those columns will be completely ignored.
", "votes": 0, "replies": [] }, { "id": 18530, "author_name": "Piggy ", "author_url": "", "post_date": "2012-12-24T08:23:14.437000", "content": "Sean,how did you deal with columns with zero variance?
", "votes": 0, "replies": [] }, { "id": 17992, "author_name": "razgon", "author_url": "", "post_date": "2012-12-17T08:14:09.457000", "content": "I have used SVM too, it could perform well..
", "votes": 0, "replies": [] }, { "id": 17951, "author_name": "Piggy ", "author_url": "", "post_date": "2012-12-16T19:51:31.920000", "content": "eh curious... i was trying the other way... SVM and then a classification method like RF.
\r\nSVM with a radial basis function can be squeezed up to a 97% of accuracy... not bad. I'm struggling a bit for that 3% :)
", "votes": 0, "replies": [] }, { "id": 17740, "author_name": "Ivan Yurchenko", "author_url": "", "post_date": "2012-12-12T18:37:23.567000", "content": "Friends, I have another question on PCA. What data set do you use to fit it? Simply whole train dataset? Or some subset of it?
", "votes": 0, "replies": [] }, { "id": 17603, "author_name": "Ivan Yurchenko", "author_url": "", "post_date": "2012-12-10T19:17:23.773000", "content": "To Sean:
\r\nYeah. I just used PCA library in a wrong way :)
", "votes": 0, "replies": [] }, { "id": 17573, "author_name": "Sean", "author_url": "", "post_date": "2012-12-09T21:11:49.823000", "content": "2 Ivan - pca took maybe 1 minute on my 8gb windows 7 (64bit) laptop
", "votes": 0, "replies": [] }, { "id": 17568, "author_name": "Ivan Yurchenko", "author_url": "", "post_date": "2012-12-09T19:18:45.600000", "content": "\r\n>> I used PCA to extract features retaining 90% of the variance (goes from 784 to about 80 features) and then used 1-nearest-neighbor.
\r\nHow long did it take you to apply PCA on train data?
", "votes": 0, "replies": [] }, { "id": 16812, "author_name": "M. Takagiwa", "author_url": "", "post_date": "2012-11-12T10:34:22.950000", "content": "I used dense SIFT and AOSO-Logitboost[1], which work pretty good.
\r\n[1]: http://icml.cc/discuss/2012/569.html
", "votes": 0, "replies": [] }, { "id": 16747, "author_name": "", "author_url": "", "post_date": "2012-11-09T15:37:53.580000", "content": "", "votes": 0, "replies": [] }, { "id": 16678, "author_name": "", "author_url": "", "post_date": "2012-11-07T20:45:32.120000", "content": "", "votes": 0, "replies": [] }, { "id": 16419, "author_name": "", "author_url": "", "post_date": "2012-10-31T22:16:24.743000", "content": "", "votes": 0, "replies": [] }, { "id": 15417, "author_name": "", "author_url": "", "post_date": "2012-10-11T00:44:54.510000", "content": "", "votes": 0, "replies": [] }, { "id": 15334, "author_name": "", "author_url": "", "post_date": "2012-10-09T08:43:58.110000", "content": "", "votes": 0, "replies": [] }, { "id": 15302, "author_name": "", "author_url": "", "post_date": "2012-10-08T20:01:42.803000", "content": "", "votes": 0, "replies": [] }, { "id": 13641, "author_name": "", "author_url": "", "post_date": "2012-08-30T12:18:21.610000", "content": "", "votes": 0, "replies": [] }, { "id": 13539, "author_name": "", "author_url": "", "post_date": "2012-08-27T23:34:14.357000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "15712": "", "16676": "", "24407": "", "18807": "", "16716": "", "16681": "", "16679": "", "16672": "", "15406": "", "15400": "", "24479": "", "24476": "", "24401": "", "18955": "", "18921": "", "18813": "", "18812": "", "18811": "", "18809": "", "18808": "", "18804": "", "18797": "", "18530": "", "17992": "", "17951": "", "17740": "", "17603": "", "17573": "", "17568": "", "16812": "", "16747": "", "16678": "", "16419": "", "15417": "", "15334": "", "15302": "", "13641": "", "13539": "", "13062": "" } }