I'm hoping this can be a place to discuss SVM details. What I have done so far: third degree polynomial kernel, using default parameter values of the LIBSVM library; predictions are made using a voting strategy of the 45 trained binary classifiers. At the\r\n moment, I'm part of a six-way tie for 25th place, so I expect some of those entries are using the same method. I'm curious to know what kernels others are using, as well as the other SVM parameters. Soon I hope to have a gaussian kernel up and running, but\r\n have been having strange difficulties with this...
\r\nIf there's interest in getting started with SVM, I don't mind linking to the code I've written. My implementation uses python and LIBSVM, allowing for this task to be tackled fairly quickly.
", "messages": [ { "id": 19083, "postDate": "2013-01-08T04:58:39.990Z", "content": "I realize 4th degree was a guess, but still you had to provide parameters for your 4th degree polynomial, did you guess at those to? Just curious. As for caret, here is some helpful info to help you get started.
\r\nWith caret you will have setup a trainControl, it will look something like this:
\r\nfor example I have repeats commented out, but I could have selected a different method such as repeatedcv.
\r\nthen you have your model. When you first do your model you will use the tuneLength generally:
\r\nrbfSVM method="svmRadial",
\r\n## center and scale
\r\n# preProc = c("center", "scale"),
\r\n## Length of default tuning parameter grid
\r\ntuneLength = 5,
\r\n# or
\r\n# tuneGrid = expand.grid(.sigma=c(0.0118),.C=c(8,16,32,64,128)),
\r\n# tuneGrid = expand.grid(.sigma=c(0.0118),.C=c(64)),
\r\n## train control
\r\ntrControl=MyTrainControl,
\r\n# fit for Kappa or Accuracy
\r\n# metric = "Kappa",
\r\n# Pass arguments to ksvm
\r\nfit = FALSE
\r\n)
So tuneLength says build a grid with 5 sigma parameters and 5 costs and it tries all combinations and will report back Accuracy. It will use sigest() and come up with a very good sigma to use. So then based on the output, you will run your model again but\r\n this time instead of tuneLength you will use tuneGrid, likely with a fixed sigma value and perhaps a range of costs. Once you have narrowed down sigma and Cost, then you can pass them into tuneGrid as single values as I do above in one of my commented out\r\n lines.
\r\nObviously my model has its own particulars but here is some partial code because if your like me, its easier to understand things when you see some code:
\r\n\r\n
** Note: do NOT use any multi-core with the GUI interface, it will not work right, if you wish to use multi-core, which is GREAT for gridSearch by the way, use it from command line only.
\r\nI also switched from e1071 to caret, so if you run into any troubles making your stuff work on caret I may be able to help if you post in the public forum.
\r\nI'm hoping this can be a place to discuss SVM details. What I have done so far: third degree polynomial kernel, using default parameter values of the LIBSVM library; predictions are made using a voting strategy of the 45 trained binary classifiers. At the\r\n moment, I'm part of a six-way tie for 25th place, so I expect some of those entries are using the same method. I'm curious to know what kernels others are using, as well as the other SVM parameters. Soon I hope to have a gaussian kernel up and running, but\r\n have been having strange difficulties with this...
\r\nIf there's interest in getting started with SVM, I don't mind linking to the code I've written. My implementation uses python and LIBSVM, allowing for this task to be tackled fairly quickly.
", "votes": 3 }, { "id": 26287, "postDate": "2013-06-27T09:51:18.997Z", "content": "[quote=Artem Yankov;25613]
\r\nI have been running SVM with linear kernel for 8 hours already and not finished yet :) Using scikit-learn python package. Is it supposed to be that slow?
\r\n[/quote]
\r\nI don't think so. I'm using sci-kit with a Gaussian Kernel and I could cross validate a few runs in around 90 minutes (YMMV). I suggest that you try it on a smaller training data set first to check first.
\r\nBy the way, SVM is slow in general. But this dataset is not large either, should be manageable by sci-kit without much issues.
", "votes": 1 }, { "id": 19101, "postDate": "2013-01-08T18:41:19.837Z", "content": "Sorry about the guessing on 4th degree polynomial comment. I realized what you were asking a few minutes later and edited my response. I just went with the defaults because I didn't know what other values were reasonable.\r\n
\r\nFor nearZeroVar: it turns out that you can base that on just the training set. If you run it on the test set you get almost the same list of pixels. The models are able to make predictions based on those fields. I did something like this:
\r\ntrain2 <- train[,-1]
\r\nnzv <- nearZeroVar(train2)
\r\ntrain2 <- train2[,-nzv]
\r\ntest2 <- test[-nzv]
label <- factor(train[,1])
\r\nThank you for the code. I will try to use this. It takes my notebook computer about 5-6 hours to fit the svm model with 5-fold cross validation so I expect tuning to take a couple of days if I try several values for the cost and gamma parameters. I will\r\n try using a random sample to speed this up.
", "votes": 1 }, { "id": 19081, "postDate": "2013-01-08T03:55:38.797Z", "content": "What method did you use to get your parameters for your 4th degree polynomial?
\r\nA common way to get hyperparameters is using a grid search. Packages such as caret can do this for you. The syntax is a bit different than e1071, but once you get accustomed to caret, you will not look back! Also, caret works with many models, not just SVM's\r\n so you can use the caret framework with those as well.
\r\ncaret uses kernlab for SVM, and kernlab has a sigest() function which estimates the sigma hyper parameter for radial kernels. This works very well. Of course if you wish to do a gridsearch or manually provide hyper parameters you can. Once you use caret,\r\n and perform a grid search you will get the idea behind it and realize its a very good way to get hyper parameter values.
\r\nThat said, in the most accurate models known, the 4th degree polynomial I believe has outperformed radial kernels. The 5th degree has outperformed the 4th and the 9th degree has outperformed both.
\r\nhttp://yann.lecun.com/exdb/mnist/
", "votes": 1 }, { "id": 69288, "postDate": "2015-04-01T09:08:50.860Z", "content": "I am new to SVM. I am making a project which involves predicting exchange rate volatility. I am using scikit and have a confusion that how can I use more than 2 features for training SVM and what will x and y axis represent if I plot various kernels.
\nThank you.
" }, { "id": 58127, "postDate": "2014-11-16T06:54:09.753Z", "content": "I used 2nd degree polynomial kernel. My accuracy is 0.97886
\nI chose second degree after some grid search over different parameters. In the forum many people used 3/4th degree polynomial. How did you chose them and what is the accuracy.
\nI eliminated 3/4th degree after grid search based analysis on a smaller data set.
\nAlso, I wanted to understand why polynomial kernel is working better than RBF for digit recognition. Is there a scientific reason behind it?
" }, { "id": 39199, "postDate": "2014-02-21T14:07:58.003Z", "content": "Hi,
\nWhen I load caret, following code helps me to tune the parameters and also predict using SVM but I am not sure which SVM package did it really use. http://www.jstatsoft.org/v28/i05/paper says it looks like kernlab which I did not load in r. In addition, Kernlab SVM code usuallly uses ksvm command. I am now confused if it is really ksvm pakage? Would you please help me figure this out ASAP?
\nfitControl=trainControl(method="cv",number=10)
set.seed(3)
svmGrid=expand.grid(.C=c(1,10,10000), .sigma=(1:5)/10^5)
svmFit=train(train.x,train.y,"svmRadial",trControl=fitControl,tuneGrid=svmGrid)
\net.train.svm=predict(svmFit,newdata=train.x)
\net.test.svm=predict(svmFit,newdata=test.x)
\n\n
Thank you,
" }, { "id": 26288, "postDate": "2013-06-27T09:54:33.583Z", "content": "[quote=Deepak Chaturvedi;26065]
\r\nHi There,
\r\nI am also using SVM using R ,kernel=polynomial,degree=4.It took more than 12 hours for me.
\r\nI am not able to figure out,where I am going wrong ,cause I am getting 0.097 as my accuracy on kaggle result.
\r\nSome help will be appreciated.
\r\n[/quote]
\r\nWhat are your results on training with the public data set (train.csv)? Could it be you are overfitting (that you're getting very good classification results via the public data set, but very poor when submitted to Kaggle)?
\r\nOne other possibility could be you have a poor selection of C and gamma, too. I've tested SVM and it CAN get very poor with a big gamma, etc... you have to experiment first on a small dataset, you'll be surprised.
\r\nTry to use a smaller dataset to verify your code first. I use only 2000 instances to validate.
" }, { "id": 26286, "postDate": "2013-06-27T09:48:31.067Z", "content": "[quote=Luis_Jaraquemada;25601]
\r\nAnybody using Gaussian Kernel in this problem??
\r\nHave you conducted any feature selection? or dimensionality reduction?
\r\nthanks,
\r\n[/quote]
\r\nI am using a Gaussian Kernel without any feature selection of dimensionality reduction, but with limited success at a score of 0.97643. I will try dimensionality reduction if I have time.
" }, { "id": 26065, "postDate": "2013-06-21T02:12:32.147Z", "content": "Hi There,
\r\nI am also using SVM using R ,kernel=polynomial,degree=4.It took more than 12 hours for me.
\r\nI am not able to figure out,where I am going wrong ,cause I am getting 0.097 as my accuracy on kaggle result.
\r\nSome help will be appreciated.
" }, { "id": 25613, "postDate": "2013-06-14T14:55:51.900Z", "content": "I have been running SVM with linear kernel for 8 hours already and not finished yet :) Using scikit-learn python package. Is it supposed to be that slow?
" }, { "id": 25601, "postDate": "2013-06-14T08:47:42.993Z", "content": "Anybody using Gaussian Kernel in this problem??
\r\nHave you conducted any feature selection? or dimensionality reduction?
\r\nthanks,
" }, { "id": 24838, "postDate": "2013-05-27T13:47:53.483Z", "content": "Guys, I am new to this and have never done any coding in R. I am trying to learn this. I want to solve this problem using SVM but not sure how to start. Could someone provide me help with code. Thank you, Anurag
" }, { "id": 19084, "postDate": "2013-01-08T05:18:51.857Z", "content": "also just curious about how you are removing your near zero variance fields. Are you removing the fields from both training and test data? something along the lines of:
\r\nnzvCols <- nearZeroVar(train)
\r\ntrain <- train[, -nzvCols]
I assume you would have combined both train/test, then ran nearZeroVar, and then separated them back out? Because you couldn't remove a column in train or test unless you did it in both, as SVM would trip up on the different number of columns.
" }, { "id": 19082, "postDate": "2013-01-08T04:43:08.733Z", "content": "[quote=Brian Feeny;19081]
\r\nWhat method did you use to get your parameters for your 4th degree polynomial?
\r\nA common way to get hyperparameters is using a grid search. Packages such as caret can do this for you. The syntax is a bit different than e1071, but once you get accustomed to caret, you will not look back! Also, caret works with many models, not just SVM's\r\n so you can use the caret framework with those as well.
\r\ncaret uses kernlab for SVM, and kernlab has a sigest() function which estimates the sigma hyper parameter for radial kernels. This works very well. Of course if you wish to do a gridsearch or manually provide hyper parameters you can. Once you use caret,\r\n and perform a grid search you will get the idea behind it and realize its a very good way to get hyper parameter values.
\r\nThat said, in the most accurate models known, the 4th degree polynomial I believe has outperformed radial kernels. The 5th degree has outperformed the 4th and the 9th degree has outperformed both.
\r\nhttp://yann.lecun.com/exdb/mnist/
\r\n[/quote]
\r\nThank you. I will try caret. I am already using it for preprocessing tasks like removing near zero variance fields. I have been reluctant to use the grid search because I found the documentation a little confusing but now I am motivated to try this.
\r\nFor the 4th degree kernel, I just used the default values. Same for the others. Today I tried using cost=1000 with the radial kernel and got a small improvement over my previous best model. Need to try this with 4th degree tomorrow.
" }, { "id": 19080, "postDate": "2013-01-08T03:24:10.813Z", "content": "Can anyone give advice on how to tune parameters for SVM? I am using package e1071 in R. I got the best results (just over 98%) with 4 degree polynomial. Ialso tried 3-degree, radial and sigmoid. I think the radial should work best if I can get the parameters\r\n right but I have no intuition about what to try. Links to literature would be great.\r\n
\r\nAnother question. I have tried PCA and center/scale preprocessing but these do not improve the results. So far, just eliminating near zero variance fields works best. Are there other preprocessing techniques worth trying?
" }, { "id": 17049, "postDate": "2012-11-21T05:02:27.063Z", "content": "I am using SVM as well. Currently I am using a radial kernel as that has worked best for me in testing. I will later explore the polynomial kernels. Tuning takes a long time, I have a Quad Core 2.66Ghz Mac Pro, using single threaded R, and there really is\r\n nothing straight forward to do in multi-threading this. doMC() and others don't seem to help much there.
\r\nI plan to use a strict test regimin, likely 10-fold cross validation and really try to fine tune my parameters.
\r\nI am also considering doing some preprocessing on the data.
" }, { "id": 14111, "postDate": "2012-09-08T22:41:38.650Z", "content": "My implementation is single-threaded. It's possible there is some hidden parallel work being done, but looking at a resource monitor shows only one of my four cores being used.
\r\nHowever, I have noticed that it doesn't take very many changes to significantly change behavior. An example I've noticed is that scaling the data to [0,1] takes a bit more time (about 30 minutes instead of 5). I'm not positive, but it seems like scaling\r\n the data in this way results in many more support vectors being used for classification; this is judged by looking at the saved model files spit out by LIBSVM, and how the model files require much more disk space when trained on scaled data (guessing that\r\n the extra disk space is for the extra support vectors?).
" }, { "id": 14084, "postDate": "2012-09-07T23:04:36.860Z", "content": "A few minutes? that's still is pretty fast. Does your implementation take advantage of muticore/parallel processing or is that build into the python library you are using?
" }, { "id": 14074, "postDate": "2012-09-07T19:00:56.417Z", "content": "I was probably a little too loose in what I considered "fairly quickly", because I haven't yet worked on tuning parameters. "Fairly quicky" for me is how long a single cross-validation step takes:
\r\n1) Training the 45 binary classifiers on 80% of the training data: 2-3 minutes
\r\n2) Predicting labels on the remaining 20% of the data using the models from step 1): 2-3 minutes
\r\nSo everything can be done in about five minutes. But I can see that maybe this is way too slow for grid search.
" }, { "id": 14061, "postDate": "2012-09-07T15:42:31.040Z", "content": "Hey Patrick, I am using the svm package libsvm also with R package e1071. When you say "fairly quickly" how long do you mean? I perfomed a grid tuning for svm and after a day of computation on my 6-core machine, it still hasn't completed. Although I have\r\n to say I am using a four degree polynomial with the whole dataset and no feature selection...
" }, { "id": 14046, "postDate": "2012-09-07T07:35:23.717Z", "content": "I don't have the best reasons :)
\r\nI chose the polynomial kernel because of the Cortest Vapnik (1995) paper, which uses a fourth degree polynomial on the MNIST data. I instead used a third degree polynomial because it performed better during cross-validation of the training data. I haven't\r\n yet bothered with parameter tuning, so it's possible third is not the best degree for the problem.
" }, { "id": 14043, "postDate": "2012-09-07T05:44:46.743Z", "content": "Hi Patrick,
\r\nhow did you choose the degree of the polynomial kernel and why this kernel?
\r\nThank you,
\r\nIldefons
I realize 4th degree was a guess, but still you had to provide parameters for your 4th degree polynomial, did you guess at those to? Just curious. As for caret, here is some helpful info to help you get started.
\r\nWith caret you will have setup a trainControl, it will look something like this:
\r\nfor example I have repeats commented out, but I could have selected a different method such as repeatedcv.
\r\nthen you have your model. When you first do your model you will use the tuneLength generally:
\r\nrbfSVM method="svmRadial",
\r\n## center and scale
\r\n# preProc = c("center", "scale"),
\r\n## Length of default tuning parameter grid
\r\ntuneLength = 5,
\r\n# or
\r\n# tuneGrid = expand.grid(.sigma=c(0.0118),.C=c(8,16,32,64,128)),
\r\n# tuneGrid = expand.grid(.sigma=c(0.0118),.C=c(64)),
\r\n## train control
\r\ntrControl=MyTrainControl,
\r\n# fit for Kappa or Accuracy
\r\n# metric = "Kappa",
\r\n# Pass arguments to ksvm
\r\nfit = FALSE
\r\n)
So tuneLength says build a grid with 5 sigma parameters and 5 costs and it tries all combinations and will report back Accuracy. It will use sigest() and come up with a very good sigma to use. So then based on the output, you will run your model again but\r\n this time instead of tuneLength you will use tuneGrid, likely with a fixed sigma value and perhaps a range of costs. Once you have narrowed down sigma and Cost, then you can pass them into tuneGrid as single values as I do above in one of my commented out\r\n lines.
\r\nObviously my model has its own particulars but here is some partial code because if your like me, its easier to understand things when you see some code:
\r\n\r\n
** Note: do NOT use any multi-core with the GUI interface, it will not work right, if you wish to use multi-core, which is GREAT for gridSearch by the way, use it from command line only.
\r\nI also switched from e1071 to caret, so if you run into any troubles making your stuff work on caret I may be able to help if you post in the public forum.
\r\n[quote=Artem Yankov;25613]
\r\nI have been running SVM with linear kernel for 8 hours already and not finished yet :) Using scikit-learn python package. Is it supposed to be that slow?
\r\n[/quote]
\r\nI don't think so. I'm using sci-kit with a Gaussian Kernel and I could cross validate a few runs in around 90 minutes (YMMV). I suggest that you try it on a smaller training data set first to check first.
\r\nBy the way, SVM is slow in general. But this dataset is not large either, should be manageable by sci-kit without much issues.
", "votes": 1, "replies": [] }, { "id": 19101, "author_name": "AlKhwarizmi", "author_url": "", "post_date": "2013-01-08T18:41:19.837000", "content": "Sorry about the guessing on 4th degree polynomial comment. I realized what you were asking a few minutes later and edited my response. I just went with the defaults because I didn't know what other values were reasonable.\r\n
\r\nFor nearZeroVar: it turns out that you can base that on just the training set. If you run it on the test set you get almost the same list of pixels. The models are able to make predictions based on those fields. I did something like this:
\r\ntrain2 <- train[,-1]
\r\nnzv <- nearZeroVar(train2)
\r\ntrain2 <- train2[,-nzv]
\r\ntest2 <- test[-nzv]
label <- factor(train[,1])
\r\nThank you for the code. I will try to use this. It takes my notebook computer about 5-6 hours to fit the svm model with 5-fold cross validation so I expect tuning to take a couple of days if I try several values for the cost and gamma parameters. I will\r\n try using a random sample to speed this up.
", "votes": 1, "replies": [] }, { "id": 19081, "author_name": "Signal", "author_url": "", "post_date": "2013-01-08T03:55:38.797000", "content": "What method did you use to get your parameters for your 4th degree polynomial?
\r\nA common way to get hyperparameters is using a grid search. Packages such as caret can do this for you. The syntax is a bit different than e1071, but once you get accustomed to caret, you will not look back! Also, caret works with many models, not just SVM's\r\n so you can use the caret framework with those as well.
\r\ncaret uses kernlab for SVM, and kernlab has a sigest() function which estimates the sigma hyper parameter for radial kernels. This works very well. Of course if you wish to do a gridsearch or manually provide hyper parameters you can. Once you use caret,\r\n and perform a grid search you will get the idea behind it and realize its a very good way to get hyper parameter values.
\r\nThat said, in the most accurate models known, the 4th degree polynomial I believe has outperformed radial kernels. The 5th degree has outperformed the 4th and the 9th degree has outperformed both.
\r\nhttp://yann.lecun.com/exdb/mnist/
", "votes": 1, "replies": [] }, { "id": 69288, "author_name": "Neel Shah", "author_url": "", "post_date": "2015-04-01T09:08:50.860000", "content": "I am new to SVM. I am making a project which involves predicting exchange rate volatility. I am using scikit and have a confusion that how can I use more than 2 features for training SVM and what will x and y axis represent if I plot various kernels.
\nThank you.
", "votes": 0, "replies": [] }, { "id": 58127, "author_name": "Sourabh", "author_url": "", "post_date": "2014-11-16T06:54:09.753000", "content": "I used 2nd degree polynomial kernel. My accuracy is 0.97886
\nI chose second degree after some grid search over different parameters. In the forum many people used 3/4th degree polynomial. How did you chose them and what is the accuracy.
\nI eliminated 3/4th degree after grid search based analysis on a smaller data set.
\nAlso, I wanted to understand why polynomial kernel is working better than RBF for digit recognition. Is there a scientific reason behind it?
", "votes": 0, "replies": [] }, { "id": 39199, "author_name": "SVM help", "author_url": "", "post_date": "2014-02-21T14:07:58.003000", "content": "Hi,
\nWhen I load caret, following code helps me to tune the parameters and also predict using SVM but I am not sure which SVM package did it really use. http://www.jstatsoft.org/v28/i05/paper says it looks like kernlab which I did not load in r. In addition, Kernlab SVM code usuallly uses ksvm command. I am now confused if it is really ksvm pakage? Would you please help me figure this out ASAP?
\nfitControl=trainControl(method="cv",number=10)
set.seed(3)
svmGrid=expand.grid(.C=c(1,10,10000), .sigma=(1:5)/10^5)
svmFit=train(train.x,train.y,"svmRadial",trControl=fitControl,tuneGrid=svmGrid)
\net.train.svm=predict(svmFit,newdata=train.x)
\net.test.svm=predict(svmFit,newdata=test.x)
\n\n
Thank you,
", "votes": 0, "replies": [] }, { "id": 26288, "author_name": "Log0", "author_url": "", "post_date": "2013-06-27T09:54:33.583000", "content": "[quote=Deepak Chaturvedi;26065]
\r\nHi There,
\r\nI am also using SVM using R ,kernel=polynomial,degree=4.It took more than 12 hours for me.
\r\nI am not able to figure out,where I am going wrong ,cause I am getting 0.097 as my accuracy on kaggle result.
\r\nSome help will be appreciated.
\r\n[/quote]
\r\nWhat are your results on training with the public data set (train.csv)? Could it be you are overfitting (that you're getting very good classification results via the public data set, but very poor when submitted to Kaggle)?
\r\nOne other possibility could be you have a poor selection of C and gamma, too. I've tested SVM and it CAN get very poor with a big gamma, etc... you have to experiment first on a small dataset, you'll be surprised.
\r\nTry to use a smaller dataset to verify your code first. I use only 2000 instances to validate.
", "votes": 0, "replies": [] }, { "id": 26286, "author_name": "Log0", "author_url": "", "post_date": "2013-06-27T09:48:31.067000", "content": "[quote=Luis_Jaraquemada;25601]
\r\nAnybody using Gaussian Kernel in this problem??
\r\nHave you conducted any feature selection? or dimensionality reduction?
\r\nthanks,
\r\n[/quote]
\r\nI am using a Gaussian Kernel without any feature selection of dimensionality reduction, but with limited success at a score of 0.97643. I will try dimensionality reduction if I have time.
", "votes": 0, "replies": [] }, { "id": 26065, "author_name": "Deepak Chaturvedi", "author_url": "", "post_date": "2013-06-21T02:12:32.147000", "content": "Hi There,
\r\nI am also using SVM using R ,kernel=polynomial,degree=4.It took more than 12 hours for me.
\r\nI am not able to figure out,where I am going wrong ,cause I am getting 0.097 as my accuracy on kaggle result.
\r\nSome help will be appreciated.
", "votes": 0, "replies": [] }, { "id": 25613, "author_name": "Artem Yankov", "author_url": "", "post_date": "2013-06-14T14:55:51.900000", "content": "I have been running SVM with linear kernel for 8 hours already and not finished yet :) Using scikit-learn python package. Is it supposed to be that slow?
", "votes": 0, "replies": [] }, { "id": 25601, "author_name": "Luis_Jaraquemada", "author_url": "", "post_date": "2013-06-14T08:47:42.993000", "content": "Anybody using Gaussian Kernel in this problem??
\r\nHave you conducted any feature selection? or dimensionality reduction?
\r\nthanks,
", "votes": 0, "replies": [] }, { "id": 24838, "author_name": "anurag nayak", "author_url": "", "post_date": "2013-05-27T13:47:53.483000", "content": "Guys, I am new to this and have never done any coding in R. I am trying to learn this. I want to solve this problem using SVM but not sure how to start. Could someone provide me help with code. Thank you, Anurag
", "votes": 0, "replies": [] }, { "id": 19084, "author_name": "Signal", "author_url": "", "post_date": "2013-01-08T05:18:51.857000", "content": "also just curious about how you are removing your near zero variance fields. Are you removing the fields from both training and test data? something along the lines of:
\r\nnzvCols <- nearZeroVar(train)
\r\ntrain <- train[, -nzvCols]
I assume you would have combined both train/test, then ran nearZeroVar, and then separated them back out? Because you couldn't remove a column in train or test unless you did it in both, as SVM would trip up on the different number of columns.
", "votes": 0, "replies": [] }, { "id": 19082, "author_name": "AlKhwarizmi", "author_url": "", "post_date": "2013-01-08T04:43:08.733000", "content": "[quote=Brian Feeny;19081]
\r\nWhat method did you use to get your parameters for your 4th degree polynomial?
\r\nA common way to get hyperparameters is using a grid search. Packages such as caret can do this for you. The syntax is a bit different than e1071, but once you get accustomed to caret, you will not look back! Also, caret works with many models, not just SVM's\r\n so you can use the caret framework with those as well.
\r\ncaret uses kernlab for SVM, and kernlab has a sigest() function which estimates the sigma hyper parameter for radial kernels. This works very well. Of course if you wish to do a gridsearch or manually provide hyper parameters you can. Once you use caret,\r\n and perform a grid search you will get the idea behind it and realize its a very good way to get hyper parameter values.
\r\nThat said, in the most accurate models known, the 4th degree polynomial I believe has outperformed radial kernels. The 5th degree has outperformed the 4th and the 9th degree has outperformed both.
\r\nhttp://yann.lecun.com/exdb/mnist/
\r\n[/quote]
\r\nThank you. I will try caret. I am already using it for preprocessing tasks like removing near zero variance fields. I have been reluctant to use the grid search because I found the documentation a little confusing but now I am motivated to try this.
\r\nFor the 4th degree kernel, I just used the default values. Same for the others. Today I tried using cost=1000 with the radial kernel and got a small improvement over my previous best model. Need to try this with 4th degree tomorrow.
", "votes": 0, "replies": [] }, { "id": 19080, "author_name": "AlKhwarizmi", "author_url": "", "post_date": "2013-01-08T03:24:10.813000", "content": "Can anyone give advice on how to tune parameters for SVM? I am using package e1071 in R. I got the best results (just over 98%) with 4 degree polynomial. Ialso tried 3-degree, radial and sigmoid. I think the radial should work best if I can get the parameters\r\n right but I have no intuition about what to try. Links to literature would be great.\r\n
\r\nAnother question. I have tried PCA and center/scale preprocessing but these do not improve the results. So far, just eliminating near zero variance fields works best. Are there other preprocessing techniques worth trying?
", "votes": 0, "replies": [] }, { "id": 17049, "author_name": "Signal", "author_url": "", "post_date": "2012-11-21T05:02:27.063000", "content": "I am using SVM as well. Currently I am using a radial kernel as that has worked best for me in testing. I will later explore the polynomial kernels. Tuning takes a long time, I have a Quad Core 2.66Ghz Mac Pro, using single threaded R, and there really is\r\n nothing straight forward to do in multi-threading this. doMC() and others don't seem to help much there.
\r\nI plan to use a strict test regimin, likely 10-fold cross validation and really try to fine tune my parameters.
\r\nI am also considering doing some preprocessing on the data.
", "votes": 0, "replies": [] }, { "id": 14111, "author_name": "Patrick Mason", "author_url": "", "post_date": "2012-09-08T22:41:38.650000", "content": "My implementation is single-threaded. It's possible there is some hidden parallel work being done, but looking at a resource monitor shows only one of my four cores being used.
\r\nHowever, I have noticed that it doesn't take very many changes to significantly change behavior. An example I've noticed is that scaling the data to [0,1] takes a bit more time (about 30 minutes instead of 5). I'm not positive, but it seems like scaling\r\n the data in this way results in many more support vectors being used for classification; this is judged by looking at the saved model files spit out by LIBSVM, and how the model files require much more disk space when trained on scaled data (guessing that\r\n the extra disk space is for the extra support vectors?).
", "votes": 0, "replies": [] }, { "id": 14084, "author_name": "seylom ayivi", "author_url": "", "post_date": "2012-09-07T23:04:36.860000", "content": "A few minutes? that's still is pretty fast. Does your implementation take advantage of muticore/parallel processing or is that build into the python library you are using?
", "votes": 0, "replies": [] }, { "id": 14074, "author_name": "Patrick Mason", "author_url": "", "post_date": "2012-09-07T19:00:56.417000", "content": "I was probably a little too loose in what I considered "fairly quickly", because I haven't yet worked on tuning parameters. "Fairly quicky" for me is how long a single cross-validation step takes:
\r\n1) Training the 45 binary classifiers on 80% of the training data: 2-3 minutes
\r\n2) Predicting labels on the remaining 20% of the data using the models from step 1): 2-3 minutes
\r\nSo everything can be done in about five minutes. But I can see that maybe this is way too slow for grid search.
", "votes": 0, "replies": [] }, { "id": 14061, "author_name": "seylom ayivi", "author_url": "", "post_date": "2012-09-07T15:42:31.040000", "content": "Hey Patrick, I am using the svm package libsvm also with R package e1071. When you say "fairly quickly" how long do you mean? I perfomed a grid tuning for svm and after a day of computation on my 6-core machine, it still hasn't completed. Although I have\r\n to say I am using a four degree polynomial with the whole dataset and no feature selection...
", "votes": 0, "replies": [] }, { "id": 14046, "author_name": "Patrick Mason", "author_url": "", "post_date": "2012-09-07T07:35:23.717000", "content": "I don't have the best reasons :)
\r\nI chose the polynomial kernel because of the Cortest Vapnik (1995) paper, which uses a fourth degree polynomial on the MNIST data. I instead used a third degree polynomial because it performed better during cross-validation of the training data. I haven't\r\n yet bothered with parameter tuning, so it's possible third is not the best degree for the problem.
", "votes": 0, "replies": [] }, { "id": 14043, "author_name": "Ildefons Magrans", "author_url": "", "post_date": "2012-09-07T05:44:46.743000", "content": "Hi Patrick,
\r\nhow did you choose the degree of the polynomial kernel and why this kernel?
\r\nThank you,
\r\nIldefons