{
  "id": 11195,
  "title": "Do SVM algorithms fit here?",
  "url": "/competitions/avazu-ctr-prediction/discussion/11195",
  "author_name": "",
  "post_date": "2014-12-10T10:36:01.623Z",
  "votes": null,
  "comment_count": 25,
  "views": 9221,
  "content": "<p>All,&nbsp;</p>\n<p>This is my first competition and real data project. I have been using SVM for toy project and wanted to use a similar approach to propose something.&nbsp;</p>\n<p>However I found out that the feature vectors might ended up being very long if I want to use for example the hash-trick method. For example each site_id will become a feature. I think that this is not realistic. &nbsp;</p>\n<p>My first run was to select some samples (1 000 000 = NumSamples) randomly. For each each feature determine the probability of this feature ( p(val) = NumberOfApperance(val)/ NumSamples) and reconstruct the feature vector replacing the each feature (val) &nbsp;by its probability (p(val)) but did not get meaningful improvement.&nbsp;</p>\n<p>Online-SVM might be a good try in case i use this hash-trick but I do not have enough experience with this yet.&nbsp;</p>\n<p>I would like to know if someone already tried the SVM or if you have &nbsp;better way to represent the feature.&nbsp;</p>\n\n<p>I will appreciate your help.&nbsp;</p>\n<p>Regards,</p>",
  "messages": [
    {
      "id": "59942",
      "postDate": "12/10/2014 10:36:01",
      "content": "<p>All,&nbsp;</p>\n<p>This is my first competition and real data project. I have been using SVM for toy project and wanted to use a similar approach to propose something.&nbsp;</p>\n<p>However I found out that the feature vectors might ended up being very long if I want to use for example the hash-trick method. For example each site_id will become a feature. I think that this is not realistic. &nbsp;</p>\n<p>My first run was to select some samples (1 000 000 = NumSamples) randomly. For each each feature determine the probability of this feature ( p(val) = NumberOfApperance(val)/ NumSamples) and reconstruct the feature vector replacing the each feature (val) &nbsp;by its probability (p(val)) but did not get meaningful improvement.&nbsp;</p>\n<p>Online-SVM might be a good try in case i use this hash-trick but I do not have enough experience with this yet.&nbsp;</p>\n<p>I would like to know if someone already tried the SVM or if you have &nbsp;better way to represent the feature.&nbsp;</p>\n\n<p>I will appreciate your help.&nbsp;</p>\n<p>Regards,</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "59949",
      "postDate": "12/10/2014 15:00:23",
      "content": "<p>Feature hashing, as described&nbsp;<a href=\"http://en.wikipedia.org/wiki/Feature_hashing\">here</a> is a good start.</p>\n<p>SVMs, in general, are memory intensive and don't scale well for high dimensional data. I suggest, start with a simple classification algorithm (logistic?) to get a baseline score and optimize from there. Also, I found the information in this post quite useful. <a href=\"http://www.kaggle.com/c/avazu-ctr-prediction/forums/t/10927/beat-the-benchmark-with-less-than-1mb-of-memory\" target=\"_blank\">beat-the-benchmark-with-less-than-1mb-of-memory</a></p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "59955",
      "postDate": "12/10/2014 16:17:24",
      "content": "<p>@logloss thank you very much.&nbsp;</p>\n<p>I will give it a try and try to build my code on it</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "59978",
      "postDate": "12/11/2014 00:27:07",
      "content": "<p>Absolutely they do. It's a little known fact, but SVM is actually highly scalable ;-) That is because it only requires to look at support vectors --you can discard the quasi-totality of the vectors in the dataset when training a SVM, as long as you keep the right ones.</p>\n<p style=\"padding-left: 30px\"><em>&quot;Everything what does SVM, it does in terms of similarity between samples.&quot;</em> --Vladimir Vapnik</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "60027",
      "postDate": "12/11/2014 17:41:06",
      "content": "<p>[quote=fchollet;59978]</p>\n<p>Absolutely they do. It's a little known fact, but SVM is actually highly scalable ;-) That is because it only requires to look at support vectors --you can discard the quasi-totality of the vectors in the dataset when training a SVM, as long as you keep the right ones.</p>\n<p style=\"padding-left: 30px\"><em>&quot;Everything what does SVM, it does in terms of similarity between samples.&quot;</em> --Vladimir Vapnik</p>\n<p>[/quote]</p>\n<p>I understand in concept that svm only looks at the data near the decision boundary. But why is it so slow in practice? I'm using sklearn's svm. Does it have something to do with kernel?&nbsp;</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "60030",
      "postDate": "12/11/2014 18:06:11",
      "content": "<p>[quote=rcarson;60027]</p>\n<p>I understand in concept that svm only looks at the data near the decision boundary. But why is it so slow in practice? I'm using sklearn's svm. Does it have something to do with kernel?&nbsp;</p>\n<p>[/quote]</p>\n<p>If someone provides a trick to speed up sklearn's svm, I've give TWO up-votes! &nbsp;:-)</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "60031",
      "postDate": "12/11/2014 18:09:44",
      "content": "<p>Use this:&nbsp;http://patternsonascreen.net/cuSVM.html</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "60032",
      "postDate": "12/11/2014 18:11:33",
      "content": "<p>[quote=rcarson;60027]</p>\n<p>I understand in concept that svm only looks at the data near the decision boundary. But why is it so slow in practice?&nbsp;</p>\n<p>[/quote]</p>\n<p>This is wrong. SVM &quot;looks&quot; for all data, but its solution is a linear combination of few support vectors (which lies near decision boundary).</p>\n<p>There is no method for fast (<strong>O(samples)</strong>) fitting SVM with custom kernel.</p>\n<p>But for linear kernel - there is. For example, <a href=\"http://www.csie.ntu.edu.tw/~cjlin/liblinear/\">LibLinear</a>&nbsp;&nbsp;(<a href=\"http://scikit-learn.org/stable/modules/generated/sklearn.svm.LinearSVC.html\">sklearn's LinearSVC</a> use this implementation)</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "60033",
      "postDate": "12/11/2014 18:20:10",
      "content": "<p>For large dataset use onehot (or hash trick) and LinearSVC. If you want to optimize logloss - build LogisticRegression upon LinearSVC.decision_function(), for example.</p>\n<p>If want to use non-linear approaches - find out how to reduce this problem to linear =)</p>\n<p>Maybe you need to do some kind of kernel mapping (as in <a href=\"http://fastml.com/the-secret-of-the-big-guys/\">Sofia-ML</a>) or 2-3-features interaction (as in Vowpal Wabbit) or some tree-based features transformation.</p>\n<p>But in the end almost every problem&nbsp;is a&nbsp;linear problem.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "60034",
      "postDate": "12/11/2014 18:26:51",
      "content": "<p>[quote=Mikhail Trofimov;60032]</p>\n<p>[quote=rcarson;60027]</p>\n<p>I understand in concept that svm only looks at the data near the decision boundary. But why is it so slow in practice?&nbsp;</p>\n<p>[/quote]</p>\n<p>This is wrong. SVM &quot;looks&quot; for all data, but its solution is a linear combination of few support vectors (which lies near decision boundary).</p>\n<p>[/quote]</p>\n<p>Thank you very much for correcting me. Mmm, this must be why I got a B.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "60035",
      "postDate": "12/11/2014 18:35:33",
      "content": "<p>[quote=Mikhail Trofimov;60033]</p>\n<p>If you want to optimize logloss - build LogisticRegression upon LinearSVC.desicion_function(), for example.</p>\n<p>[/quote]</p>\n<p>May I ask how this could be done?&nbsp;</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "60036",
      "postDate": "12/11/2014 18:42:46",
      "content": "<p>[quote=rcarson;60035]</p>\n<p>[quote=Mikhail Trofimov;60033]</p>\n<p>If you want to optimize logloss - build LogisticRegression upon LinearSVC.decision_function(), for example.</p>\n<p>[/quote]</p>\n<p>May I ask how this could be done?&nbsp;</p>\n<p>[/quote]</p>\n<p>Split your train data for 80/20. On 80% fit LinearSVC. It cannot return estimated probabilities, but you can get the distance for dicision boundry by calling .decision_function().</p>\n<p>Get this distances for holded 20% and use Logistic Regression for searching the best&nbsp;sigmoid transformation.</p>\n<p>For final predicting use similar chain of actions - get distances -&gt; sigmoid transformation -&gt; estimated probabilities</p>\n<p>It is just a one way, probably not the best.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "60039",
      "postDate": "12/11/2014 19:01:00",
      "content": "<p>The only difference between&nbsp;SVM and logistic regression (in terms of modelling, forget about training at this time)&nbsp;is their loss function. Logistic regression using log-loss, which gives your probability estimation and IS the metric&nbsp;they are using to evaluate&nbsp;your result&nbsp;in this contest.</p>\n<p>SVM, in contrast, gives your hinge loss and thus it makes no sense to get probability estimates from it.</p>\n<p>You are gonna get&nbsp;log-loss to infinity if you got the binary label wrong...which is not as&nbsp;ROBUST as when&nbsp;SVM is just being used as a classifier. Use it when you have other metrics on your result, such as precision, f1, or just accuracy.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "60063",
      "postDate": "12/12/2014 00:39:43",
      "content": "<p>After fchollet's comment I looked into the tricks people to do speed up SVM. It seems like there is a variant called CB-SVM where you hierarchically cluster the data, train the SVM, and then de-cluster the clumps that are intersected by the decision boundary. It looks like it should be N log(N) in the data - the authors do example sets up to something like 4 million data points in their <a href=\"http://dl.acm.org/citation.cfm?id=1103738\">paper</a>,&nbsp;and that was 2005, so 40 million might be manageable.</p>\n<p>They do say that it's bad at high-dimensional feature spaces though, so one-hotting is probably not the best idea.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "60069",
      "postDate": "12/12/2014 03:16:58",
      "content": "<p>And then you could hash your feature into space of small number of&nbsp;dimension.</p>\n<p>How about 2^10?</p>\n\n<p>Edit: by the way, the link you've given is broken.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "60072",
      "postDate": "12/12/2014 04:58:17",
      "content": "<p>Huh, for some reason the forums really didn't like that link. I found another link to the paper, but this one might need institutional access to get the full text. Anyhow, a google search for CB-SVM finds a bunch more stuff about it including some presentations going through it in detail.</p>\n<p>For hashing,&nbsp;it feels like if your categorical variables were truly independent then using such a small space would cause problems when e.g. you hash a variable that predicts a high average CTR into the same feature as a variable that predicts a low average CTR. If you make it really tiny, eventually I guess you'd just get every data row having all features set to '1'.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "60074",
      "postDate": "12/12/2014 05:06:11",
      "content": "<p>Your guess is not true; there are 21 variables in the beginning, so you can't have a row with more than 21 non zero elements.</p>\n<p>However, you are right that small size of hash function will cause collision problems. I am actually telling&nbsp;a joke of hashing it to only 1k size. But it might work for 1M, I don't know.</p>\n<p>Although I am not familiar with CBSVM, but based on their citation count (published 10 years ago), this method may not be very useful, plus it's hard to find a good implementation. I'd rather suggest <a href=\"http://www.csie.ntu.edu.tw/~cjlin/liblinear/\">liblinear</a>&nbsp;(which is cited 406 times since 2008), whose linear support vector classifier has been proved competitive on both performance and&nbsp;scalability.</p>\n<p>Please correct me if I am wrong.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "60077",
      "postDate": "12/12/2014 05:42:55",
      "content": "<p>I'd have to try both to tell. I guess the question is, do you get more from being able to use the kernel trick and a non-linear kernel, or from being able to train the SVM on more of the data. How important is it to retain the ability to make use of that non-linearity?</p>\n<p>There may be some 'trivial' non-linearity that is actually quite important to capture (such as the app_id/site_id bifurcation), but which you could resolve using feature engineering or by subdividing the data.</p>\n<p>I honestly haven't had much luck with SVM for this. I used feature histograms for my dimensionality reduction and pushed a bunch of sets of about 15000 data rows through Scikit-Learn's SVM implementation. After&nbsp;bagging and grid-search, I still just barely break 0.41 with that method, and going from 10000 to 15000 data rows didn't make much difference. I suppose 15000 to 40 million may still do so however, and being able to actually work with the full categorical information (or ~16 million features worth of it) should in principle be a big gain over feature histograms.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "60079",
      "postDate": "12/12/2014 05:59:50",
      "content": "<p>Given that the dimension is large already, which kernel do you think will do the job? The whole point of SVM is linear separability, and I doubt if it's really necessary to map the hot encoding into some larger space to get that.</p>\n<p>In other words, I think the data is quite separable by itself, but such separability does not generalize well to the test set. Or maybe&nbsp;the feasible&nbsp;C is just&nbsp;too large. My CV shows quite consistent results with Logistic Regression, wihich is around 0.405.</p>\n<p>Having said that, I don't think there's much space to improve beyond 0.39. That&nbsp;simple script&nbsp;&nbsp;boosts hundreds people to their current point though. The data is sampled in a undisclosed, class dependent way, after all...and that just makes feature engineering really hard.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "60080",
      "postDate": "12/12/2014 06:03:31",
      "content": "<p>[quote=Nicholas Guttenberg;60077]</p>\n<p>&nbsp;'trivial' non-linearity that is actually quite important to capture (such as the app_id/site_id bifurcation)</p>\n<p>[/quote]</p>\n<p>Can you add some comments to this?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "60084",
      "postDate": "12/12/2014 08:05:30",
      "content": "<p>[quote=Mikhail Trofimov;60080]</p>\n<p>[quote=Nicholas Guttenberg;60077]</p>\n<p>&nbsp;'trivial' non-linearity that is actually quite important to capture (such as the app_id/site_id bifurcation)</p>\n<p>[/quote]</p>\n<p>Can you add some comments to this?</p>\n<p>[/quote]</p>\n<p>As I understand it, some part of the data is from mobile phone app ads, and the rest of the data is from website ads. Some of the properties only make sense for one of those two cases,&nbsp;so e.g. a bunch of features associated with the mobile phone ads are set to a null value for all the web ads, and vice versa.</p>\n<p>Some people have reported good results splitting the data into two sets and training two separate classifiers, one for the mobile ads, and another for the web ads.&nbsp;That's basically what I mean by a 'trivial nonlinearity', because the value of the effective coefficient of one feature in the overall model depends on the value of another feature (e.g. whatever feature is used to split the data).</p>\n<p>If a linear model gives you a single hyperplane to separate the data, fitting two linear models to subsets of the data is sort of like having two hyperplanes that meet with a single kink somewhere.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "60086",
      "postDate": "12/12/2014 09:27:51",
      "content": "<p>Thanks a lot for explanation!</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "60187",
      "postDate": "12/14/2014 13:54:37",
      "content": "<p>Can we consider all the values as non numerical to make bag of words for all them. I mean for example he time, instead of int values, changing it to 24 features? If someone already tried that please let me know.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "60188",
      "postDate": "12/14/2014 14:46:26",
      "content": "<p>[quote=Adjeiinfo;60187]</p>\n<p>Can we consider all the values as non numerical to make bag of words for all them. I mean for example he time, instead of int values, changing it to 24 features? If someone already tried that please let me know.</p>\n<p>[/quote]</p>\n<p>I'm sure many people are doing that. The &quot;beat the benchmark&quot; codes does this, for example.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "60215",
      "postDate": "12/15/2014 01:21:58",
      "content": "<p>@inversion</p>\n<p>Thank you. This is getting fun. I know i wont make the money but happy to learn and practice.&nbsp;</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "63515",
      "postDate": "02/03/2015 22:08:54",
      "content": "<p>These are called dummy variables or one hot encoding. It's the most natural way to deal with this dataset. Another option is to compute the&nbsp;E( y | x ) and assign it instead of the &nbsp;categorical variables.&nbsp;</p>\n<p>There is some discussion of this here:</p>\n<p>https://github.com/tqchen/xgboost/issues/121</p>\n\n<p>The technique was used in another similar competition succesfully.</p>",
      "rawMarkdown": "",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 59949,
      "author_name": "hogwild",
      "author_url": "",
      "post_date": "12/10/2014 15:00:23",
      "content": "<p>Feature hashing, as described&nbsp;<a href=\"http://en.wikipedia.org/wiki/Feature_hashing\">here</a> is a good start.</p>\n<p>SVMs, in general, are memory intensive and don't scale well for high dimensional data. I suggest, start with a simple classification algorithm (logistic?) to get a baseline score and optimize from there. Also, I found the information in this post quite useful. <a href=\"http://www.kaggle.com/c/avazu-ctr-prediction/forums/t/10927/beat-the-benchmark-with-less-than-1mb-of-memory\" target=\"_blank\">beat-the-benchmark-with-less-than-1mb-of-memory</a></p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 59955,
      "author_name": "adjeiinfo",
      "author_url": "",
      "post_date": "12/10/2014 16:17:24",
      "content": "<p>@logloss thank you very much.&nbsp;</p>\n<p>I will give it a try and try to build my code on it</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 59978,
      "author_name": "fchollet",
      "author_url": "",
      "post_date": "12/11/2014 00:27:07",
      "content": "<p>Absolutely they do. It's a little known fact, but SVM is actually highly scalable ;-) That is because it only requires to look at support vectors --you can discard the quasi-totality of the vectors in the dataset when training a SVM, as long as you keep the right ones.</p>\n<p style=\"padding-left: 30px\"><em>&quot;Everything what does SVM, it does in terms of similarity between samples.&quot;</em> --Vladimir Vapnik</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 60027,
      "author_name": "jiweiliu",
      "author_url": "",
      "post_date": "12/11/2014 17:41:06",
      "content": "<p>[quote=fchollet;59978]</p>\n<p>Absolutely they do. It's a little known fact, but SVM is actually highly scalable ;-) That is because it only requires to look at support vectors --you can discard the quasi-totality of the vectors in the dataset when training a SVM, as long as you keep the right ones.</p>\n<p style=\"padding-left: 30px\"><em>&quot;Everything what does SVM, it does in terms of similarity between samples.&quot;</em> --Vladimir Vapnik</p>\n<p>[/quote]</p>\n<p>I understand in concept that svm only looks at the data near the decision boundary. But why is it so slow in practice? I'm using sklearn's svm. Does it have something to do with kernel?&nbsp;</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 60030,
      "author_name": "inversion",
      "author_url": "",
      "post_date": "12/11/2014 18:06:11",
      "content": "<p>[quote=rcarson;60027]</p>\n<p>I understand in concept that svm only looks at the data near the decision boundary. But why is it so slow in practice? I'm using sklearn's svm. Does it have something to do with kernel?&nbsp;</p>\n<p>[/quote]</p>\n<p>If someone provides a trick to speed up sklearn's svm, I've give TWO up-votes! &nbsp;:-)</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 60031,
      "author_name": "abhishek",
      "author_url": "",
      "post_date": "12/11/2014 18:09:44",
      "content": "<p>Use this:&nbsp;http://patternsonascreen.net/cuSVM.html</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 60032,
      "author_name": "mikhailtrofimov",
      "author_url": "",
      "post_date": "12/11/2014 18:11:33",
      "content": "<p>[quote=rcarson;60027]</p>\n<p>I understand in concept that svm only looks at the data near the decision boundary. But why is it so slow in practice?&nbsp;</p>\n<p>[/quote]</p>\n<p>This is wrong. SVM &quot;looks&quot; for all data, but its solution is a linear combination of few support vectors (which lies near decision boundary).</p>\n<p>There is no method for fast (<strong>O(samples)</strong>) fitting SVM with custom kernel.</p>\n<p>But for linear kernel - there is. For example, <a href=\"http://www.csie.ntu.edu.tw/~cjlin/liblinear/\">LibLinear</a>&nbsp;&nbsp;(<a href=\"http://scikit-learn.org/stable/modules/generated/sklearn.svm.LinearSVC.html\">sklearn's LinearSVC</a> use this implementation)</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 60033,
      "author_name": "mikhailtrofimov",
      "author_url": "",
      "post_date": "12/11/2014 18:20:10",
      "content": "<p>For large dataset use onehot (or hash trick) and LinearSVC. If you want to optimize logloss - build LogisticRegression upon LinearSVC.decision_function(), for example.</p>\n<p>If want to use non-linear approaches - find out how to reduce this problem to linear =)</p>\n<p>Maybe you need to do some kind of kernel mapping (as in <a href=\"http://fastml.com/the-secret-of-the-big-guys/\">Sofia-ML</a>) or 2-3-features interaction (as in Vowpal Wabbit) or some tree-based features transformation.</p>\n<p>But in the end almost every problem&nbsp;is a&nbsp;linear problem.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 60034,
      "author_name": "jiweiliu",
      "author_url": "",
      "post_date": "12/11/2014 18:26:51",
      "content": "<p>[quote=Mikhail Trofimov;60032]</p>\n<p>[quote=rcarson;60027]</p>\n<p>I understand in concept that svm only looks at the data near the decision boundary. But why is it so slow in practice?&nbsp;</p>\n<p>[/quote]</p>\n<p>This is wrong. SVM &quot;looks&quot; for all data, but its solution is a linear combination of few support vectors (which lies near decision boundary).</p>\n<p>[/quote]</p>\n<p>Thank you very much for correcting me. Mmm, this must be why I got a B.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 60035,
      "author_name": "jiweiliu",
      "author_url": "",
      "post_date": "12/11/2014 18:35:33",
      "content": "<p>[quote=Mikhail Trofimov;60033]</p>\n<p>If you want to optimize logloss - build LogisticRegression upon LinearSVC.desicion_function(), for example.</p>\n<p>[/quote]</p>\n<p>May I ask how this could be done?&nbsp;</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 60036,
      "author_name": "mikhailtrofimov",
      "author_url": "",
      "post_date": "12/11/2014 18:42:46",
      "content": "<p>[quote=rcarson;60035]</p>\n<p>[quote=Mikhail Trofimov;60033]</p>\n<p>If you want to optimize logloss - build LogisticRegression upon LinearSVC.decision_function(), for example.</p>\n<p>[/quote]</p>\n<p>May I ask how this could be done?&nbsp;</p>\n<p>[/quote]</p>\n<p>Split your train data for 80/20. On 80% fit LinearSVC. It cannot return estimated probabilities, but you can get the distance for dicision boundry by calling .decision_function().</p>\n<p>Get this distances for holded 20% and use Logistic Regression for searching the best&nbsp;sigmoid transformation.</p>\n<p>For final predicting use similar chain of actions - get distances -&gt; sigmoid transformation -&gt; estimated probabilities</p>\n<p>It is just a one way, probably not the best.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 60039,
      "author_name": "",
      "author_url": "",
      "post_date": "12/11/2014 19:01:00",
      "content": "<p>The only difference between&nbsp;SVM and logistic regression (in terms of modelling, forget about training at this time)&nbsp;is their loss function. Logistic regression using log-loss, which gives your probability estimation and IS the metric&nbsp;they are using to evaluate&nbsp;your result&nbsp;in this contest.</p>\n<p>SVM, in contrast, gives your hinge loss and thus it makes no sense to get probability estimates from it.</p>\n<p>You are gonna get&nbsp;log-loss to infinity if you got the binary label wrong...which is not as&nbsp;ROBUST as when&nbsp;SVM is just being used as a classifier. Use it when you have other metrics on your result, such as precision, f1, or just accuracy.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 60063,
      "author_name": "ngutten",
      "author_url": "",
      "post_date": "12/12/2014 00:39:43",
      "content": "<p>After fchollet's comment I looked into the tricks people to do speed up SVM. It seems like there is a variant called CB-SVM where you hierarchically cluster the data, train the SVM, and then de-cluster the clumps that are intersected by the decision boundary. It looks like it should be N log(N) in the data - the authors do example sets up to something like 4 million data points in their <a href=\"http://dl.acm.org/citation.cfm?id=1103738\">paper</a>,&nbsp;and that was 2005, so 40 million might be manageable.</p>\n<p>They do say that it's bad at high-dimensional feature spaces though, so one-hotting is probably not the best idea.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 60069,
      "author_name": "",
      "author_url": "",
      "post_date": "12/12/2014 03:16:58",
      "content": "<p>And then you could hash your feature into space of small number of&nbsp;dimension.</p>\n<p>How about 2^10?</p>\n\n<p>Edit: by the way, the link you've given is broken.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 60072,
      "author_name": "ngutten",
      "author_url": "",
      "post_date": "12/12/2014 04:58:17",
      "content": "<p>Huh, for some reason the forums really didn't like that link. I found another link to the paper, but this one might need institutional access to get the full text. Anyhow, a google search for CB-SVM finds a bunch more stuff about it including some presentations going through it in detail.</p>\n<p>For hashing,&nbsp;it feels like if your categorical variables were truly independent then using such a small space would cause problems when e.g. you hash a variable that predicts a high average CTR into the same feature as a variable that predicts a low average CTR. If you make it really tiny, eventually I guess you'd just get every data row having all features set to '1'.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 60074,
      "author_name": "",
      "author_url": "",
      "post_date": "12/12/2014 05:06:11",
      "content": "<p>Your guess is not true; there are 21 variables in the beginning, so you can't have a row with more than 21 non zero elements.</p>\n<p>However, you are right that small size of hash function will cause collision problems. I am actually telling&nbsp;a joke of hashing it to only 1k size. But it might work for 1M, I don't know.</p>\n<p>Although I am not familiar with CBSVM, but based on their citation count (published 10 years ago), this method may not be very useful, plus it's hard to find a good implementation. I'd rather suggest <a href=\"http://www.csie.ntu.edu.tw/~cjlin/liblinear/\">liblinear</a>&nbsp;(which is cited 406 times since 2008), whose linear support vector classifier has been proved competitive on both performance and&nbsp;scalability.</p>\n<p>Please correct me if I am wrong.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 60077,
      "author_name": "ngutten",
      "author_url": "",
      "post_date": "12/12/2014 05:42:55",
      "content": "<p>I'd have to try both to tell. I guess the question is, do you get more from being able to use the kernel trick and a non-linear kernel, or from being able to train the SVM on more of the data. How important is it to retain the ability to make use of that non-linearity?</p>\n<p>There may be some 'trivial' non-linearity that is actually quite important to capture (such as the app_id/site_id bifurcation), but which you could resolve using feature engineering or by subdividing the data.</p>\n<p>I honestly haven't had much luck with SVM for this. I used feature histograms for my dimensionality reduction and pushed a bunch of sets of about 15000 data rows through Scikit-Learn's SVM implementation. After&nbsp;bagging and grid-search, I still just barely break 0.41 with that method, and going from 10000 to 15000 data rows didn't make much difference. I suppose 15000 to 40 million may still do so however, and being able to actually work with the full categorical information (or ~16 million features worth of it) should in principle be a big gain over feature histograms.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 60079,
      "author_name": "",
      "author_url": "",
      "post_date": "12/12/2014 05:59:50",
      "content": "<p>Given that the dimension is large already, which kernel do you think will do the job? The whole point of SVM is linear separability, and I doubt if it's really necessary to map the hot encoding into some larger space to get that.</p>\n<p>In other words, I think the data is quite separable by itself, but such separability does not generalize well to the test set. Or maybe&nbsp;the feasible&nbsp;C is just&nbsp;too large. My CV shows quite consistent results with Logistic Regression, wihich is around 0.405.</p>\n<p>Having said that, I don't think there's much space to improve beyond 0.39. That&nbsp;simple script&nbsp;&nbsp;boosts hundreds people to their current point though. The data is sampled in a undisclosed, class dependent way, after all...and that just makes feature engineering really hard.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 60080,
      "author_name": "mikhailtrofimov",
      "author_url": "",
      "post_date": "12/12/2014 06:03:31",
      "content": "<p>[quote=Nicholas Guttenberg;60077]</p>\n<p>&nbsp;'trivial' non-linearity that is actually quite important to capture (such as the app_id/site_id bifurcation)</p>\n<p>[/quote]</p>\n<p>Can you add some comments to this?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 60084,
      "author_name": "ngutten",
      "author_url": "",
      "post_date": "12/12/2014 08:05:30",
      "content": "<p>[quote=Mikhail Trofimov;60080]</p>\n<p>[quote=Nicholas Guttenberg;60077]</p>\n<p>&nbsp;'trivial' non-linearity that is actually quite important to capture (such as the app_id/site_id bifurcation)</p>\n<p>[/quote]</p>\n<p>Can you add some comments to this?</p>\n<p>[/quote]</p>\n<p>As I understand it, some part of the data is from mobile phone app ads, and the rest of the data is from website ads. Some of the properties only make sense for one of those two cases,&nbsp;so e.g. a bunch of features associated with the mobile phone ads are set to a null value for all the web ads, and vice versa.</p>\n<p>Some people have reported good results splitting the data into two sets and training two separate classifiers, one for the mobile ads, and another for the web ads.&nbsp;That's basically what I mean by a 'trivial nonlinearity', because the value of the effective coefficient of one feature in the overall model depends on the value of another feature (e.g. whatever feature is used to split the data).</p>\n<p>If a linear model gives you a single hyperplane to separate the data, fitting two linear models to subsets of the data is sort of like having two hyperplanes that meet with a single kink somewhere.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 60086,
      "author_name": "mikhailtrofimov",
      "author_url": "",
      "post_date": "12/12/2014 09:27:51",
      "content": "<p>Thanks a lot for explanation!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 60187,
      "author_name": "adjeiinfo",
      "author_url": "",
      "post_date": "12/14/2014 13:54:37",
      "content": "<p>Can we consider all the values as non numerical to make bag of words for all them. I mean for example he time, instead of int values, changing it to 24 features? If someone already tried that please let me know.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 60188,
      "author_name": "inversion",
      "author_url": "",
      "post_date": "12/14/2014 14:46:26",
      "content": "<p>[quote=Adjeiinfo;60187]</p>\n<p>Can we consider all the values as non numerical to make bag of words for all them. I mean for example he time, instead of int values, changing it to 24 features? If someone already tried that please let me know.</p>\n<p>[/quote]</p>\n<p>I'm sure many people are doing that. The &quot;beat the benchmark&quot; codes does this, for example.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 60215,
      "author_name": "adjeiinfo",
      "author_url": "",
      "post_date": "12/15/2014 01:21:58",
      "content": "<p>@inversion</p>\n<p>Thank you. This is getting fun. I know i wont make the money but happy to learn and practice.&nbsp;</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 63515,
      "author_name": "marbel",
      "author_url": "",
      "post_date": "02/03/2015 22:08:54",
      "content": "<p>These are called dummy variables or one hot encoding. It's the most natural way to deal with this dataset. Another option is to compute the&nbsp;E( y | x ) and assign it instead of the &nbsp;categorical variables.&nbsp;</p>\n<p>There is some discussion of this here:</p>\n<p>https://github.com/tqchen/xgboost/issues/121</p>\n\n<p>The technique was used in another similar competition succesfully.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "59942": "",
    "59949": "",
    "59955": "",
    "59978": "",
    "60027": "",
    "60030": "",
    "60031": "",
    "60032": "",
    "60033": "",
    "60034": "",
    "60035": "",
    "60036": "",
    "60039": "",
    "60063": "",
    "60069": "",
    "60072": "",
    "60074": "",
    "60077": "",
    "60079": "",
    "60080": "",
    "60084": "",
    "60086": "",
    "60187": "",
    "60188": "",
    "60215": "",
    "63515": ""
  },
  "source": "meta"
}