{ "id": 8188, "title": "Problem with features with SVM and Linear Regression", "url": "/competitions/acquire-valued-shoppers-challenge/discussion/8188", "author_name": "", "post_date": "2014-05-17T11:16:38.063Z", "votes": null, "comment_count": 7, "views": 2890, "content": "

I have been trying to use some features similar to those described by Triskelion here but nothing works.

\n

I have tried using SVM and Logistic Regression (edit: not Linear Regression as I put on the title) from SciKit but the result is really poor.

\n

My features have a lot of zeros since the customer might have purchased something e.g. in the category of the offer but not in the brand or company etc.

\n

Do you think that this might be causing the problem? Does anyone else use SVM or Linear Regression for the problem?

", "messages": [ { "id": "44717", "postDate": "05/17/2014 11:16:38", "content": "

I have been trying to use some features similar to those described by Triskelion here but nothing works.

\n

I have tried using SVM and Logistic Regression (edit: not Linear Regression as I put on the title) from SciKit but the result is really poor.

\n

My features have a lot of zeros since the customer might have purchased something e.g. in the category of the offer but not in the brand or company etc.

\n

Do you think that this might be causing the problem? Does anyone else use SVM or Linear Regression for the problem?

", "rawMarkdown": "", "votes": null }, { "id": "44749", "postDate": "05/17/2014 18:28:02", "content": "

Too high dimensional data can be a problem with certain algo's. You can try Decomposition to lower dimensionality. 

\n

Logistic Regression benefits from a little regularization (tweak C and intercept_scaling).

\n

For SVM, note that:

\n

Support Vector Machine algorithms are not scale invariant, so it is highly recommended to scale your data. For example, scale each attribute on the input vector X to [0,1] or [-1,+1], or standardize it to have mean 0 and variance 1. Note that the same scaling must be applied to the test vector to obtain meaningful results. See section Preprocessing data for more details on scaling and normalization. (preprocessing.MinMaxScaler)

\n

For both algo's, run it a few time with cross_val_score scoring=roc_auc so you know which parameters attribute to a higher score. Good luck!

", "rawMarkdown": "", "votes": null }, { "id": "44797", "postDate": "05/18/2014 23:32:45", "content": "

Thank you for your reply!

\n

I found out a mistake in my code that fixed the problem.

", "rawMarkdown": "", "votes": null }, { "id": "44799", "postDate": "05/18/2014 23:53:48", "content": "

@Konstantina

\n

how long is SVM taking to train? I wanted to give it a try, but optimizing the hyper-parameters with grid search seemed a bit prohibitive.

", "rawMarkdown": "", "votes": null }, { "id": "44857", "postDate": "05/20/2014 00:08:33", "content": "

With the size of data that you have maybe SVM run so slow because need so much time for split the information.

", "rawMarkdown": "", "votes": null }, { "id": "44870", "postDate": "05/20/2014 00:40:55", "content": "

@ fernando nogueira

\n

The LinearSVC of scikit learn runs quite fast (< 1 minute) but if you want to use another kernel it takes long. I actually did not have the patience to wait for the output.

\n

For the moment my best score is found with Vowpal Wabbit.

", "rawMarkdown": "", "votes": null }, { "id": "45987", "postDate": "05/21/2014 13:19:42", "content": "

My features have a lot of zeros since the customer might have purchased something e.g. in the category of the offer but not in the brand or company etc.

\n

As you mentioned you have lots of zeros in the features does it mean you have created variables like trxn_count_C1, trxn_count_C2,trxn_count_C3,trxn_count_C4 so on...

\n

eg. trxn_count_C1 = total transactions of the customer with company 1 in 1 year

\n

Similarly you have created variables for Brand and category? i.e lets say if there are 100 companies in the data you will create 100 such variables for transaction counts.

\n

Your reply will be really helpful for me. Thanks!

", "rawMarkdown": "", "votes": null }, { "id": "46153", "postDate": "05/23/2014 23:22:24", "content": "

@Decipher,

\n

I did not create features for each company but for the company of the offer received by the client.

\n

For each client you find out what is the offer that he got in the offers file. Then you compute features for the specific company, brand and category of the offer NOT for every company, brand and category because you cannot relate them to the offer.

\n

I hope this clarifies things..

", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 44749, "author_name": "triskelion", "author_url": "", "post_date": "05/17/2014 18:28:02", "content": "

Too high dimensional data can be a problem with certain algo's. You can try Decomposition to lower dimensionality. 

\n

Logistic Regression benefits from a little regularization (tweak C and intercept_scaling).

\n

For SVM, note that:

\n

Support Vector Machine algorithms are not scale invariant, so it is highly recommended to scale your data. For example, scale each attribute on the input vector X to [0,1] or [-1,+1], or standardize it to have mean 0 and variance 1. Note that the same scaling must be applied to the test vector to obtain meaningful results. See section Preprocessing data for more details on scaling and normalization. (preprocessing.MinMaxScaler)

\n

For both algo's, run it a few time with cross_val_score scoring=roc_auc so you know which parameters attribute to a higher score. Good luck!

", "votes": null, "replies": [] }, { "id": 44797, "author_name": "kontoudi", "author_url": "", "post_date": "05/18/2014 23:32:45", "content": "

Thank you for your reply!

\n

I found out a mistake in my code that fixed the problem.

", "votes": null, "replies": [] }, { "id": 44799, "author_name": "fnogueira", "author_url": "", "post_date": "05/18/2014 23:53:48", "content": "

@Konstantina

\n

how long is SVM taking to train? I wanted to give it a try, but optimizing the hyper-parameters with grid search seemed a bit prohibitive.

", "votes": null, "replies": [] }, { "id": 44857, "author_name": "pablolecce", "author_url": "", "post_date": "05/20/2014 00:08:33", "content": "

With the size of data that you have maybe SVM run so slow because need so much time for split the information.

", "votes": null, "replies": [] }, { "id": 44870, "author_name": "kontoudi", "author_url": "", "post_date": "05/20/2014 00:40:55", "content": "

@ fernando nogueira

\n

The LinearSVC of scikit learn runs quite fast (< 1 minute) but if you want to use another kernel it takes long. I actually did not have the patience to wait for the output.

\n

For the moment my best score is found with Vowpal Wabbit.

", "votes": null, "replies": [] }, { "id": 45987, "author_name": "decipher", "author_url": "", "post_date": "05/21/2014 13:19:42", "content": "

My features have a lot of zeros since the customer might have purchased something e.g. in the category of the offer but not in the brand or company etc.

\n

As you mentioned you have lots of zeros in the features does it mean you have created variables like trxn_count_C1, trxn_count_C2,trxn_count_C3,trxn_count_C4 so on...

\n

eg. trxn_count_C1 = total transactions of the customer with company 1 in 1 year

\n

Similarly you have created variables for Brand and category? i.e lets say if there are 100 companies in the data you will create 100 such variables for transaction counts.

\n

Your reply will be really helpful for me. Thanks!

", "votes": null, "replies": [] }, { "id": 46153, "author_name": "kontoudi", "author_url": "", "post_date": "05/23/2014 23:22:24", "content": "

@Decipher,

\n

I did not create features for each company but for the company of the offer received by the client.

\n

For each client you find out what is the offer that he got in the offers file. Then you compute features for the specific company, brand and category of the offer NOT for every company, brand and category because you cannot relate them to the offer.

\n

I hope this clarifies things..

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "44717": "", "44749": "", "44797": "", "44799": "", "44857": "", "44870": "", "45987": "", "46153": "" }, "source": "meta" }