I tried neural network. Seems neural network is totally useless here. May be I have something wrong in network parameters. Can anybody give me direction?
", "messages": [ { "id": "60073", "postDate": "12/12/2014 05:00:19", "content": "I tried neural network. Seems neural network is totally useless here. May be I have something wrong in network parameters. Can anybody give me direction?
", "rawMarkdown": "", "votes": null }, { "id": "60082", "postDate": "12/12/2014 07:27:45", "content": "Shameless boast of my own work in a similar competition.
\nhttp://www.kaggle.com/c/criteo-display-ad-challenge/forums/t/10534/required-model-documentation-and-code/55683#post55683
However.. the data in this competition is a bit "simpler" in the sense that everything can naturally be treated as a categorical. But we'll see in the end how everything turns out.
Hi Julian, I'd like to implement your neural network with torch or theano, which do you think is better?
\nAnd I don't have a gpu for now. Will it be a huge slowdown?
\nThank you very much!
", "rawMarkdown": "", "votes": null }, { "id": "60106", "postDate": "12/12/2014 15:41:24", "content": "[quote=rcarson;60098]
\nHi Julian, I'd like to implement your neural network with torch or theano, which do you think is better?
\nAnd I don't have a gpu for now. Will it be a huge slowdown?
\nThank you very much!
\n[/quote]
\n\nI am a beginner of machine learning. I use a C package called FANN. May something wrong in argument. The result seems very bad.
", "rawMarkdown": "", "votes": null }, { "id": "60110", "postDate": "12/12/2014 16:49:03", "content": "First of all.. Everything in this post is just my experience with NN's. I might be dead wrong about some things. If so.. please post!
\n@Zhang Junhyan
Be sure to have L1 regularization or to recode/rare unseen variables in the train and test set.
If you don't do this the randomly initialized weights are not adjusted during training and you will get strange results when predicting. That was the most important thing for these CTR problems with many new/rare values.
If you are a beginner I'd start with the "beat the benchmark" scripts.. They are much more rewarding. Once you get some results with those you can try a NN and compare. (or even better.. blend!)
@Rcarson..
You see my profile pic ? No hair.. That's because I tried them all.. j/k :)
Just my opinion in terms of "runnability".
1. Caffe. I think this one is the "biggest" and I believe it has python support.
2. Theano. Python is great.. Getting it running was pretty hard.
3. Torch. Lua scripting is not mainstream and tutorials are confusing due to the many versions.
I've seen all of them do good in competitions.
Problem is that most of them are focused on vision. For this competition it's almost necessary to support sparsity at the input layer. I don't know which one has this feature.
Having a GPU does not hurt but in this case (taking advantage of sparse input) it *might* be barable on CPU.
@Last.
I don't want to be like a "pied piper" leading you to terrible pains with NN's :)
For this comp. you basically have binary input.
X*Y features can be found manually. That perhaps leaves some binary relations to be found for the NN. Let's say X AND Y AND NOT Z. If they are present and hard to find a NN might give you a little edge. However.. the NN is not guaranteed to find these features.
With criteo the data was more diverse since you also had numeric data.
So there were more hard to find features and perhaps that was my luck in that competition.
As for now.. I just don't know yet if NN's will give an extra edge.
So far I've found that even a very simple NN (input layer as small as 80) training on only a single day of data with backprop performs moderately on CV: ~0.41. Definitely not competition winning at that level of complexity, but not so terrible either.
", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 60082, "author_name": "juliandewit", "author_url": "", "post_date": "12/12/2014 07:27:45", "content": "Shameless boast of my own work in a similar competition.
\nhttp://www.kaggle.com/c/criteo-display-ad-challenge/forums/t/10534/required-model-documentation-and-code/55683#post55683
However.. the data in this competition is a bit "simpler" in the sense that everything can naturally be treated as a categorical. But we'll see in the end how everything turns out.
Hi Julian, I'd like to implement your neural network with torch or theano, which do you think is better?
\nAnd I don't have a gpu for now. Will it be a huge slowdown?
\nThank you very much!
", "votes": null, "replies": [] }, { "id": 60106, "author_name": "zhangjunyan", "author_url": "", "post_date": "12/12/2014 15:41:24", "content": "[quote=rcarson;60098]
\nHi Julian, I'd like to implement your neural network with torch or theano, which do you think is better?
\nAnd I don't have a gpu for now. Will it be a huge slowdown?
\nThank you very much!
\n[/quote]
\n\nI am a beginner of machine learning. I use a C package called FANN. May something wrong in argument. The result seems very bad.
", "votes": null, "replies": [] }, { "id": 60110, "author_name": "juliandewit", "author_url": "", "post_date": "12/12/2014 16:49:03", "content": "First of all.. Everything in this post is just my experience with NN's. I might be dead wrong about some things. If so.. please post!
\n@Zhang Junhyan
Be sure to have L1 regularization or to recode/rare unseen variables in the train and test set.
If you don't do this the randomly initialized weights are not adjusted during training and you will get strange results when predicting. That was the most important thing for these CTR problems with many new/rare values.
If you are a beginner I'd start with the "beat the benchmark" scripts.. They are much more rewarding. Once you get some results with those you can try a NN and compare. (or even better.. blend!)
@Rcarson..
You see my profile pic ? No hair.. That's because I tried them all.. j/k :)
Just my opinion in terms of "runnability".
1. Caffe. I think this one is the "biggest" and I believe it has python support.
2. Theano. Python is great.. Getting it running was pretty hard.
3. Torch. Lua scripting is not mainstream and tutorials are confusing due to the many versions.
I've seen all of them do good in competitions.
Problem is that most of them are focused on vision. For this competition it's almost necessary to support sparsity at the input layer. I don't know which one has this feature.
Having a GPU does not hurt but in this case (taking advantage of sparse input) it *might* be barable on CPU.
@Last.
I don't want to be like a "pied piper" leading you to terrible pains with NN's :)
For this comp. you basically have binary input.
X*Y features can be found manually. That perhaps leaves some binary relations to be found for the NN. Let's say X AND Y AND NOT Z. If they are present and hard to find a NN might give you a little edge. However.. the NN is not guaranteed to find these features.
With criteo the data was more diverse since you also had numeric data.
So there were more hard to find features and perhaps that was my luck in that competition.
As for now.. I just don't know yet if NN's will give an extra edge.
So far I've found that even a very simple NN (input layer as small as 80) training on only a single day of data with backprop performs moderately on CV: ~0.41. Definitely not competition winning at that level of complexity, but not so terrible either.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "60073": "", "60082": "", "60098": "", "60106": "", "60110": "", "60111": "" }, "source": "meta" }