{ "id": 10927, "title": "Beat the benchmark with less than 1MB of memory.", "url": "/competitions/avazu-ctr-prediction/discussion/10927", "author_name": "", "post_date": "2014-11-14T05:56:00.830Z", "votes": 192, "comment_count": 188, "views": 101291, "content": "

Notice (11/18/2014): added the third version of the script fast_solution_v3.py

\n

Notice (11/20/2014): here is a post on how to get 0.3977417 on the public leaderboard

\n

-

\n

Dear fellow Kagglers, tin-ert-gu here.

\n

-

\n

This is a python implementation of adaptive learning rate L1 and L2 regularized logistic regression using hash-trick one-hot encoding. This algorithm is also used by Google for their online CTR prediction system.
All credits of this script goes to their hard work of research.

\n

-

\n

How to run?

\n

In your terminal, simply type

\n

python fast_solution.py

\n

The script is also python3 compatible, so you can use python3 as well

\n

python3 fast_solution.py

\n

However, since the code only depends on native modules, it is highly suggested that you run the script with pypy for a huge speed-up. To use the script with pypy on Ubuntu, first type

\n

sudo apt-get install pypy

\n

Then use the pypy interpreter to run the script

\n

pypy fast_solution.py

\n

-

\n

Changelog over the previous version of the script

\n
    \n
  1. Added L1 regularization
    2. \n
  2. Added L2 regularization
    3. \n
  3. Build in multiple epoch training
    4. \n
  4. Build in holdout set validation support
    5. \n
  5. Build in support for poly 2 feature interactions
    6. \n
  6. Re-factored code for better extendibility
    7. \n
  7. EVEN MORE COMMENTS!
    8. \n
\n

-

\n

Performance

\n

Training time for a single epoch, on a single Intel(R) Xeon(R) CPU E5-2630L v2 @ 2.40GHz:

\n\n

Memory usage:

\n

This mainly depends on the D parameter in the script.

\n\n

-

\n

The entire algorithm is disclosed, while using only out of the box modules provided by Python, so it should also be an easy sandbox code to build new ideas upon.

\n

Good luck and have fun!

\n

-

\n

@Abhishek:

\n

Can't submit right now, so I don't know the leader-board score, but I would say it should easily beat 0.40

\n

-

\n

EDIT 1

\n

add fast_solution_v2.py to fix a null reference bug under memory saving mode

\n

EDIT 2

\n

add fast_solution_v3.py

\n

A great thanks for Paweł's feedback, the following changes are made as a result

\n\n

By considering fchollet's experiment

\n\n

This should be the last time I modify this benchmark for this competition. Hope the actual data will get release soon, and happy predicting!

", "messages": [ { "id": "58054", "postDate": "11/14/2014 05:56:00", "content": "

Notice (11/18/2014): added the third version of the script fast_solution_v3.py

\n

Notice (11/20/2014): here is a post on how to get 0.3977417 on the public leaderboard

\n

-

\n

Dear fellow Kagglers, tin-ert-gu here.

\n

-

\n

This is a python implementation of adaptive learning rate L1 and L2 regularized logistic regression using hash-trick one-hot encoding. This algorithm is also used by Google for their online CTR prediction system.
All credits of this script goes to their hard work of research.

\n

-

\n

How to run?

\n

In your terminal, simply type

\n

python fast_solution.py

\n

The script is also python3 compatible, so you can use python3 as well

\n

python3 fast_solution.py

\n

However, since the code only depends on native modules, it is highly suggested that you run the script with pypy for a huge speed-up. To use the script with pypy on Ubuntu, first type

\n

sudo apt-get install pypy

\n

Then use the pypy interpreter to run the script

\n

pypy fast_solution.py

\n

-

\n

Changelog over the previous version of the script

\n
    \n
  1. Added L1 regularization
    2. \n
  2. Added L2 regularization
    3. \n
  3. Build in multiple epoch training
    4. \n
  4. Build in holdout set validation support
    5. \n
  5. Build in support for poly 2 feature interactions
    6. \n
  6. Re-factored code for better extendibility
    7. \n
  7. EVEN MORE COMMENTS!
    8. \n
\n

-

\n

Performance

\n

Training time for a single epoch, on a single Intel(R) Xeon(R) CPU E5-2630L v2 @ 2.40GHz:

\n\n

Memory usage:

\n

This mainly depends on the D parameter in the script.

\n\n

-

\n

The entire algorithm is disclosed, while using only out of the box modules provided by Python, so it should also be an easy sandbox code to build new ideas upon.

\n

Good luck and have fun!

\n

-

\n

@Abhishek:

\n

Can't submit right now, so I don't know the leader-board score, but I would say it should easily beat 0.40

\n

-

\n

EDIT 1

\n

add fast_solution_v2.py to fix a null reference bug under memory saving mode

\n

EDIT 2

\n

add fast_solution_v3.py

\n

A great thanks for Paweł's feedback, the following changes are made as a result

\n\n

By considering fchollet's experiment

\n\n

This should be the last time I modify this benchmark for this competition. Hope the actual data will get release soon, and happy predicting!

", "rawMarkdown": "", "votes": null }, { "id": "58055", "postDate": "11/14/2014 06:02:21", "content": "

I wish I could vote up more than once. 

\n

This code is classic. Thank you!

\n\n

And 1MB ? really ? :P

", "rawMarkdown": "", "votes": null }, { "id": "58056", "postDate": "11/14/2014 06:03:32", "content": "

[quote=rcarson;58055]

\n

I wish I could vote up more than once. 

\n

This code is classic. Thank you!

\n

And 1MB ? really ? :P

\n

[/quote]

\n

1MB is too much for beating the benchmark. It is hard not to beat the benchmark for this one.

", "rawMarkdown": "", "votes": null }, { "id": "58057", "postDate": "11/14/2014 06:56:29", "content": "

Hi all,

\n

How one can check memory usage during model training in Linux?

", "rawMarkdown": "", "votes": null }, { "id": "58058", "postDate": "11/14/2014 08:08:44", "content": "

[quote=Herimanitra;58057]

\n

Hi all,

\n

How one can check memory usage during model training in Linux?

\n

[/quote]

\n

top

\n

https://duckduckgo.com/?q=top+linux&t=canonical

", "rawMarkdown": "", "votes": null }, { "id": "58061", "postDate": "11/14/2014 10:14:39", "content": "

Classy!

\n

Congratulations & many thanks!

", "rawMarkdown": "", "votes": null }, { "id": "58062", "postDate": "11/14/2014 10:21:05", "content": "

Revised version, with more control capabilities from command line.

\n

This is really now big fun!

", "rawMarkdown": "", "votes": null }, { "id": "58063", "postDate": "11/14/2014 10:44:12", "content": "

[quote=Yannick Martel;58062]

\n

Revised version, with more control capabilities from command line.

\n

This is really now big fun!

\n

[/quote]

\n

Should I create a github repo so that people, like you, can also contribute and make the script better?

", "rawMarkdown": "", "votes": null }, { "id": "58064", "postDate": "11/14/2014 10:53:07", "content": "

You don't want it becoming a burden on you

", "rawMarkdown": "", "votes": null }, { "id": "58065", "postDate": "11/14/2014 10:54:19", "content": "

[quote=ACS69;58064]

\n

You don't want it becoming a burden on you

\n

[/quote]

\n

OK, giving up right now, I'll just continue to share on the forums :P

\n

BTW, nice Homer thumbnail.

", "rawMarkdown": "", "votes": null }, { "id": "58066", "postDate": "11/14/2014 10:56:08", "content": "

lol! Your code is too beautiful to become a burden!

\n\n

edit: Homer in a West Ham kit

", "rawMarkdown": "", "votes": null }, { "id": "58069", "postDate": "11/14/2014 13:02:50", "content": "

The subject should've been "Beat the benchmark in future with less than 1MB of memory" !

", "rawMarkdown": "", "votes": null }, { "id": "58072", "postDate": "11/14/2014 13:52:53", "content": "

[quote=tinrtgu;58063]

\n

[quote=Yannick Martel;58062]

\n

Revised version, with more control capabilities from command line.

\n

This is really now big fun!

\n

[/quote]

\n

Should I create a github repo so that people, like you, can also contribute and make the script better?

\n

[/quote]  

\n

great idea .. let's create a pykaggle lib. :)

", "rawMarkdown": "", "votes": null }, { "id": "58083", "postDate": "11/14/2014 18:06:40", "content": "

[quote=Faron;58072]

\n

[quote=tinrtgu;58063]

\n

[quote=Yannick Martel;58062]

\n

Revised version, with more control capabilities from command line.

\n

This is really now big fun!

\n

[/quote]

\n

Should I create a github repo so that people, like you, can also contribute and make the script better?

\n

[/quote]  

\n

great idea .. let's create a pykaggle lib. :)

\n

[/quote]

\n

Yes, it would be a really great idea.

", "rawMarkdown": "", "votes": null }, { "id": "58089", "postDate": "11/14/2014 21:17:37", "content": "

Awesome! Thank you for sharing this. The method is pretty much state of the art, and the code quality is excellent! Huge thumb up.

\n

If you guys haven't already, be sure to check out the Google paper that describes this method. Hint: it contains useful info on how to improve the algo :) 

\n

http://www.eecs.tufts.edu/~dsculley/papers/ad-click-prediction.pdf

", "rawMarkdown": "", "votes": null }, { "id": "58091", "postDate": "11/14/2014 21:26:03", "content": "

@tinrtgu Great job! Also I like your optimism. You already assumed that there will be a 2nd revision of the dataset :).

\n

train = 'train_rev2'
test = 'test_rev2'

\n

I will try to rewrite this in Cython to see what speed-up I can get.

", "rawMarkdown": "", "votes": null }, { "id": "58092", "postDate": "11/14/2014 21:53:46", "content": "

[quote=Pawel;58091]

\n

@tinrtgu Great job! Also I like your optimism. You already assumed that there will be a 2nd revision of the dataset :).

\n

train = 'train_rev2'
test = 'test_rev2'

\n

I will try to rewrite this in Cython to see what speed-up I can get.

\n

[/quote]

\n

Actually, we'll be on rev3 once the competition starts back up. Rev2 was also broken.

", "rawMarkdown": "", "votes": null }, { "id": "58093", "postDate": "11/14/2014 21:59:21", "content": "

Oh. I must have missed that revision :).

", "rawMarkdown": "", "votes": null }, { "id": "58095", "postDate": "11/14/2014 22:42:12", "content": "

[quote=Pawel;58091]

\n

I will try to rewrite this in Cython to see what speed-up I can get.

\n

[/quote]

\n

I think you are able to get a fair amount of speed up, since I was trying to go for maximum readability the script is actually lacking a bit in the run speed and memory utilization department.

", "rawMarkdown": "", "votes": null }, { "id": "58108", "postDate": "11/15/2014 11:38:10", "content": "

Around line 268 there is this 

\n

for key in sorted(row): # sort is for preserving feature ordering

\n

I think this sort is unnecessary. The results are the same if you skip it.

\n

for key in row:

", "rawMarkdown": "", "votes": null }, { "id": "58110", "postDate": "11/15/2014 12:11:00", "content": "

I'm trying to wrap my head around this problem. I don't understand why the lazy weights (self.w) are changed in the predict method. IMO model shouldn't be changed during prediction. It would be more logical to me if the predict method loop looked like this:

\n

wTx = 0.
for i in self._indices(x):
    wTx += w[i]

\n

And move all the updates of the model in the actual update method. It does change slightly the results because the updates are made after seeing the observation

\n

See this gist:

\n

https://gist.github.com/pjankiewicz/8ab7094d263bf0d4cfb8

", "rawMarkdown": "", "votes": null }, { "id": "58111", "postDate": "11/15/2014 12:30:39", "content": "

[quote=Pawel;58108]

\n

Around line 268 there is this 

\n

for key in sorted(row): # sort is for preserving feature ordering

\n

I think this sort is unnecessary. The results are the same if you skip it.

\n

for key in row:

\n

[/quote]

\n

Yep, you are right and thanks for pointing this out. But that is only because I used the following hashing scheme for feature interactions, which does not depends on the order of i and j.

\n

for i in xrange(1, L): # skip bias term, so we start at 1
    for j in xrange(i+1, L):
        yield (i * j) % D

\n

I actually have thought about this, to leave in the sort of the take it out. In the end I've decided to leave in the sort since I think there is a high chance people might want to change the hashing method and this would be a hard to catch bug for them.

\n

Since you know what you are doing please feel free to remove it.

", "rawMarkdown": "", "votes": null }, { "id": "58112", "postDate": "11/15/2014 12:34:34", "content": "

[quote=Pawel]

\n

I'm trying to wrap my head around this problem. I don't understand why the lazy weights (self.w) are changed in the predict method. IMO model shouldn't be changed during prediction. It would be more logical to me if the predict method loop looked like this:

\n

wTx = 0.
for i in self._indices(x):
    wTx += w[i]

\n

And move all the updates of the model in the actual update method. It does change slightly the results because the updates are made after seeing the observation

\n

See this gist:

\n

https://gist.github.com/pjankiewicz/8ab7094d263bf0d4cfb8

\n

[/quote]

\n

You are right again, geez you are good at this!

\n

Calculating w at prediction time is there for the support of memory saving mode. If you try to search for "# use this for memory usage reduction" in the code you can actually find out that we actually don't need the full w all the time. We can remove (the full) w and save up to 30% of the memory consumption.

", "rawMarkdown": "", "votes": null }, { "id": "58118", "postDate": "11/15/2014 18:07:45", "content": "

tinrtgu: You should start a series of "Beating the benchmark with xxx score using less than yyy memory in just zzz lines of code!" and give Abhishek a run for his money!

\n

I won't be surprised if some day an innocent but wise Kaggler wins a competition by ensembling all these wonderful BeatingTheBenchmark codes from the forums! ;-)

", "rawMarkdown": "", "votes": null }, { "id": "58130", "postDate": "11/16/2014 08:10:53", "content": "

[quote=Rohan Rao;58118]

\n

tinrtgu: You should start a series of "Beating the benchmark with xxx score using less than yyy memory in just zzz lines of code!" and give Abhishek a run for his money!

\n

[/quote]

\n

I'm hoping more of a hackathon competition where there are 100 datasets and 100 benchmarks, the winner is the one who can beat the most in 24 hours.

", "rawMarkdown": "", "votes": null }, { "id": "58131", "postDate": "11/16/2014 11:40:04", "content": "

[quote=tinrtgu;58130]

\n

I'm hoping more of a hackathon competition where there are 100 datasets and 100 benchmarks, the winner is the one who can beat the most in 24 hours.

\n

[/quote]

\n

Hack-a-ton!

", "rawMarkdown": "", "votes": null }, { "id": "58158", "postDate": "11/16/2014 22:22:56", "content": "

@tinrtgu I got another thing ;). Much more subtle this time.

\n

The way you introduced interactions:

\n

yield (i * j) % D

\n

is somehow faulty because it produces non-uniform hashes. For example 2 random numbers are much more likely to produce 0 as the last digit.

\n

A small histogram of hashes for ((i * j) % D) % 10, where i,j are random numbers from range D

\n

0 *******************
1 ***
2 ******
3 ***
4 *********
5 ***
6 ******
7 ***
8 ************
9 ***

\n

Interactions should be introduced before hashing with something like to prevent this:

\n

hash(col1 + '_' + value1 + '_' + col2 + '_' + value2) % D

\n

EDIT: There may be a simpler solution actually 

\n

yield hash(str(i) + '_' + str(j)) % D

", "rawMarkdown": "", "votes": null }, { "id": "58160", "postDate": "11/16/2014 23:07:18", "content": "

[quote=Pawel;58158]

\n

The way you introduced interactions:

\n

yield (i * j) % D

\n

is somehow faulty because it produces non-uniform hashes. For example 2 random numbers are much more likely to produce 0 as the last digit.

\n

Interactions should be introduced before hashing with something like to prevent this:

\n

hash(col1 + '_' + value1 + '_' + col2 + '_' + value2) % D

\n

EDIT: There may be a simpler solution actually 

\n

yield hash(str(i) + '_' + str(j)) % D

\n

[/quote]

\n

Yep, it is a trade-off between speed and accuracy, and this is also the reason that I left in the sorted feature list. I really appreciate all your feedback. Do you mind if I add your feedback into the script so that stuff is more clear to others?

", "rawMarkdown": "", "votes": null }, { "id": "58161", "postDate": "11/16/2014 23:12:07", "content": "

Sure. You can add it to the script.

", "rawMarkdown": "", "votes": null }, { "id": "58182", "postDate": "11/17/2014 07:07:49", "content": "

great work is going on here .. thumps up!

", "rawMarkdown": "", "votes": null }, { "id": "58220", "postDate": "11/17/2014 19:50:22", "content": "

[quote=tinrtgu;58054]

\n

I would say it should easily beat 0.40

\n

[/quote]

\n

Anyone tried training this code on the first 9 days and testing on the 10th? Which would be a similar setup to the leaderboard. 

\n

Doing this gets me a score of 0.45 after one epoch, which does not beat a constant CTR prediction (~0.43). Some parameter optimization gets me down to 0.40, but that is still quite high compared to what I would expect from this approach. 

\n

In general, the holdout validation will show very low scores because the samples have strong local correlation (temporally). It is much harder to get a model that generalizes to the next day. That is made even more prominent by using high learning rates, since then the model becomes locally specialized (overfit) to whatever segment of time it is currently learning.

\n

Some ideas to fix it (hope that will help others):

\n

- shuffling the training data (difficult to do in a memory-conscious way)

\n

- prediction calibration

", "rawMarkdown": "", "votes": null }, { "id": "58221", "postDate": "11/17/2014 20:23:55", "content": "

@fchollet: one approach to shuffling which kinda seems to work is shuffle the data as a separate task before commencing actual modelling:

\n

- load the data in R using the ff package (super low memory footprint compared to the file size - direct read.table fails miserably)

\n

- shuffle using ffindexget / ffindexorder

\n

- store using ff wrapper around write.csv

\n\n

Admittedly not very clean, but it does get the job done.

", "rawMarkdown": "", "votes": null }, { "id": "58223", "postDate": "11/17/2014 20:31:28", "content": "

Another idea is to create a shell script that reads the standard input, buffers it (it is a parameter), shuffles and prints to standard output. I've seen such script in one of the Criteo solutions.

", "rawMarkdown": "", "votes": null }, { "id": "58277", "postDate": "11/18/2014 04:45:07", "content": "

[quote=fchollet;58220]

\n

[quote=tinrtgu;58054]

\n

I would say it should easily beat 0.40

\n

[/quote]

\n

Anyone tried training this code on the first 9 days and testing on the 10th? Which would be a similar setup to the leaderboard. 

\n

Doing this gets me a score of 0.45 after one epoch, which does not beat a constant CTR prediction (~0.43). Some parameter optimization gets me down to 0.40, but that is still quite high compared to what I would expect from this approach. 

\n

[/quote]

\n

Weird, I tried to do the same thing as you stated, train with day 1 to 9 and test on day 10, and I can get logloss=0.392 on day 10 with just one epoch.

\n

Below are the parameters that I used.
alpha = .1
beta = 1.
L1 = 1.
L2 = 1.
D = 2 ** 20

\n

EDIT

\n

I have added my code (fast_solution_v3.py) to the original post, without changing any parameters it should train on day 1 to 9 for one epoch and give a validation score of 0.392 on day 10.

", "rawMarkdown": "", "votes": null }, { "id": "58334", "postDate": "11/18/2014 18:48:58", "content": "

v3. without change: LB: 0.6931472

", "rawMarkdown": "", "votes": null }, { "id": "58335", "postDate": "11/18/2014 18:54:40", "content": "

strange - are you sure it's not writing the submission file wrong? as it's the sample submission score

", "rawMarkdown": "", "votes": null }, { "id": "58336", "postDate": "11/18/2014 19:01:27", "content": "

[quote=ACS69;58335]

\n

strange - are you sure it's not writing the submission file wrong? as it's the sample submission score

\n

[/quote]

\n

sure. all predictions generated by v3 are 0.5

", "rawMarkdown": "", "votes": null }, { "id": "58341", "postDate": "11/18/2014 19:59:52", "content": "

[quote=Abhishek;58336]

\n

[quote=ACS69;58335]

\n

strange - are you sure it's not writing the submission file wrong? as it's the sample submission score

\n

[/quote]

\n

sure. all predictions generated by v3 are 0.5

\n

[/quote]

\n

No, your predictions are 0.5 because you are running an untrained model (all predictions are 0, and sigmoid(0) = 0.5 with the sigmoid function used in the code). And you are running an untrained model because of the old date-based code switch that controls when to train the model.

\n

Since the dates in the data have changed, you need to update the code to take into account the new dates.

\n

Quick fix: replace line ~230

\n

date = int(row['hour'][4:6])

\n

with 

\n

date = int(row['hour'][4:6]) - 20

", "rawMarkdown": "", "votes": null }, { "id": "58343", "postDate": "11/18/2014 20:15:30", "content": "

or you can change 

\n

holdafter = 9

\n

to 

\n

holdafter = 29

\n

[quote=fchollet;58341]

\n

[quote=Abhishek;58336]

\n

[quote=ACS69;58335]

\n

strange - are you sure it's not writing the submission file wrong? as it's the sample submission score

\n

[/quote]

\n

sure. all predictions generated by v3 are 0.5

\n

[/quote]

\n

No, your predictions are 0.5 because you are running an untrained model (all predictions are 0, and sigmoid(0) = 0.5 with the sigmoid function used in the code). And you are running an untrained model because of the old date-based code switch that controls when to train the model.

\n

Since the dates in the data have changed, you need to update the code to take into account the new dates.

\n

Quick fix: replace line ~230

\n

date = int(row['hour'][4:6])

\n

with 

\n

date = int(row['hour'][4:6]) - 20

\n

[/quote]

", "rawMarkdown": "", "votes": null }, { "id": "58352", "postDate": "11/18/2014 21:50:33", "content": "

Version 3

\n

hold after 29: validation loss 0.399625, LB 0.3977417

\n

set D to 2**24

", "rawMarkdown": "", "votes": null }, { "id": "58366", "postDate": "11/19/2014 03:54:09", "content": "

Here is the record of my testing on a workstation with Xeon CPU:

\n

python fast_solution_v3.py
Epoch 0 finished, validation logloss: 0.399873, elapsed time: 0:36:26.541000
Epoch 1 finished, validation logloss: 0.400307, elapsed time: 1:12:45.315000
Epoch 2 finished, validation logloss: 0.401073, elapsed time: 1:48:59.706000
Epoch 3 finished, validation logloss: 0.402093, elapsed time: 2:25:13.176000
Epoch 4 finished, validation logloss: 0.403243, elapsed time: 3:01:27.582000
Epoch 5 finished, validation logloss: 0.404274, elapsed time: 3:37:40.116000
Epoch 6 finished, validation logloss: 0.405214, elapsed time: 4:13:53.289000
Epoch 7 finished, validation logloss: 0.406166, elapsed time: 4:50:06.210000
Epoch 8 finished, validation logloss: 0.407127, elapsed time: 5:26:19.867000
Epoch 9 finished, validation logloss: 0.408056, elapsed time: 6:02:41.431000

", "rawMarkdown": "", "votes": null }, { "id": "58380", "postDate": "11/19/2014 07:09:50", "content": "

@Birchwood try pypy? it is much faster

", "rawMarkdown": "", "votes": null }, { "id": "58448", "postDate": "11/19/2014 21:56:45", "content": "

Here is the log of my attempt with tinrtgu's script and a slight tweaking of hyperparameters!

\n

alpha = .15

\n

beta = 1.1

\n

L1 = 1.1

\n

L2 = 1.1

\n

D = 2 ** 22

\n

$ pypy fast_solution_v3.py
Epoch 0 finished, validation logloss: 0.399544, elapsed time: 0:42:58.529000
Epoch 1 finished, validation logloss: 0.399377, elapsed time: 1:35:50.060000

\n

LB score: 0.3992365

", "rawMarkdown": "", "votes": null }, { "id": "58464", "postDate": "11/20/2014 00:35:50", "content": "

[quote=binga;58448]

\n

Here is the log of my attempt with tinrtgu's script and a slight tweaking of hyperparameters!

\n

alpha = .15

\n

beta = 1.1

\n

L1 = 1.1

\n

L2 = 1.1

\n

D = 2 ** 22

\n

$ pypy fast_solution_v3.py
Epoch 0 finished, validation logloss: 0.399544, elapsed time: 0:42:58.529000
Epoch 1 finished, validation logloss: 0.399377, elapsed time: 1:35:50.060000

\n

LB score: 0.3992365

\n

[/quote]

\n

Just a friendly reminder: it is not a good idea of sharing fine-tuned parameters before the end of the competition.

", "rawMarkdown": "", "votes": null }, { "id": "58497", "postDate": "11/20/2014 07:11:20", "content": "

[quote=phunter;58464]

\n

[quote=binga;58448]

\n

Here is the log of my attempt with tinrtgu's script and a slight tweaking of hyperparameters!

\n

alpha = .15

\n

beta = 1.1

\n

L1 = 1.1

\n

L2 = 1.1

\n

D = 2 ** 22

\n

$ pypy fast_solution_v3.py
Epoch 0 finished, validation logloss: 0.399544, elapsed time: 0:42:58.529000
Epoch 1 finished, validation logloss: 0.399377, elapsed time: 1:35:50.060000

\n

LB score: 0.3992365

\n

[/quote]

\n

Just a friendly reminder: it is not a good idea of sharing fine-tuned parameters before the end of the competition.

\n

[/quote]

\n

I got that score by just running the code, which I believe anybody who is interested in solving could do! As we have a lot of time to the end of competition, this score will really not matter, as I am somewhere at 60th on the LB. Anyways, if the guys here are still insecure about it, I'll keep good things to myself. Never mind.

", "rawMarkdown": "", "votes": null }, { "id": "58514", "postDate": "11/20/2014 11:49:39", "content": "

It's going to take a lot more than just optimizing the parameters to get into the top 10.

", "rawMarkdown": "", "votes": null }, { "id": "58522", "postDate": "11/20/2014 13:45:32", "content": "

I just downloaded and tried pypy, it really is a lot faster!

\n

on my machine (v3, D = 2 ** 24, holdafter=29)

\n

pypy: Epoch 0 finished, validation logloss: 0.399601, elapsed time: 0:16:30.555000

\n

python: Epoch 0 finished, validation logloss: 0.399601, elapsed time: 0:56:32.461000

", "rawMarkdown": "", "votes": null }, { "id": "59144", "postDate": "11/28/2014 08:13:38", "content": "

Hello all,

\n

I have worked on the fast_solution great piece of code and added a few features:

\n

- parameter control from command line

\n

- dropout: where you randomly drop features out of consideration - prevents overfitting and features adaptation

\n

- inclusion of the # of samples seen per device_ip / device_id as a feature

\n

- inclusion of weedkay as a feature

\n

I am down to roughly 0.3948 on the LB with a lot of tuning, by running on all test samples - with a few hours of processing using pypy on an i7, using a few GB's of RAM. LB is very close to my own cross-validation (run on all days except 30, test on 30).

\n

The code probably would benefit from some optimization. I have written the first version in Julia - it was faster than C-Python, but 2-3 times slower than pypy, and I quickly reverted to Python.

\n

I would be glad of any constructive feedback.

\n

Yannick

", "rawMarkdown": "", "votes": null }, { "id": "59146", "postDate": "11/28/2014 08:22:37", "content": "

@Yannick Martel

\n

Glad to see you adding new features to the benchmark code, and I really like your idea of using dropout.

", "rawMarkdown": "", "votes": null }, { "id": "59149", "postDate": "11/28/2014 09:50:04", "content": "

[quote=Yannick Martel;59144]

\n

- inclusion of the # of samples seen per device_ip / device_id as a feature

\n

[/quote]

\n

For the counters, I think we should use defaultdict, then we don't have to catch KeyError exception. For instance, just replace:

\n

device_ip_counter = {}
device_id_counter = {}

\n

by

\n

device_ip_counter = defaultdict(int)
device_id_counter =  defaultdict(int)

", "rawMarkdown": "", "votes": null }, { "id": "59200", "postDate": "11/29/2014 04:19:39", "content": "

Thanks Yannick.. curious to know how the dropout is impacting your accuracy...

\n

[quote=Yannick Martel;59144]

\n

Hello all,

\n

I have worked on the fast_solution great piece of code and added a few features:

\n

- parameter control from command line

\n

- dropout: where you randomly drop features out of consideration - prevents overfitting and features adaptation

\n

- inclusion of the # of samples seen per device_ip / device_id as a feature

\n

- inclusion of weedkay as a feature

\n

I am down to roughly 0.3948 on the LB with a lot of tuning, by running on all test samples - with a few hours of processing using pypy on an i7, using a few GB's of RAM. LB is very close to my own cross-validation (run on all days except 30, test on 30).

\n

The code probably would benefit from some optimization. I have written the first version in Julia - it was faster than C-Python, but 2-3 times slower than pypy, and I quickly reverted to Python.

\n

I would be glad of any constructive feedback.

\n

Yannick

\n

[/quote]

", "rawMarkdown": "", "votes": null }, { "id": "59210", "postDate": "11/29/2014 09:04:25", "content": "

@Yannick

\n

Could you give an example (not your optimal parameter) on how to run this script. I'm stuck with the command line parameters.

\n

I already have all the stuff installed...

\n

Thks

\n

[quote=Yannick Martel;59144]

\n

Hello all,

\n

I have worked on the fast_solution great piece of code and added a few features:

\n

- parameter control from command line

\n

- dropout: where you randomly drop features out of consideration - prevents overfitting and features adaptation

\n

- inclusion of the # of samples seen per device_ip / device_id as a feature

\n

- inclusion of weedkay as a feature

\n

I am down to roughly 0.3948 on the LB with a lot of tuning, by running on all test samples - with a few hours of processing using pypy on an i7, using a few GB's of RAM. LB is very close to my own cross-validation (run on all days except 30, test on 30).

\n

The code probably would benefit from some optimization. I have written the first version in Julia - it was faster than C-Python, but 2-3 times slower than pypy, and I quickly reverted to Python.

\n

I would be glad of any constructive feedback.

\n

Yannick

\n

[/quote]

", "rawMarkdown": "", "votes": null }, { "id": "59216", "postDate": "11/29/2014 10:40:15", "content": "

@all

\n

Yes, the script is not as user-friendly as I would have liked :-)

\n

You can have some help output by running:

\n

python fast_solution_plus.py --help

\n

Then to learn you do something like:

\n

  python fast_solution_plus.py train --train <training csv file> -o <output model file>

\n

[ like in: python fast_solution_plus.py train --train train.csv -o first.model.gz ]

\n

Once your model is trained, you can predict:

\n

  python fast_solution_plus.py predict --test test.csv -i first.model.gz -p my.submission.csv.gz

\n

You can:

\n

- usually provide a .csv.gz instead of .csv only - it decompress on the fly

\n

- use pypy instead of python

\n

- provide additionnal tuning parameters: --L1 1.0 --L2 0.1 --dropout 0.4, and so on (you get the list when running with -h or --help)

\n

Have fun!

\n

@ Patrick: on my experiment, dropout does reduce indeed overfit and improve logloss. I get very similar figures between my own cross-validation and the LB.

\n

With dropout, I recommend to run multiple passes (--n_epochs) to avoid "loosing" some pieces of samples, depending on the dropout factor.

\n\n

Yannick

", "rawMarkdown": "", "votes": null }, { "id": "59235", "postDate": "11/29/2014 18:07:42", "content": "

Did anyone have any luck with feature interactions? It seems so attractive to add them, but howevery I try, they do not give any improvement.

", "rawMarkdown": "", "votes": null }, { "id": "59252", "postDate": "11/30/2014 02:34:18", "content": "

I'm not sure I understand how the dropout parameter is working in this new code. It seems to me that meaningful values should be between 0 and 1, but when I was doing parameter tuning I found that the logloss changed continuously even for values above dropout=1 (I went as high as 1.4, and the logloss just kept smoothly changing).

\n

When I looked at the code, it seems that 1 is treated as a special case, but that anything above 1 should be identical (drop all features, I guess, so I should get nonsense or at least just the logloss associated with the mean ctr).

\n

Any idea what's going on here?

", "rawMarkdown": "", "votes": null }, { "id": "59253", "postDate": "11/30/2014 03:28:57", "content": "

Here's my understanding:

\n\n

If dropout > 1 you don't drop anything (random.random() > dropout is always False), but you do divide wTx by dropout because the dropped list is [] instead of None (if dropped != None: wTx /= dropout).

\n\n

Thus you are using your regular model, except you're tweaking wTx on each prediction.

", "rawMarkdown": "", "votes": null }, { "id": "59256", "postDate": "11/30/2014 05:25:43", "content": "

Ah, I see, that makes sense then, thanks!

", "rawMarkdown": "", "votes": null }, { "id": "59283", "postDate": "11/30/2014 19:01:16", "content": "

Can you, guys, please explain how you do cross-validation with the code? As I understood the code calculates logloss for each 100th (default setting) sample eg 100,200,.... At the end it prints something like average logloss for all samles that were not used for training. How accurate is it, I mean how close is it to LB score? What should I change to train it on let's say all days except the 30th day and then test it  on the 30th day (as was described by Yannick) eg change holdout pattern from samples to days? My PC is very old but I am trying to do my best with it. It runs 1 epoch for several hours with PYPY. Maybe I could also try to reduce training size eg train on just 5-10 days and test it on any single day that was not used for training.

\n

Thanks

", "rawMarkdown": "", "votes": null }, { "id": "59289", "postDate": "11/30/2014 19:29:37", "content": "

If you are using the original script posted by tinrtgu, you need to set the holdafter parameter to wherever you want to end your training (eg. 29), and set holdout to None.

\n\n

If your computer is old, you might want to split the training data in one file per day and run the script on a subset of the training data. This won't be as accurate but being able to run a lot of experiment quickly is definitely a big advantage.

\n\n

If your computer has enough RAM, you might also want to run several experiments in parallel.

\n\n

Personally I currently train on 21-27, then independently validate on 28, 29 and 30, and print the logloss on each validation day and the average. It's nowhere close to my LB score because my classifier's score varies widely depending on the CTR of the day, and I still haven't figured out how to fix that. But overall if an experiment improves my validation score, it also improves my LB score.

", "rawMarkdown": "", "votes": null }, { "id": "59372", "postDate": "12/02/2014 03:28:47", "content": "

I've been able to go down to 0.376 in validation with interactions, but LB=0.41 ...

", "rawMarkdown": "", "votes": null }, { "id": "59441", "postDate": "12/03/2014 05:21:57", "content": "

[quote=TanoPereira;59372]

\n

I've been able to go down to 0.376 in validation with interactions, but LB=0.41 ...

\n

[/quote]

\n

Just a hint, overfitting is more severe with interactions so it would be a good idea to increase regularizations.

", "rawMarkdown": "", "votes": null }, { "id": "59472", "postDate": "12/03/2014 16:06:40", "content": "

I've run this code with pypy on Ubuntu. I wanted to run it on my laptop, W7 64-bit with Python 64 bit, but it seems like pypy will work only with 32-bit Python...

\n

Has anyone ran this on a 64 bit Python on a Windows machine?

", "rawMarkdown": "", "votes": null }, { "id": "59474", "postDate": "12/03/2014 16:11:56", "content": "

[quote=Giulio;59472]

\n

I've run this code with pypy on Ubuntu. I wanted to run it on my laptop, W7 64-bit with Python 64 bit, but it seems like pypy will work only with 32-bit Python...

\n

Has anyone ran this on a 64 bit Python on a Windows machine?

\n

[/quote]

\n

That's why I switched to Numba (using the Anaconda distribution). It's at least as fast, but also gives me access to my entire system memory.

", "rawMarkdown": "", "votes": null }, { "id": "59483", "postDate": "12/03/2014 17:35:10", "content": "

Can you provide pointers on how to use numba?

\n

I have tried but get compilation errors all over the place...

\n

Thanks!

", "rawMarkdown": "", "votes": null }, { "id": "59488", "postDate": "12/03/2014 18:30:50", "content": "

[quote=TanoPereira;59483]

\n

Can you provide pointers on how to use numba?

\n

I have tried but get compilation errors all over the place...

\n

Thanks!

\n

[/quote]

\n

I had the same issue. I had to "de-object-orient" the code to make it work. In other words, I took the code out of the Class definition.

\n

Sometimes it can be tricky to get the decorators right. You can use these to get you started. (You'll need to modify depending on how you define your dtypes.)

\n

from numba import int32, float32, float64, void, jit

\n

@jit(float64(int32[:],float32[:],float32[:],float32[:]))
def predict(x, w, z, n):
...

\n

@jit(void(int32[:],float32[:],float32[:],float32[:],float64,float64))
def update(x, w, z, n, p, y):
...
@jit(float64(float64,float64))
def logloss(p, y):

\n\n

Also, you need to change your lists (x, w, z, n) to arrays, e.g.,

\n

w = np.zeros(D, dtype=np.float32)

\n\n

Let me know if you have any other issues. I can assure you, it's worth the time to figure it out.

", "rawMarkdown": "", "votes": null }, { "id": "59493", "postDate": "12/03/2014 20:40:02", "content": "

For my information, how efficient is numba as compared to Pypy? How long does it take for one pass over the full training set?

\n

On my own i7 Linux PC with Pypy, the code provided on the forum takes around 2:30.

", "rawMarkdown": "", "votes": null }, { "id": "59497", "postDate": "12/03/2014 21:05:26", "content": "

[quote=Yannick Martel;59493]

\n

For my information, how efficient is numba as compared to Pypy? How long does it take for one pass over the full training set?

\n

On my own i7 Linux PC with Pypy, the code provided on the forum takes around 2:30.

\n

[/quote]

\n

I don't have a paired comparison for this code, but I ran the (very similar) benchmark code in the Tradeshift challenge.

\n

Results were:

\n", "rawMarkdown": "", "votes": null }, { "id": "59499", "postDate": "12/03/2014 21:21:51", "content": "

Can you run the python script with numba in command line: python predict.py?

", "rawMarkdown": "", "votes": null }, { "id": "59501", "postDate": "12/03/2014 22:01:08", "content": "

[quote=Birchwood;59499]

\n

Can you run the python script with numba in command line: python predict.py?

\n

[/quote]

\n

No, you need the special Numba IDE. 

", "rawMarkdown": "", "votes": null }, { "id": "59511", "postDate": "12/04/2014 01:21:11", "content": "

Thank you inversion!

\n

I was able to get the code out of the class, however I get no speed up at all :(

\n

I must be doing something really wrong. My Python abilities are really scarcer than I'd thought.

", "rawMarkdown": "", "votes": null }, { "id": "59515", "postDate": "12/04/2014 03:22:41", "content": "

Thanks for the pointers, inversion!

\n

I also had issues with Numba/NumbaPro - I've found that it negatively impacted performance.  I'm using the 64-bit version of PyPy, however, so the comparison is likely different.  So far it's been significantly faster than a numpy rewrite and Numba/NumbaPro.  I'm considering moving the code to Theano or C++ just for fun... we'll see how it compares.

", "rawMarkdown": "", "votes": null }, { "id": "59523", "postDate": "12/04/2014 09:41:09", "content": "

If 'train.csv' is not comma separated, It raises an error. Any hints?

\n

[quote=Yannick Martel;59216]

\n

@all

\n

....

\n

[ like in: python fast_solution_plus.py train --train train.csv -o first.model.gz ]

\n

...

\n

Yannick

\n

[/quote]

", "rawMarkdown": "", "votes": null }, { "id": "59526", "postDate": "12/04/2014 09:46:49", "content": "

[quote=Herimanitra;59523]

\n

If 'train.csv' is not comma separated, It raises an error. Any hints?

\n

[quote=Yannick Martel;59216]

\n

@all

\n

....

\n

[ like in: python fast_solution_plus.py train --train train.csv -o first.model.gz ]

\n

...

\n

Yannick

\n

[/quote]

\n

[/quote]

\n\n

Use: csv.DictReader(f, delimiter='\\t', quoting=csv.QUOTE_NONE)

", "rawMarkdown": "", "votes": null }, { "id": "59530", "postDate": "12/04/2014 10:36:47", "content": "

Thanks Abishek,

\n

In my case, DictReader(f_train, delimiter=';')  works!

\n

The script should allow user to bring their own features  into the algorithm, one hot encoding should be an option :)

\n

[quote=Abhishek;59526]

\n

[quote=Herimanitra;59523]

\n

If 'train.csv' is not comma separated, It raises an error. Any hints?

\n

[quote=Yannick Martel;59216]

\n

@all

\n

....

\n

[ like in: python fast_solution_plus.py train --train train.csv -o first.model.gz ]

\n

...

\n

Yannick

\n

[/quote]

\n

[/quote]

\n

Use: csv.DictReader(f, delimiter='\\t', quoting=csv.QUOTE_NONE)

\n

[/quote]

", "rawMarkdown": "", "votes": null }, { "id": "59534", "postDate": "12/04/2014 11:41:47", "content": "

Yes, there are very strong assumptions on the file being similar to the train.csv and test.csv provided for the competition: comma separated, with "ID", "click" and "hour" columns.

\n

All features are accepted, and they are all treated as categorical via feature hashing.

", "rawMarkdown": "", "votes": null }, { "id": "59538", "postDate": "12/04/2014 12:04:58", "content": "

should be an option then as people like me :) may already have numerical features ready

\n

for training.

\n

Something like:

\n

pypy fast_solution_plus.py train --train train.csv -o train_model.gz  --train_as_is

\n

[quote=Yannick Martel;59534]

\n

All features are accepted, and they are all treated as categorical via feature hashing.

\n

[/quote]

", "rawMarkdown": "", "votes": null }, { "id": "59542", "postDate": "12/04/2014 13:09:02", "content": "

Yes, you are right, it would be a nice enhancement, to use with some new features - engineered features probably as the raw features are more categorical to me.

\n

We would need to have as argument to --train_as_is the list of features, as you probably want to mix:

\n

pypy fast_solution_plus.py train --train train.csv -o train_model.gz --train_as_is feature1,feature2

", "rawMarkdown": "", "votes": null }, { "id": "59554", "postDate": "12/04/2014 17:37:32", "content": "

Does anyone know how I might be able to get this to work in terminal on a Mac with both Python 2.7.8 and Pypy installed? I've been trying but all I've been able to do is get the 'id' and 'click' columns in my submission file. (I'm fairly new to python, so any assistance would be greatly appreciated! Just trying to learn!)

\n

I've been noticing a lot of indentation errors come up in terminal. Could this warrant any changes that might help be create a submission file?

", "rawMarkdown": "", "votes": null }, { "id": "59595", "postDate": "12/05/2014 04:07:11", "content": "

I am wondering why the values of self.n and self.z are not updated? 

\n

are they updated automatically after n and z are updated?

\n\n\n

[quote=tinrtgu;58054]

\n

Notice (11/18/2014): added the third version of the script fast_solution_v3.py

\n

Notice (11/20/2014): here is a post on how to get 0.3977417 on the public leaderboard

\n

-

\n

Dear fellow Kagglers, tin-ert-gu here.

\n

-

\n

This is a python implementation of adaptive learning rate L1 and L2 regularized logistic regression using hash-trick one-hot encoding. This algorithm is also used by Google for their online CTR prediction system.
All credits of this script goes to their hard work of research.

\n

-

\n

How to run?

\n

In your terminal, simply type

\n

python fast_solution.py

\n

The script is also python3 compatible, so you can use python3 as well

\n

python3 fast_solution.py

\n

However, since the code only depends on native modules, it is highly suggested that you run the script with pypy for a huge speed-up. To use the script with pypy on Ubuntu, first type

\n

sudo apt-get install pypy

\n

Then use the pypy interpreter to run the script

\n

pypy fast_solution.py

\n

-

\n

Changelog over the previous version of the script

\n
    \n
  1. Added L1 regularization
    2. \n
  2. Added L2 regularization
    3. \n
  3. Build in multiple epoch training
    4. \n
  4. Build in holdout set validation support
    5. \n
  5. Build in support for poly 2 feature interactions
    6. \n
  6. Re-factored code for better extendibility
    7. \n
  7. EVEN MORE COMMENTS!
    8. \n
\n

-

\n

Performance

\n

Training time for a single epoch, on a single Intel(R) Xeon(R) CPU E5-2630L v2 @ 2.40GHz:

\n\n

Memory usage:

\n

This mainly depends on the D parameter in the script.

\n\n

-

\n

The entire algorithm is disclosed, while using only out of the box modules provided by Python, so it should also be an easy sandbox code to build new ideas upon.

\n

Good luck and have fun!

\n

-

\n

@Abhishek:

\n

Can't submit right now, so I don't know the leader-board score, but I would say it should easily beat 0.40

\n

-

\n

EDIT 1

\n

add fast_solution_v2.py to fix a null reference bug under memory saving mode

\n

EDIT 2

\n

add fast_solution_v3.py

\n

A great thanks for Paweł's feedback, the following changes are made as a result

\n\n

By considering fchollet's experiment

\n\n

This should be the last time I modify this benchmark for this competition. Hope the actual data will get release soon, and happy predicting!

\n

[/quote]

", "rawMarkdown": "", "votes": null }, { "id": "59635", "postDate": "12/05/2014 18:40:47", "content": "

[quote=Ying;59595]

\n

I am wondering why the values of self.n and self.z are not updated? 

\n

are they updated automatically after n and z are updated?

\n

[/quote]

\n

It's Python. So n and z are just references to self.n and self.z. They are not copies! So when you update n you actually update self.n.

", "rawMarkdown": "", "votes": null }, { "id": "59680", "postDate": "12/06/2014 15:49:18", "content": "

If I remove this piece of code:

\n

# build x
x = [0] # 0 is the index of the bias term
for key in row:
value = row[key]

\n

# one-hot encode everything with hash trick
index = abs(hash(key + '_' + value)) % D
x.append(index)

\n

Should I expect that  x will be treated as is?

", "rawMarkdown": "", "votes": null }, { "id": "59705", "postDate": "12/07/2014 01:53:24", "content": "

I keep getting the error shown in the image attached. Is there a particular 'train' and 'test' file I should be downloading? Or is there a special place I should save them to?

", "rawMarkdown": "", "votes": null }, { "id": "59709", "postDate": "12/07/2014 03:03:07", "content": "

[quote=PBoswell;59705]

\n

I keep getting the error shown in the image attached. Is there a particular 'train' and 'test' file I should be downloading? Or is there a special place I should save them to?

\n

[/quote]

\n

Modify line 32 to 34 of the script to the correct file path.

", "rawMarkdown": "", "votes": null }, { "id": "59710", "postDate": "12/07/2014 03:06:56", "content": "

[quote=Herimanitra;59680]

\n

If I remove this piece of code:

\n

# build x
x = [0] # 0 is the index of the bias term
for key in row:
value = row[key]

\n

# one-hot encode everything with hash trick
index = abs(hash(key + '_' + value)) % D
x.append(index)

\n

Should I expect that  x will be treated as is?

\n

[/quote]

\n

I think an exception will be raised.

", "rawMarkdown": "", "votes": null }, { "id": "59715", "postDate": "12/07/2014 04:25:09", "content": "

seems I figure out ( the script run without problem, x seems to be treated as is):

\n

# turn hour really into hour, it was originally YYMMDDHH

\n

#date = row['hour'][0:6]
#row['hour'] = row['hour'][6:]

\n

# stderr.write("_%s_" % date)

\n

#if dayfilter != None and not date in dayfilter:
#continue

\n

#if dayfeature:
## extract date
#row['wd'] = str(int(date) % 7)
#row['wd_hour'] = "%s_%s" % (row['wd'], row['hour'])

\n

#if counters:
#d_ip = row['device_ip']
#d_id = row ["device_id"]
#try:
#device_ip_counter [d_ip] += 1
#device_id_counter [d_id] += 1
#except KeyError:
#device_ip_counter [d_ip] = 1
#device_id_counter [d_id] = 1
#row["ipc"] = str(min(device_ip_counter[d_ip], 8))
#row["idc"] = str(min(device_id_counter[d_id], 8))

\n

# build x
x = [0] # 0 is the index of the bias term
for key in row:
value = row[key]

\n

## one-hot encode everything with hash trick
#index = abs(hash(key + '_' + value)) % D
#x.append(index)

\n

Could you confirm?

", "rawMarkdown": "", "votes": null }, { "id": "59724", "postDate": "12/07/2014 06:36:39", "content": "

can someone point me in the right direction for resources on some of these parameters (i.e. regularization, beta, D parameter).

\n

Also what is the verdict on learning rate. Is it truly the (s)lower the better?

", "rawMarkdown": "", "votes": null }, { "id": "59738", "postDate": "12/07/2014 17:15:52", "content": "

This paper might help:

\n

L1 L2 Paper

\n

The beta parameter to my understanding doesn't need much tuning since it becomes nearly inconsequential after many samples. It's really just used for smoothing in the first few occurrences of a value. As for D, generally larger is better since you get fewer collisions but a smaller D also serves as dimension reduction so that will need some tinkering around with.

\n

As for the learning rate, like you're implying, a lower rate slows convergence but can prevent oscillation of the weight vector when you choose a larger learning rate. Again, probably another parameter to tinker. From what I've seen in literature, 0.1 is a good place to start.

\n

Cheers!

", "rawMarkdown": "", "votes": null }, { "id": "59753", "postDate": "12/08/2014 00:57:23", "content": "

@tinrtgu - Thank you for writing this beautiful piece of code - very clean & and is easy to read.

\n

I have been using the code, and tweaking the hyper-parameters manually.  I am curious if there is a method to going about doing this - does one select the hyper-parameters by brute force, or is there some intuition/methodology that goes behind selecting the optimal parameters?  

\n

If there are any papers on this, I would love to read them!

\n

--update:  I didn't see the post above this one.

", "rawMarkdown": "", "votes": null }, { "id": "59760", "postDate": "12/08/2014 04:35:37", "content": "

[quote=sneakyfox;59753]

\n

@tinrtgu - Thank you for writing this beautiful piece of code - very clean & and is easy to read.

\n

I have been using the code, and tweaking the hyper-parameters manually.  I am curious if there is a method to going about doing this - does one select the hyper-parameters by brute force, or is there some intuition/methodology that goes behind selecting the optimal parameters?  

\n

If there are any papers on this, I would love to read them!

\n

--update:  I didn't see the post above this one.

\n

[/quote]

\n

Yeah, the post above you is pretty nice.

\n

You don't really need to tune \\beta, just leave it alone at 1 is good enough. From my experience, so please take it with a grain of salt, without feature interactions the most important parameter is the learning rate, and with interactions L1 and L2 are important.

\n

As how to find the optimal parameters, I just use the old fashion grid search with validation.

", "rawMarkdown": "", "votes": null }, { "id": "59771", "postDate": "12/08/2014 09:51:11", "content": "

[quote=Yannick Martel;59144]

\n

Hello all,

\n

I have worked on the fast_solution great piece of code and added a few features:

\n

- parameter control from command line

\n

- dropout: where you randomly drop features out of consideration - prevents overfitting and features adaptation

\n

- inclusion of the # of samples seen per device_ip / device_id as a feature

\n

- inclusion of weedkay as a feature

\n

I am down to roughly 0.3948 on the LB with a lot of tuning, by running on all test samples - with a few hours of processing using pypy on an i7, using a few GB's of RAM. LB is very close to my own cross-validation (run on all days except 30, test on 30).

\n

The code probably would benefit from some optimization. I have written the first version in Julia - it was faster than C-Python, but 2-3 times slower than pypy, and I quickly reverted to Python.

\n

I would be glad of any constructive feedback.

\n

Yannick

\n

[/quote]

\n\n

Using Yannick's script, I get the error "too few arguments". Any ideas why?

", "rawMarkdown": "", "votes": null }, { "id": "59782", "postDate": "12/08/2014 13:04:05", "content": "

You need to provide one argument "train" or "predict" to determine what the script will do, plus some more in the forme -o model.gz for the arguments.

\n

For instance:

\n

> pypy fast_solution_plus.py train --train train.csv.gz -o model.gz

\n

> pypy fast_solution_plus.py predict --test test.csv.gz -i model.gz -p predict.gz

\n

You can also do

\n

> pypy fast_solution_plus.py --help

\n

to get more information.

", "rawMarkdown": "", "votes": null }, { "id": "59800", "postDate": "12/08/2014 17:06:07", "content": "

Yannick,

\n

Thank you for the quick response. A couple more questions, if you don't mind. Is this model training on 9 days and validating on 10th? I noticed the fast_solution_v3.py script includes a D parameter; do you have an equivalent or what is the default? Also, what does the 'holdout' feature do (in the other script it is defaulted at 0?

\n\n

Thanks!

", "rawMarkdown": "", "votes": null }, { "id": "59801", "postDate": "12/08/2014 17:10:50", "content": "

On another note, I keep getting the following error when running Yannick's 'fast_solution_plus.py' script (attached photo)

", "rawMarkdown": "", "votes": null }, { "id": "59802", "postDate": "12/08/2014 17:15:34", "content": "

@PBoswell:

\n

On your first question, the logic is a bit different from fast_solution_v3.py: I validate on 1 per 100 (with holdout = 100), and not on a separate day. I recommend to train on n days and validate on n+1, but you have to script if outside of the python executable.

\n

The D parameters is indirect: you can specify --bits 23 on the command line to get D=2**23.

\n

On the memory problem, it seems that gzip is very memory hungry when you save the model. Maybe try first on a .model file, not a .model.gz. I have 16GB on my PC, so it works, but it might not on a smaller machine.

\n

Any idea to get an efficient serialisation plus compression of Python data structure on disk is welcome! Plucker plus gzip is reasonably simple, but not so fast nor so efficient on disk.

", "rawMarkdown": "", "votes": null }, { "id": "59806", "postDate": "12/08/2014 17:53:50", "content": "

You are awesome, Yannick!

", "rawMarkdown": "", "votes": null }, { "id": "59808", "postDate": "12/08/2014 18:09:09", "content": "

More of a high level discussion I want to start here:

\n\n

Has anyone been able to beat .397 with these scripts? That is, is the underlying modeling used here limited? Or do you really believe with parameter tuning you could reach #1 on LB?

\n\n

From my experience, increasing L1 has consistently improved my LB score, but with diminishing returns. Changing learning rate either way does not seem to help, but maybe I need to change learning rate ALONG WITH something else?

", "rawMarkdown": "", "votes": null }, { "id": "59809", "postDate": "12/08/2014 18:14:15", "content": "

[quote=PBoswell;59808]

\n

Has anyone been able to beat .397 with these scripts? 

\n

[/quote]

\n

The default parameters give .395 if you train on the whole data.

", "rawMarkdown": "", "votes": null }, { "id": "59811", "postDate": "12/08/2014 18:23:35", "content": "

you mean train on all data by making holdafter = None/10?

", "rawMarkdown": "", "votes": null }, { "id": "59812", "postDate": "12/08/2014 19:15:25", "content": "

[quote=PBoswell;59808]

\n

Has anyone been able to beat .397 with these scripts? 

\n

[/quote]

\n

Using this model, I've reached a 0.3915 LB score, and still making fairly steady progress.

", "rawMarkdown": "", "votes": null }, { "id": "59813", "postDate": "12/08/2014 19:27:27", "content": "

Well I don't know what you guys are doing, because the default parameters yield me 0.3976439 when not changing any parameters.

", "rawMarkdown": "", "votes": null }, { "id": "59821", "postDate": "12/08/2014 20:18:00", "content": "

How do you train on the whole data?

", "rawMarkdown": "", "votes": null }, { "id": "59826", "postDate": "12/08/2014 20:30:38", "content": "

[quote=inversion;59812]

\n

[quote=PBoswell;59808]

\n

Has anyone been able to beat .397 with these scripts? 

\n

[/quote]

\n

Using this model, I've reached a 0.3915 LB score, and still making fairly steady progress.

\n

[/quote]

\n

Without any feature engineering? 

", "rawMarkdown": "", "votes": null }, { "id": "59828", "postDate": "12/08/2014 20:36:22", "content": "

[quote=clustifier;59826]

\n

[quote=inversion;59812]

\n

[quote=PBoswell;59808]

\n

Has anyone been able to beat .397 with these scripts? 

\n

[/quote]

\n

Using this model, I've reached a 0.3915 LB score, and still making fairly steady progress.

\n

[/quote]

\n

Without any feature engineering? 

\n

[/quote]

\n

Also just to clarify, are we talking about the original 'fast_solution_v3' script, or Yannick's 'fast_solution_plus'?

", "rawMarkdown": "", "votes": null }, { "id": "59829", "postDate": "12/08/2014 20:44:47", "content": "

[quote=PBoswell;59828]

\n

[quote=clustifier;59826]

\n

[quote=inversion;59812]

\n

[/quote]

\n

Using this model, I've reached a 0.3915 LB score, and still making fairly steady progress.

\n

Without any feature engineering? 

\n

[/quote]

\n

Also just to clarify, are we talking about the original 'fast_solution_v3' script, or Yannick's 'fast_solution_plus'?

\n

[/quote]

\n

I'm pretty much using v1 (I had to modify it to make it work with numba), I'm not doing anything special with feature engineering other than brute-force interactions.

", "rawMarkdown": "", "votes": null }, { "id": "59830", "postDate": "12/08/2014 20:45:43", "content": "

so you think v1 is better than v3?

", "rawMarkdown": "", "votes": null }, { "id": "59831", "postDate": "12/08/2014 20:49:43", "content": "

[quote=PBoswell;59830]

\n

so you think v1 is better than v3?

\n

[/quote]

\n

I don't know what to think. Probably best to try different methods. There are lots of ways to approach the problem.

", "rawMarkdown": "", "votes": null }, { "id": "59851", "postDate": "12/09/2014 00:49:41", "content": "

With no feature engineering I'm at 0.3931 LB with Yannick's version of the script, but the runs take about 6 hours and maybe 4-6GB of RAM. Looks like I've still got some parameter optimization to do.

", "rawMarkdown": "", "votes": null }, { "id": "59858", "postDate": "12/09/2014 02:08:57", "content": "

[quote=Nicholas Guttenberg;59851]

\n

With no feature engineering I'm at 0.3931 LB with Yannick's version of the script, but the runs take about 6 hours and maybe 4-6GB of RAM. Looks like I've still got some parameter optimization to do.

\n

[/quote]

\n\n

do you mind sharing what droupout you used?

", "rawMarkdown": "", "votes": null }, { "id": "59859", "postDate": "12/09/2014 02:30:45", "content": "

I haven't actually optimized the dropout yet, so I'm just using whatever the default is.

", "rawMarkdown": "", "votes": null }, { "id": "59861", "postDate": "12/09/2014 04:33:19", "content": "

I don't know what is going on, but I continuously get a 'MemoryError' when running 'fast_solution_plus.py' when trying to save the output model.

\n

I am running on 8GB of RAM. I monitored the usage during process and never went above 50% memory usage.

\n

Anyone have any ideas?

", "rawMarkdown": "", "votes": null }, { "id": "59862", "postDate": "12/09/2014 04:44:42", "content": "

[quote=PBoswell;59861]

\n

I don't know what is going on, but I continuously get a 'MemoryError' when running 'fast_solution_plus.py' when trying to save the output model.

\n

I am running on 8GB of RAM. I monitored the usage during process and never went above 50% memory usage.

\n

Anyone have any ideas?

\n

[/quote]

\n

As Yannick mentioned, one suggestion could be to save the model without gzip. I guess it should work.

", "rawMarkdown": "", "votes": null }, { "id": "59863", "postDate": "12/09/2014 04:47:44", "content": "

[quote=binga;59862]

\n

[quote=PBoswell;59861]

\n

I don't know what is going on, but I continuously get a 'MemoryError' when running 'fast_solution_plus.py' when trying to save the output model.

\n

I am running on 8GB of RAM. I monitored the usage during process and never went above 50% memory usage.

\n

Anyone have any ideas?

\n

[/quote]

\n

As Yannick mentioned, one suggestion could be to save the model without gzip. I guess it should work.

\n

[/quote]

\n

I have tried both. Both yield the same error.

", "rawMarkdown": "", "votes": null }, { "id": "59864", "postDate": "12/09/2014 05:00:09", "content": "

It just means that 8GB RAM isnt sufficient for the operation. You could keep an eye on system memory and check which chunk of code is consuming all the RAM

", "rawMarkdown": "", "votes": null }, { "id": "59865", "postDate": "12/09/2014 05:11:48", "content": "

so it seems that despite the error, it is saving 'model.gz' to my directory. However, when I try to predict I get:

\n

'IOerror: CRC Check failed'

\n

What does that mean!?

", "rawMarkdown": "", "votes": null }, { "id": "59869", "postDate": "12/09/2014 07:26:45", "content": "

It means that when you tried saving your model during the train operation, if you have encountered the memory error, even then the model.gz is saved to your disk but it is corrupted. And, that is why you encounter a CRC check failed error while unpacking the model.gz during predict operation.

", "rawMarkdown": "", "votes": null }, { "id": "59872", "postDate": "12/09/2014 07:59:35", "content": "

Here's a turnaround. You could save the file without gzip-ping it.

\n

Modify write_learner function to - with open(model_save, "wb") as model_file:

\n

& modify the load_learner function to - with open(model_save, "rb") as model_file:

\n

It should work now!

", "rawMarkdown": "", "votes": null }, { "id": "59889", "postDate": "12/09/2014 15:20:35", "content": "

@inversion I am trying to rewrite the script for numba. I am a bit confused how to treat _indices: how to modify it and call it from the predict function. Could You please give an advice?

", "rawMarkdown": "", "votes": null }, { "id": "59891", "postDate": "12/09/2014 15:52:00", "content": "

[quote=Matfyzak;59889]

\n

@inversion I am trying to rewrite the script for numba. I am a bit confused how to treat _indices: how to modify it and call it from the predict function. Could You please give an advice?

\n

[/quote]

\n

Oh, right, this tripped me up to. I just ended up getting rid of it and pulling it into the data subroutine (see attached). And then, instead of

\n

for i in self._indices(x):

\n

I use

\n

for i in x:

\n\n

Make sense?

", "rawMarkdown": "", "votes": null }, { "id": "59899", "postDate": "12/09/2014 19:50:09", "content": "

I am pretty new to machine-learning, and I am curious if anybody has a method for developing a robust validation framework to guard against over-fitting.

\n

In the original code sample given by tinrtgu, there is a field that allows us to validate on one of the days, or specify a holdout.  There is also a field for "epochs" or the number of passes through the dataset.

\n

My concerns - why are epochs useful, and how do we interpret it?  If I were to set it to be 10, and get 10 sets of log-losses from it, do I just take the average and have that to be my overall log-loss?

\n

Currently, the way I am approaching the problem is to shuffle the whole training dataset, and setting the holdout to be various numbers (usually 10 or 100).  My reasoning for shuffling the dataset is to get rid of any serial dependencies the entries may have had upon one another.  Is this a solid approach?

\n

We are getting very suspicious of the results that the validation set that kaggle is spitting out, because it is only on a sub-set.  We feel that we are over-fitting to that set specifically by making the submissions.

", "rawMarkdown": "", "votes": null }, { "id": "59907", "postDate": "12/09/2014 21:07:26", "content": "

[quote=sneakyfox;59899]

\n

I am pretty new to machine-learning, and I am curious if anybody has a method for developing a robust validation framework to guard against over-fitting.[/quote]

\n

The most widely-used and broadly-applicable and all-around-awesome method is simply cross validation (CV)--test on something you didn't train on. Since you're "holding out" every nth sample or full days, you're already doing this :) 

\n

[quote=sneakyfox;59899]

\n

 why are epochs useful, and how do we interpret it?[/quote]

\n

Epochs are the number of times you pass through the same data; training then re-training on it. In an online learner, you typically use a small "learning rate", so that the information stored in a single data point isn't all jammed into your model at once. For instance, if you train on a data point, then immediately predict that exact same point, you won't get the true value. This is actually very desirable, since you don't want your model to simply reproduce the most recent point perfectly, but rather take the big picture into account.

\n

By using multiple epochs, you allow each data point to contribute to the model more than once. If you have too little data, this can be really useful. But be careful, because if you use too many epochs, you run the risk of over-fitting to your training data and may lose generalizability.

\n

Train with however many epochs you want, then use that trained model to predict on some test data. Use the number of epochs that gives you a good CV score.

\n

[quote=sneakyfox;59899]

\n

Currently, the way I am approaching the problem is to shuffle the whole training dataset, and setting the holdout to be various numbers (usually 10 or 100).  My reasoning for shuffling the dataset is to get rid of any serial dependencies the entries may have had upon one another.  Is this a solid approach?[/quote]

\n

Good question. I don't understand the subtleties of the shuffle/don't shuffle argument very well.

\n

[quote=sneakyfox;59899]

\n

We are getting very suspicious of the results that the validation set that kaggle is spitting out, because it is only on a sub-set.  We feel that we are over-fitting to that set specifically by making the submissions.[/quote]

\n

This one has a simple solution: ignore the leader board. It's fun, and it can, potentially, give you some information--but you should trust in your CV. Usually at the end of the competition you're allowed to mark multiple submissions for final evaluation, so usually you can choose your best one or two based on CV and still submit your best LB version as well :)

", "rawMarkdown": "", "votes": null }, { "id": "59909", "postDate": "12/09/2014 21:23:48", "content": "

Thanks for the informative answer @LiamHuber! Would give 2 upvotes if I could!

\n

I had some quick follow up questions - for cross-validation, I have read in some places that it still provides an optimistic estimate for the generalization error.  Should we be just weary of this fact when applying it?  

\n

Secondly, do you have and good references for online-learning model?  I would like to learn more about the topic.

\n

Another approach that we were thinking of is averaging a bunch of models together, in order to reduce the variance - if we take our best 5 models according to CV score, and do an unweighted average, it may yield a more representative log-loss.  Just a thought though, may be completely wrong.

", "rawMarkdown": "", "votes": null }, { "id": "59931", "postDate": "12/10/2014 04:18:49", "content": "

So what is the major difference between 'fast_solution_v(1,2, or 3)' and Yannick's 'fast_solution_plus'?

\n\n

That is, using the same parameters between the two, why would I get different LB scores?

", "rawMarkdown": "", "votes": null }, { "id": "59936", "postDate": "12/10/2014 05:44:46", "content": "

I am having some trouble running fast_solution_v3.py on my terminal.  Here is the  error I receive:

\n\n

Traceback (most recent call last):
File "fast_solution_v3.py", line 260, in <module>
for t, date, ID, x, y in data(train, D): # data is a generator
File "fast_solution_v3.py", line 216, in data
for t, row in enumerate(DictReader(open(path))):
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/csv.py", line 107, in next
self.fieldnames
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/csv.py", line 90, in fieldnames
self._fieldnames = self.reader.next()
_csv.Error: line contains NULL byte

\n\n

Any help would be appreciated. Thanks.

", "rawMarkdown": "", "votes": null }, { "id": "59938", "postDate": "12/10/2014 06:04:08", "content": "

[quote=sneakyfox;59909]

\n

Thanks for the informative answer @LiamHuber! Would give 2 upvotes if I could!

\n

I had some quick follow up questions - for cross-validation, I have read in some places that it still provides an optimistic estimate for the generalization error.  Should we be just weary of this fact when applying it?  

\n

Secondly, do you have and good references for online-learning model?

\n

Another approach that we were thinking of is averaging a bunch of models together...

\n

[/quote]

\n\n

Glad it was helpful!

\n

I think how optimistic your CV is will depend on what you're drawing it from. e.g. contrast CV by holding out every 100th step vs CV by holding out an entire day. In the former case, there is likely to be more similarity between the train and test data points than in the later case (perhaps because the train day is a weekday but test day is a weekend, or any other factor.) Since our goal is to predict a whole new day, I recommend CV by separate days. But even this isn't a panacea since the test set has many feature values totally missing in the training set (there's a really nice post somewhere showing a breakdown of that on the old data, but I can't find it.)

\n

I don't have any great advice for online learners. This is my first time exploring online learning as well. The script in this thread is obviously really helpful, and I've also been having moderate luck (CV ~0.41) with a fairly simple neural net.

\n

Your last idea is usually called bagging, and is an excellent idea!

", "rawMarkdown": "", "votes": null }, { "id": "59951", "postDate": "12/10/2014 15:56:33", "content": "

@inversion: It makes sense - thanks a lot! I don't have now time to complete the script, but this part makes sense! ;)

", "rawMarkdown": "", "votes": null }, { "id": "60037", "postDate": "12/11/2014 18:51:56", "content": "

Just wondering how people are getting .392 with Yannick's code? Are you using the default parameters and how many epochs did you use? We are stuck at 0.3947749 and just trying to get below .394. Thanks!

", "rawMarkdown": "", "votes": null }, { "id": "60038", "postDate": "12/11/2014 18:53:20", "content": "

Hey guys,

\n

Thanks for posting these codes, great learning!

\n

One question, how are people able to get 0.392 with Yannick's code? If you are, are you changing the parameters or using the default settings, and how many epochs do you use? 

\n

We really just want to see if we can get below 0.394 using this code...

\n

Thanks!

", "rawMarkdown": "", "votes": null }, { "id": "60040", "postDate": "12/11/2014 19:24:36", "content": "

[quote=Nicholas Guttenberg;59851]

\n

With no feature engineering I'm at 0.3931 LB with Yannick's version of the script, but the runs take about 6 hours and maybe 4-6GB of RAM. Looks like I've still got some parameter optimization to do.

\n

[/quote]

\n

how many epochs are you running?

", "rawMarkdown": "", "votes": null }, { "id": "60061", "postDate": "12/12/2014 00:28:56", "content": "

I'm not actually sure how much specific parameter information is kosher for me to share (at least, I'd like an opinion on that from the more senior Kagglers before giving too much detail, since someone was gently reminded of this for sharing their # of bits earlier in the thread).

", "rawMarkdown": "", "votes": null }, { "id": "60062", "postDate": "12/12/2014 00:36:15", "content": "

[quote=Nicholas Guttenberg;60061]

\n

I'm not actually sure how much specific parameter information is kosher for me to share (at least, I'd like an opinion on that from the more senior Kagglers before giving too much detail, since someone was gently reminded of this for sharing their # of bits earlier in the thread).

\n

[/quote]

\n

The rules allow you to share anything you want, but if you post the exact parameters that give a 0.3931 score you'll make some people mad. It's entirely up to you how much or how little you share. Remember there will be a robust discussion where a lot of people explain everything when the competition is over.

", "rawMarkdown": "", "votes": null }, { "id": "60144", "postDate": "12/13/2014 13:07:08", "content": "

Hi Tinrtgu, thanks for posting this, it's very interesting and helpful!  

\n

I have a question about the hashing: if I'm reading the code correctly, all the features have their hashed values mapped into the same list of indices (x).  Does that mean that there is a risk of collision even between the values of different features?  It seems to me like there is (even with a fairly large D) although I'm not sure it's something I should be concerned about.

", "rawMarkdown": "", "votes": null }, { "id": "60145", "postDate": "12/13/2014 13:19:45", "content": "

[quote=dlsiii;60144]

\n

Does that mean that there is a risk of collision even between the values of different features?

\n

[/quote]

\n

Yes, you are correct and I did it that way to ease the implementation. If you are really concerned you can maintain an index look-up dictionary so that there is 0 chance of collision, it should give you some improvements, although I feel it would only be marginal.

", "rawMarkdown": "", "votes": null }, { "id": "60148", "postDate": "12/13/2014 15:37:51", "content": "

[quote=tinrtgu;60145]

\n

Yes, you are correct and I did it that way to ease the implementation. If you are really concerned you can maintain an index look-up dictionary so that there is 0 chance of collision, it should give you some improvements, although I feel it would only be marginal.

\n

[/quote]

\n

Thanks for the quick response, you're probably right, when I think about it seems like the chance of two values with reasonably high frequency colliding are pretty low for a large enough D.

", "rawMarkdown": "", "votes": null }, { "id": "60155", "postDate": "12/13/2014 18:32:34", "content": "

One way to test to see if collisions are making a difference would be to do two (or more, if you have time) sets of predictions where you modify the category names in different ways before passing them into the hash function. If the log-loss varies significantly, then something important might be colliding (e.g. a frequently occurring pair of highly predictive categorical labels are getting hashed to the same thing). 

", "rawMarkdown": "", "votes": null }, { "id": "60162", "postDate": "12/13/2014 20:34:18", "content": "

Sorry to bug people with what's probably a simple I/O issue, but I get this error when running fast_solution_v3:

\n\n

--------------------------------------------------------------------------
Error Traceback (most recent call last)
<ipython-input-16-9f047f2c965d> in <module>()
4 count = 0
5
----> 6 for t, date, ID, x, y in data(train, D): # data is a generator
7 # t: just a instance counter
8 # date: you know what this is<ipython-input-8-ed0fc799fe79> in data(path, D)
29 '''
30
---> 31 for t, row in enumerate(DictReader(open(path))):
32 # process id
33 ID = row['id']/Users/David/anaconda/lib/python2.7/csv.pyc in next(self)
105 if self.line_num == 0:
106 # Used only for its side effect.
--> 107 self.fieldnames
108 row = self.reader.next()
109 self.line_num = self.reader.line_num/Users/David/anaconda/lib/python2.7/csv.pyc in fieldnames(self)
88 if self._fieldnames is None:
89 try:
---> 90 self._fieldnames = self.reader.next()
91 except StopIteration:
92 pass

\n\n

Error: new-line character seen in unquoted field - do you need to open the file in universal-newline mode?

\n\n

Does anyone know what the solution is?

\n\n

Thanks in advance...

", "rawMarkdown": "", "votes": null }, { "id": "60175", "postDate": "12/14/2014 05:44:17", "content": "

Hi,

\n

I've been playing around with the epoch values and I've noticed at after a certain number of iteration the validation log-loss seems to increase - this is because of the model is overfitting the training data right? and one possible way to avoid that is to increase regularization?

\n

(Sorry if its a noob question, I'm still a beginner)

", "rawMarkdown": "", "votes": null }, { "id": "60180", "postDate": "12/14/2014 07:34:46", "content": "

[quote=Pavitrakumar;60175]

\n

Hi,

\n

I've been playing around with the epoch values and I've noticed at after a certain number of iteration the validation log-loss seems to increase - this is because of the model is overfitting the training data right? and one possible way to avoid that is to increase regularization?

\n

(Sorry if its a noob question, I'm still a beginner)

\n

[/quote]

\n

The question is actually great, and I think it is one of the key to this competition - how to do robust validation and keep overfitting in check.

\n

As to your question, yep, it means that the model is overfitting to the training set, increasing L1 and/or L2 regularization is one way to deal with it, decreasing the learning rate will also have some positive effect, another way is to not use so much epochs to begin with.

", "rawMarkdown": "", "votes": null }, { "id": "60185", "postDate": "12/14/2014 11:44:51", "content": "

[quote=tinrtgu;60180]

\n

[quote=Pavitrakumar;60175]

\n

Hi,

\n

I've been playing around with the epoch values and I've noticed at after a certain number of iteration the validation log-loss seems to increase - this is because of the model is overfitting the training data right? and one possible way to avoid that is to increase regularization?

\n

(Sorry if its a noob question, I'm still a beginner)

\n

[/quote]

\n

The question is actually great, and I think it is one of the key to this competition - how to do robust validation and keep overfitting in check.

\n

As to your question, yep, it means that the model is overfitting to the training set, increasing L1 and/or L2 regularization is one way to deal with it, decreasing the learning rate will also have some positive effect, another way is to not use so much epochs to begin with.

\n

[/quote]

\n

Thank you for the reply, tinrtgu.

\n

As you said, I am getting better validation log-loss after tuning my L1,L2 and epoch parameters, but I am still not able to reach my existing LB score obtained using VW.

\n

I am not able to get a validation log-loss lesser than 0.398x - Can you please tell me what part am I missing here? 

", "rawMarkdown": "", "votes": null }, { "id": "60190", "postDate": "12/14/2014 15:59:34", "content": "

Hi tinrtgu,

\n

I can't seem to download the fast_solution files that you attached. Can you please take a look at it?

", "rawMarkdown": "", "votes": null }, { "id": "60197", "postDate": "12/14/2014 18:13:49", "content": "

[quote=Saurav Gupta;60190]

\n

Hi tinrtgu,

\n

I can't seem to download the fast_solution files that you attached. Can you please take a look at it?

\n

[/quote]

\n

I can download it properly, what is the problem?

", "rawMarkdown": "", "votes": null }, { "id": "60199", "postDate": "12/14/2014 19:29:09", "content": "

I don't know, it says that the server can't be found because DNS lookup failed. Can you please send it to me at sauravguptaaa@gmail.com ?

\n

Thanks!

\n

Edit: Fixed it. No need to email. Using Google's DNS servers now :)

", "rawMarkdown": "", "votes": null }, { "id": "60231", "postDate": "12/15/2014 13:24:48", "content": "

[quote=dlsiii;60144]

\n

Hi Tinrtgu, thanks for posting this, it's very interesting and helpful!  

\n

I have a question about the hashing: if I'm reading the code correctly, all the features have their hashed values mapped into the same list of indices (x).  Does that mean that there is a risk of collision even between the values of different features?  It seems to me like there is (even with a fairly large D) although I'm not sure it's something I should be concerned about.

\n

[/quote]

\n

its hash the key_value combination. e.g. C1_123456, SITE_ID_123456

\n

ln240:  index = abs(hash(key + '_' + value)) % D

", "rawMarkdown": "", "votes": null }, { "id": "60316", "postDate": "12/16/2014 03:07:16", "content": "

[quote=tinrtgu;58054]

\n

Notice (11/18/2014): added the third version of the script fast_solution_v3.py

\n

Notice (11/20/2014): here is a post on how to get 0.3977417 on the public leaderboard

\n

-

\n

Dear fellow Kagglers, tin-ert-gu here.

\n

-

\n

This is a python implementation of adaptive learning rate L1 and L2 regularized logistic regression using hash-trick one-hot encoding. This algorithm is also used by Google for their online CTR prediction system.
All credits of this script goes to their hard work of research.

\n

-

\n

How to run?

\n

In your terminal, simply type

\n

python fast_solution.py

\n

The script is also python3 compatible, so you can use python3 as well

\n

python3 fast_solution.py

\n

However, since the code only depends on native modules, it is highly suggested that you run the script with pypy for a huge speed-up. To use the script with pypy on Ubuntu, first type

\n

sudo apt-get install pypy

\n

Then use the pypy interpreter to run the script

\n

pypy fast_solution.py

\n

-

\n

Changelog over the previous version of the script

\n
    \n
  1. Added L1 regularization
    2. \n
  2. Added L2 regularization
    3. \n
  3. Build in multiple epoch training
    4. \n
  4. Build in holdout set validation support
    5. \n
  5. Build in support for poly 2 feature interactions
    6. \n
  6. Re-factored code for better extendibility
    7. \n
  7. EVEN MORE COMMENTS!
    8. \n
\n

-

\n

Performance

\n

Training time for a single epoch, on a single Intel(R) Xeon(R) CPU E5-2630L v2 @ 2.40GHz:

\n\n

Memory usage:

\n

This mainly depends on the D parameter in the script.

\n\n

-

\n

The entire algorithm is disclosed, while using only out of the box modules provided by Python, so it should also be an easy sandbox code to build new ideas upon.

\n

Good luck and have fun!

\n

-

\n

@Abhishek:

\n

Can't submit right now, so I don't know the leader-board score, but I would say it should easily beat 0.40

\n

-

\n

EDIT 1

\n

add fast_solution_v2.py to fix a null reference bug under memory saving mode

\n

EDIT 2

\n

add fast_solution_v3.py

\n

A great thanks for Paweł's feedback, the following changes are made as a result

\n\n

By considering fchollet's experiment

\n\n

This should be the last time I modify this benchmark for this competition. Hope the actual data will get release soon, and happy predicting!

\n

[/quote]

\n\n

Thanks for the code. Every time I run it, regardless of the parameter values, I get a prediction of 0.5 for every record in the test set. Am I missing something? Using fast_solution_v3.py.

", "rawMarkdown": "", "votes": null }, { "id": "60317", "postDate": "12/16/2014 03:10:27", "content": "

@AlKhwarizmi

\n

you should read previous page:

\n

https://www.kaggle.com/c/avazu-ctr-prediction/forums/t/10927/beat-the-benchmark-with-less-than-1mb-of-memory?page=2

", "rawMarkdown": "", "votes": null }, { "id": "60318", "postDate": "12/16/2014 03:37:08", "content": "

I got the following error for some of my datasets:

\n

Traceback (most recent call last):
File "app_main.py", line 72, in run_toplevel
File "fast_solution_v3.py", line 253, in <module>
for t, date, ID, x, y in data(train, D): # data is a generator
File "fast_solution_v3.py", line 222, in data
date = int(row['hour'][4:6])-20
ValueError: invalid literal for int() with base 10: ''

\n

Any hints to solve it?

\n

Thks

\n

Edit: there are some NA values in my 'hour'

\n

will fix it :)

", "rawMarkdown": "", "votes": null }, { "id": "60460", "postDate": "12/17/2014 10:25:56", "content": "

The attached graph tells a lot about how people have used your code @tinrtgu !

\n

2 humps at 0.398 and 0.395!\"Avazu

", "rawMarkdown": "", "votes": null }, { "id": "61621", "postDate": "12/21/2014 16:13:48", "content": "

[quote=tinrtgu;59441]

\n

[quote=TanoPereira;59372]

\n

I've been able to go down to 0.376 in validation with interactions, but LB=0.41 ...

\n

[/quote]

\n

Just a hint, overfitting is more severe with interactions so it would be a good idea to increase regularizations.

\n

[/quote]

\n

Dear friends,

\n

I have questions about CV score and LB score for the interactions-version code. I use day 21-28 for training, and day 29-30 for validation. For the no-interaction version code, it works fine for me(i.e., in most case, improvement in CV also results in improvement in LB score).

\n

But for the interaction version code, it seems to me that such 2-day validation approach totally corrupts. To be specific, with interactions, my 2-day validation score is about 0.387, while the LB score is 0.399! (In comparison, my best 2-day validation score of no-interaction code is about 0.391, and the corresponding LB score is 0.394).

\n

Note that,  in my interaction-version code for this 0.387 validation result, I already follows the suggestion of tinrtgu and significantly increase the regularizations to (L1=40,L2=40), which is about10 times larger than the regularizations used in my best-results of the no-interaction version.

\n

Do I miss anything here, or any explanation or hints?

\n

Thanks in advance, and have a great day!

\n

Best wishes,

\n

Shize

", "rawMarkdown": "", "votes": null }, { "id": "61642", "postDate": "12/22/2014 00:33:25", "content": "

1) Would somebody be kind enough to explain/comment how lines 65-98 actually work? For example, how does "self.w = [0.] * D" work? Doesn't multiplying by 0 just make it 0? 

\n

2) Is anyone touching the bounded sigmoid function at line 83?

", "rawMarkdown": "", "votes": null }, { "id": "61646", "postDate": "12/22/2014 04:28:11", "content": "

[quote=PBoswell;61642]

\n

For example, how does "self.w = [0.] * D" work? Doesn't multiplying by 0 just make it 0? 

\n

[/quote]

\n

[0] * 5 in python makes [0,0,0,0,0]

", "rawMarkdown": "", "votes": null }, { "id": "61700", "postDate": "12/22/2014 21:05:47", "content": "

[quote=tinrtgu;58054]

\n

Notice (11/18/2014): added the third version of the script fast_solution_v3.py

\n

Notice (11/20/2014): here is a post on how to get 0.3977417 on the public leaderboard

\n

-

\n

Dear fellow Kagglers, tin-ert-gu here.

\n

-

\n

This is a python implementation of adaptive learning rate L1 and L2 regularized logistic regression using hash-trick one-hot encoding. This algorithm is also used by Google for their online CTR prediction system.
All credits of this script goes to their hard work of research.

\n

[/quote]

\n\n

Hey Tinrtgu -

\n

Curious for the reason why the absolute value of the hash value is taken before the mod operation? Does this help with collisions?

\n

Thanks!

", "rawMarkdown": "", "votes": null }, { "id": "61708", "postDate": "12/22/2014 23:06:25", "content": "

[quote=Inspector;61700]

\n

Hey Tinrtgu -

\n

Curious for the reason why the absolute value of the hash value is taken before the mod operation? Does this help with collisions?

\n

Thanks!

\n

[/quote]

\n

Absolute value is taken, because hash values are later used to reference to different weights in w[i] where i is a hash value. And a negative value cannot be a proper list index in Python.

\n

Abs doesn't help collisions, it makes it twice worse (twice less possible values for hash to take), but these are the downsides of super fast hash trick.

", "rawMarkdown": "", "votes": null }, { "id": "61715", "postDate": "12/23/2014 00:40:10", "content": "

[quote=Ivan Lobov;61708]

\n

[quote=Inspector;61700]

\n

Hey Tinrtgu -

\n

Curious for the reason why the absolute value of the hash value is taken before the mod operation? Does this help with collisions?

\n

Thanks!

\n

[/quote]

\n

Absolute value is taken, because hash values are later used to reference to different weights in w[i] where i is a hash value. And a negative value cannot be a proper list index in Python.

\n

Abs doesn't help collisions, it makes it twice worse (twice less possible values for hash to take), but these are the downsides of super fast hash trick.

\n

[/quote]

\n

If you are using the hashed value (after %D) as input into a one-hot encoding, it would make sense to not take the abs() first - correct?

", "rawMarkdown": "", "votes": null }, { "id": "61717", "postDate": "12/23/2014 00:49:17", "content": "

I don't know why, I end up getting equally probable for all ads, with this solution. 0.5 for all. Could someone kindly point out what I am doing wrong. I realize this might be an extremely stupid question to ask. 

", "rawMarkdown": "", "votes": null }, { "id": "61718", "postDate": "12/23/2014 00:58:10", "content": "

[quote=Dee Dee;61717]

\n

I don't know why, I end up getting equally probable for all ads, with this solution. 0.5 for all. Could someone kindly point out what I am doing wrong. I realize this might be an extremely stupid question to ask. 

\n

[/quote]

\n

Hi Dee,

\n

I guess you didn't change the parameter holdafter=9 to holdafter=29? There are different versions of the code, and for v3, if you use holdafter=9, it simply means that no data were used for training (i.e., all data are used for validation), and this is why you get all 0.5 prediction.

\n

If you set the parameter holdafter=29 and D=2^24, you would get LB score about 0.376.

\n\n

Hope this might be helpful for you.

\n

Good night!

\n

Best wishes,

\n

Shize

", "rawMarkdown": "", "votes": null }, { "id": "61720", "postDate": "12/23/2014 01:11:19", "content": "

[quote=Inspector;61715]

\n

If you are using the hashed value (after %D) as input into a one-hot encoding, it would make sense to not take the abs() first - correct?

\n

[/quote]

\n

Yeah, feel free to remove it. I added that abs() without giving it a thought, thanks for pointing it out. 

", "rawMarkdown": "", "votes": null }, { "id": "61722", "postDate": "12/23/2014 01:22:55", "content": "

[quote=tinrtgu;61720]

\n

[quote=Inspector;61715]

\n

If you are using the hashed value (after %D) as input into a one-hot encoding, it would make sense to not take the abs() first - correct?

\n

[/quote]

\n

Yeah, feel free to remove it. I added that abs() without giving it a thought, thanks for pointing it out. 

\n

[/quote]

\n\n

OK, so it is not actually a required part of the program? I didn't see where it was actually needed. Cant thank you enough for sharing this - super cool

", "rawMarkdown": "", "votes": null }, { "id": "61774", "postDate": "12/23/2014 19:52:50", "content": "

In version 3 of tngrtu's script, does setting holdafter to, say, 28 mean validation on every day after 28 or just the next day? If it is the former, how would I specify a non-consecutive series of days?

", "rawMarkdown": "", "votes": null }, { "id": "61777", "postDate": "12/23/2014 20:19:13", "content": "

[quote=PBoswell;61774]

\n

In version 3 of tngrtu's script, does setting holdafter to, say, 28 mean validation on every day after 28 or just the next day? If it is the former, how would I specify a non-consecutive series of days?

\n

[/quote]

\n\n

Hi PBoswell,

\n\n

In v3 code, setting holdafter=28 means validation on everyday after 28 (i.e., 29 and 30). 

\n\n

To specify a non-consecutive series of days for validation, a simple and intuitive way is that you can change the validation condition line in the code file from " if (holdafter and date > holdafter) or (holdout and t % holdout == 0):" to any conditions you want, such as " if (holdafter and (date ==30 or date==24 or date==26)) or (holdout and t % holdout == 0):". 

\n\n

But still one issue associated with the validation approach specified by " if (holdafter and (date ==30 or date==24 or date==26)) or (holdout and t % holdout == 0):" is that the validation loss is not computed after that the model training  is finished. To be specific, the model start training on data day 21 -23 and compute logloss on day 24, and then continue model training on day 25, and then compute logloss on day 26, and then continue training on day 27-29, and then compute logloss on day 30. It is easy to see that such validation loss is not accurate enough to evaluate the trained model's efficiency (for accurate validation loss computation, ideally, it should first train the model on day 21-23, day 25, day 27-29 data, and then start to compute the validation loss on day 24, 26 and 30). Therefore, for more accurate validation loss, I think you could use one small script to output the validation data into a separate csv file, and then you could compute the validation logloss after the whole training procedure finished. (Currently I am still using day 29 and day 30 for validation, so I didn't use such small script for writing separate validation set. But I believe it is easy to write such small script if you want.)

\n\n

Hope this might be helpful for you.

\n\n

I am also interested to know if anybody else has a much easier way to do nonconsecutive-series of days for validation.

\n\n

Thanks, and have a good day!

\n\n

Best wishes,

\n

Shize

", "rawMarkdown": "", "votes": null }, { "id": "62122", "postDate": "12/31/2014 07:12:18", "content": "

Thanks for the sharing. 

\n\n

I have a question about the bias term. Generally, the bias term should represent the average score of a model. But I found the bias generated by it is near zero. Anyone know why? 

", "rawMarkdown": "", "votes": null }, { "id": "62128", "postDate": "12/31/2014 10:30:53", "content": "

[quote=jdxyw;62122]

\n

Thanks for the sharing. 

\n

I have a question about the bias term. Generally, the bias term should represent the average score of a model. But I found the bias generated by it is near zero. Anyone know why? 

\n

[/quote]

\n

That's an excellent question, and maybe a key to improving the algorithm. I am wondering if we don't get an over-weight on some features which might block the bias term and maybe some others, and thus give an under-performing logistic regression.

\n

Any other insight?

", "rawMarkdown": "", "votes": null }, { "id": "62195", "postDate": "01/02/2015 00:09:50", "content": "

#split training data into several days

\n

from datetime import datetime

\n

#++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
#main program starts here
start = datetime.now()

\n

train = 'data/train.csv' # path to training file
target_date = '30'
save_file = 'data/date' + target_date + '.csv'
str_label = 'id,click,hour,C1,banner_pos,site_id,site_domain,site_category,app_id,app_domain,app_category,device_id,device_ip,device_model,device_type,device_conn_type,C14,C15,C16,C17,C18,C19,C20,C21'

\n

with open(save_file, 'w') as outfile:
outfile.write('%s\\n' % str_label)

f=open(train)
f.readline()
for line in f:
xx=line.split(',')
date = xx[2][4:6]
if date == target_date :
outfile.write('%s\\n' % line)

\n

f.close()

print('saved: %s, etime: %s' % (save_file, str(datetime.now() - start)))

", "rawMarkdown": "", "votes": null }, { "id": "62203", "postDate": "01/02/2015 06:28:09", "content": "

[quote=Yannick Martel;62128]

\n

That's an excellent question, and maybe a key to improving the algorithm. I am wondering if we don't get an over-weight on some features which might block the bias term and maybe some others, and thus give an under-performing logistic regression.

\n

Any other insight?

\n

[/quote]

\n

I think the reason is that the model, as it stands, is basically unidentified. To illustrate what I mean by this, let's say we have just one binary attribute and we will do the one-hot encoding. So, the linear function of the model can be written as: bias + (param1 or param2). Now the problem is that if {bias=0, param1=1, param2=2} is the "correct answer", then so is {bias=1, param1=0, param2=1} and {bias=10, param1=-9, param2=-8} etc. Ie. one of the parameters can be set to arbitrarily value.

\n

To fix this, the bias term or one of the other parameters should be set to value 0. However, due to regularization I don't think it would make a big difference though.

", "rawMarkdown": "", "votes": null }, { "id": "62409", "postDate": "01/07/2015 17:03:36", "content": "

Thanks for sharing this but why did I get a "clean" Submission with 0.5 all over the ads list?

", "rawMarkdown": "", "votes": null }, { "id": "62425", "postDate": "01/07/2015 21:34:08", "content": "

[quote=Farhani Wajdi;62409]

\n

Thanks for sharing this but why did I get a "clean" Submission with 0.5 all over the ads list?

\n

[/quote]

\n\n

This question has been answered before in the thread. You have to specify the train set by setting the holdafter variable to a correct value.

\n\n

For instance holdafter=28 means training over all the days until the 28th and apply CV over the data corresponding to the 29 and 30 day.

", "rawMarkdown": "", "votes": null }, { "id": "62447", "postDate": "01/08/2015 08:27:20", "content": "

[quote=josefernando.zazo;62425]

\n

This question has been answered before in the thread. You have to specify the train set by setting the holdafter variable to a correct value.

\n

For instance holdafter=28 means training over all the days until the 28th and apply CV over the data corresponding to the 29 and 30 day.

\n

[/quote]

\n

That's look fine, Thanks

", "rawMarkdown": "", "votes": null }, { "id": "62454", "postDate": "01/08/2015 14:01:11", "content": "

[quote=fchollet;58089]

\n

Awesome! Thank you for sharing this. The method is pretty much state of the art, and the code quality is excellent! Huge thumb up.

\n

If you guys haven't already, be sure to check out the Google paper that describes this method. Hint: it contains useful info on how to improve the algo :) 

\n

http://www.eecs.tufts.edu/~dsculley/papers/ad-click-prediction.pdf

\n

[/quote]

\n

@fchollet, can you please hint further on what part of the paper you were referring to when saying "it contains useful info on how to improve the algo". I've read it for the second time today but I still not sure what you were referring to.

\n

Thanks,

\n

C 

", "rawMarkdown": "", "votes": null }, { "id": "62462", "postDate": "01/08/2015 16:32:40", "content": "

[quote=clustifier;62454]

\n

[quote=fchollet;58089]

\n

Awesome! Thank you for sharing this. The method is pretty much state of the art, and the code quality is excellent! Huge thumb up.

\n

If you guys haven't already, be sure to check out the Google paper that describes this method. Hint: it contains useful info on how to improve the algo :) 

\n

http://www.eecs.tufts.edu/~dsculley/papers/ad-click-prediction.pdf

\n

[/quote]

\n

@fchollet, can you please hint further on what part of the paper you were referring to when saying "it contains useful info on how to improve the algo". I've read it for the second time today but I still not sure what you were referring to.

\n

Thanks,

\n

C 

\n

[/quote]

\n

I guess he is most likely reffering to "CALIBRATING PREDICTIONS"

", "rawMarkdown": "", "votes": null }, { "id": "62519", "postDate": "01/09/2015 18:35:16", "content": "

the paper seems to suggest that the only reason to use FTRL-proximal is to have a sparse weight matrix. we're not using this sparsity, we store the whole weight matrix. so it should be better to use plain Logistic Regression. Atleast that's what i got, please correct me.

", "rawMarkdown": "", "votes": null }, { "id": "62577", "postDate": "01/10/2015 18:37:25", "content": "

Some of the features we are using are maybe irrelevant for the prediction, more probably the noise is superior to the signal for one or both of these reasons:
- A lot of noise, i.e. the feature have a small count.
- A weak signal, i.e. the feature have a very small predictive value.

\n

So as I understand it, the weight vector is sparse in order to get rid of this kind of features, hopefully improving the generalisation performance, by avoiding fitting the noise.

\n

I don't see much value in storing a sparse weight vector, it is small anyway.

\n

And for the record, the FTRL-proximal is just a plain Logistic Regression with an elastic net regularisation (L1 + L2). The FTRL's advantage is to optimize with a small number of passes over the data, Google's paper gave me the impression that it is optimized for one pass.

", "rawMarkdown": "", "votes": null }, { "id": "62600", "postDate": "01/11/2015 08:18:10", "content": "

This is magic! I'm still figuring out how it works, but it's just fascinating! 

", "rawMarkdown": "", "votes": null }, { "id": "62767", "postDate": "01/15/2015 21:46:43", "content": "

hi, could someone enlighten me why FTRL induces sparsity? compared to traditional SGD?

\n

Thank you very much!

", "rawMarkdown": "", "votes": null }, { "id": "62903", "postDate": "01/19/2015 11:56:01", "content": "

As I understood, FTRL "activate" coordinate weight (makes it non-zero) only if this weight have enough information (samples). And L1-penalty introduce some sparsity.

\n

Feel free to correct me if I'm wrong =)

", "rawMarkdown": "", "votes": null }, { "id": "62905", "postDate": "01/19/2015 14:32:40", "content": "

[quote=Mikhail Trofimov;62903]

\n

As I understood, FTRL "activate" coordinate weight (makes it non-zero) only if this weight have enough information (samples). And L1-penalty introduce some sparsity.

\n

Feel free to correct me if I'm wrong =)

\n

[/quote]

\n

Thank you very much. It makes good sense to me.

", "rawMarkdown": "", "votes": null }, { "id": "64213", "postDate": "02/14/2015 04:17:36", "content": "

[quote=Yannick Martel;59144]

\n

Hello all,

\n

I have worked on the fast_solution great piece of code and added a few features:

\n

- parameter control from command line

\n

- dropout: where you randomly drop features out of consideration - prevents overfitting and features adaptation

\n

- inclusion of the # of samples seen per device_ip / device_id as a feature

\n

- inclusion of weedkay as a feature

\n

I am down to roughly 0.3948 on the LB with a lot of tuning, by running on all test samples - with a few hours of processing using pypy on an i7, using a few GB's of RAM. LB is very close to my own cross-validation (run on all days except 30, test on 30).

\n

The code probably would benefit from some optimization. I have written the first version in Julia - it was faster than C-Python, but 2-3 times slower than pypy, and I quickly reverted to Python.

\n

I would be glad of any constructive feedback.

\n

Yannick

\n

[/quote]

\n

Yannick,

\n

I am curious about the dropout code. It looks like you are dividing the linear predictor wTx by 'dropout' when predicting an example (or during test time). I thought typically the prediction was multiplied by the dropout rate. Curious about this method. Thanks!

\n

for j, i in enumerate(self._indices(x)):

if dropped != None and dropped[j]:
continue

wTx += w[i]

if dropped != None: wTx /= dropout

\n

# bounded sigmoid function, this is the probability estimation
return 1. / (1. + exp(-max(min(wTx, 35.), -35.)))

", "rawMarkdown": "", "votes": null }, { "id": "64220", "postDate": "02/14/2015 07:27:25", "content": "

[quote=Inspector;64213]

\n

[quote=Yannick Martel;59144]

\n

Hello all,

\n

I have worked on the fast_solution great piece of code and added a few features:

\n

- parameter control from command line

\n

- dropout: where you randomly drop features out of consideration - prevents overfitting and features adaptation

\n

- inclusion of the # of samples seen per device_ip / device_id as a feature

\n

- inclusion of weedkay as a feature

\n

I am down to roughly 0.3948 on the LB with a lot of tuning, by running on all test samples - with a few hours of processing using pypy on an i7, using a few GB's of RAM. LB is very close to my own cross-validation (run on all days except 30, test on 30).

\n

The code probably would benefit from some optimization. I have written the first version in Julia - it was faster than C-Python, but 2-3 times slower than pypy, and I quickly reverted to Python.

\n

I would be glad of any constructive feedback.

\n

Yannick

\n

[/quote]

\n

Yannick,

\n

I am curious about the dropout code. It looks like you are dividing the linear predictor wTx by 'dropout' when predicting an example (or during test time). I thought typically the prediction was multiplied by the dropout rate. Curious about this method. Thanks!

\n

for j, i in enumerate(self._indices(x)):

if dropped != None and dropped[j]:
continue

wTx += w[i]

if dropped != None: wTx /= dropout

\n

# bounded sigmoid function, this is the probability estimation
return 1. / (1. + exp(-max(min(wTx, 35.), -35.)))

\n

[/quote]

\n

Dear Inspector,

\n

Yes, my implementation is a bit more intricate than the way you are describing, but it is supposed to work the same.

\n

The "dropout" variable is incorrectly named: it is the rate of features that I am keeping: 1. for keeping everybody, .8 for keeping 80%, .5 for keeping only half of them.

\n

I wanted to apply all of dropout at update time, not prediction time, so I put the weight adjustment in the predict code so it is applied only when updating (predict call with dropped != None).

\n

Dividing wTx by dropout is equivalent to multiplying the features by 1/dropout - making them stronger when they are missing (1-dropout) * N of their buddies, when they should have been N without dropout.

\n

Then, in a sense, instead of reducing weights at prediction time by multiplying by the dropout rate (or more correctly:  1 - rate of dropout), I increase the weights (or features) at learning time by dividing by the same rate.

\n

It should be somewhat the same, except maybe that the impact of the alpha and beta is different.

\n

I have also tried a variant: divide not by the "dropout" variable, but by the ratio of features we are keeping for this particular sample:

\n

if wTx != 0: wTx *= len(dropped) * 1.0 / (len(dropped) - sum(dropped))

\n

It is slightly more complex, a bit longer to compute and I have not found it to improve performance.

\n

I hope the explanation helps, please tell me if not!

", "rawMarkdown": "", "votes": null }, { "id": "64234", "postDate": "02/14/2015 17:14:40", "content": "

[quote=Yannick Martel;64220]

\n

Dear Inspector,

\n

Yes, my implementation is a bit more intricate than the way you are describing, but it is supposed to work the same.

\n

The "dropout" variable is incorrectly named: it is the rate of features that I am keeping: 1. for keeping everybody, .8 for keeping 80%, .5 for keeping only half of them.

\n

I wanted to apply all of dropout at update time, not prediction time, so I put the weight adjustment in the predict code so it is applied only when updating (predict call with dropped != None).

\n

Dividing wTx by dropout is equivalent to multiplying the features by 1/dropout - making them stronger when they are missing (1-dropout) * N of their buddies, when they should have been N without dropout.

\n

Then, in a sense, instead of reducing weights at prediction time by multiplying by the dropout rate (or more correctly:  1 - rate of dropout), I increase the weights (or features) at learning time by dividing by the same rate.

\n

It should be somewhat the same, except maybe that the impact of the alpha and beta is different.

\n

I have also tried a variant: divide not by the "dropout" variable, but by the ratio of features we are keeping for this particular sample:

\n

if wTx != 0: wTx *= len(dropped) * 1.0 / (len(dropped) - sum(dropped))

\n

It is slightly more complex, a bit longer to compute and I have not found it to improve performance.

\n

I hope the explanation helps, please tell me if not!

\n

[/quote]

\n

I am still confused I am afraid. I think it might be that I am thinking about this for offline learning.

\n

I. First, is my understanding correct that the dropout adjustment (generally, not in this type of situation necessarily) is done because when training a model, (dropout rate)% features are removed, thus we expect the remaining (1-dropout rate)% coefficients to be greater in magnitude than otherwise (they are compensating for the loss of the other features). So for example, if we keep 75% of the features, the weights will be on average too large (25% too large). Attached is an example of this and the adjustment.

\n

II. Assuming this is correct, is FTRL case different? Will the weights not adjust to compensate for the missing (dropped) ones, so that each weight will end up being too small and that is why you divide the sum wTx by dropout?

\n

III. Do you divide by dropout weight when predicting the test set as well? I assume so, but just checking.

\n

Thanks!

", "rawMarkdown": "", "votes": null }, { "id": "64235", "postDate": "02/14/2015 19:03:16", "content": "

I think that we are quite close in understanding, but let me try to explain in another way, with a concrete example: if I (randomly) remove, let's say, 20% of features, then to predict correctly I must increase the weights by 1.25 = 1/0.8, in order to keep the sum of weights coherent.

\n

With dropout, we remove features only when learning, not when predicting. During learning, there are two steps:

\n
    \n
  1. predict for the value of x for which we want to learn -> with dropout we predict with a randomly selected of features, let's say we keep only [1, 2, 4, 5] out of [1, 2, 3, 4, 5];
    2. \n
  2. update the weights for the features we have seen based on the difference between the the expected y and the p_hat we get - so here we update weights for features [1, 2, 4, 5].
    3. \n
\n

In step 1, I predict with weights multiplied by 1.25 (with 80% dropout rate, or 20% of features removed). In step 2, I update weights for [1, 2, 4, 5], with no dropout factor.

\n

When predicting, I would predict with all of [1, 2, 3, 4, 5], with no weight adjustment.

\n\n

If we assume all weights are 1, then the sum of weights is:

\n\n\n

Does this make more sense to you?

", "rawMarkdown": "", "votes": null }, { "id": "64241", "postDate": "02/14/2015 20:25:39", "content": "

[quote=Yannick Martel;64235]

\n

I think that we are quite close in understanding, but let me try to explain in another way, with a concrete example: if I (randomly) remove, let's say, 20% of features, then to predict correctly I must increase the weights by 1.25 = 1/0.8, in order to keep the sum of weights coherent.

\n

With dropout, we remove features only when learning, not when predicting. During learning, there are two steps:

\n
    \n
  1. predict for the value of x for which we want to learn -> with dropout we predict with a randomly selected of features, let's say we keep only [1, 2, 4, 5] out of [1, 2, 3, 4, 5];
    2. \n
  2. update the weights for the features we have seen based on the difference between the the expected y and the p_hat we get - so here we update weights for features [1, 2, 4, 5].
    3. \n
\n

In step 1, I predict with weights multiplied by 1.25 (with 80% dropout rate, or 20% of features removed). In step 2, I update weights for [1, 2, 4, 5], with no dropout factor.

\n

When predicting, I would predict with all of [1, 2, 3, 4, 5], with no weight adjustment.

\n

If we assume all weights are 1, then the sum of weights is:

\n\n

Does this make more sense to you?

\n

[/quote]

\n

Thank you for your continued explanation!

\n

Here is my understanding of the standard way (what is discussed by Hinton) and how you are doing it here.It seems that VW uses the "standard way": http://permalink.gmane.org/gmane.comp.ai.machine-learning.vowpal-wabbit/163

\n\n

Please tell me I understand properly :)

\n\n

* N features are available
* p = probability a feature wont drop out (1-p it will)

\n

The Standard Way:
*A training example is seen and p*N of its features are used to predict the value of the output
*The update to these p*N weights is larger than it would normally be if all N features had been present. This is because the error (predicted - actual) is larger.
*Thus, on average, all the weights will be larger by 1/(1-p) and so we multiply the final weights by p at test time, when all N features will be present. Otherwise we would have trained a model with weights too large.

\n

Non-Standard Way:
*A training example is seen and p*N of its features are used to predict the value of the output (just like the standard way)
* We divide the predicted value by p, which means on average the update to the p*N weights will be what they would have been without dropout because the error was reduced (predicted / p - actual).
*At test time, no need to do anything - use the final weights as is and include all the N features for every example

", "rawMarkdown": "", "votes": null }, { "id": "64245", "postDate": "02/14/2015 21:00:06", "content": "

Congratulations, I believe that you have nailed it. Your explanation seems to me both correct and very clear!

\n

The "standard way" is what I have seen explaned in the litterature on the topic. I have implemented the "non standard way" because I wanted to keep the prediction time cleaner - no dropout coefficients to adjust.

\n

As I have mentionned above, I have tested the option to correct at learning time by the actual proportion of features which have been selected for this particular sample - but it has not changed anything. Maybe it would make a difference with a dropout rate closer to 0.5 than what I used.

\n

I have also tested with no adjustment to weights, and it performed very poorly, as you could expect...

\n

Another difference to the standard way: I apply dropout also to the bias term - not 100% sure this is a smart thing to do.

", "rawMarkdown": "", "votes": null }, { "id": "66062", "postDate": "03/12/2015 21:09:38", "content": "

Hi to all,

\n

I have created a small github repository with the latest release of my version of fast_solution.py, heavily modified with both Google's FTRL Proximal and Microsoft's AdPredictor. Here it is: http://bit.ly/GHFASTDG. My team came to the 22th rank by using an ensembling of FTRL, AdPredictor and Vowpal Wabbit models.

\n

Many thanks to tinrtgu for providing the original code and the inspiration!

\n

You can also read some write-up's on the competition at http://bit.ly/CTRK1 and http://bit.ly/CBLOGBOOK1.

", "rawMarkdown": "", "votes": null }, { "id": "89310", "postDate": "08/13/2015 17:54:13", "content": "

Hi if anyone is still reading this thread I could use some help understanding the predict method in the script. I was wondering if you could explain the reasoning behind the way wTx inner product is being implemented in the predict method. From reading the code it seems that the x features that have been hashed are not being used in the computation of the weight vector. Shouldn't the the inner product of the predict function be wTx = w[i] * i, which would follow along the lines of a typical logistic regression computation? Because currently it looks like it is just summing the computed weights, with no affect from the actual feature vector.

", "rawMarkdown": "Hi if anyone is still reading this thread I could use some help understanding the predict method in the script. I was wondering if you could explain the reasoning behind the way wTx inner product is being implemented in the predict method. From reading the code it seems that the x features that have been hashed are not being used in the computation of the weight vector. Shouldn't the the inner product of the predict function be wTx = w[i] * i, which would follow along the lines of a typical logistic regression computation? Because currently it looks like it is just summing the computed weights, with no affect from the actual feature vector.", "votes": null }, { "id": "89315", "postDate": "08/13/2015 18:56:16", "content": "

[quote=Robert;89310]

\n\n

Hi if anyone is still reading this thread I could use some help understanding the predict method in the script. I was wondering if you could explain the reasoning behind the way wTx inner product is being implemented in the predict method. From reading the code it seems that the x features that have been hashed are not being used in the computation of the weight vector. Shouldn't the the inner product of the predict function be wTx = w[i] * i, which would follow along the lines of a typical logistic regression computation? Because currently it looks like it is just summing the computed weights, with no affect from the actual feature vector.

\n\n

[/quote]

\n\n

x is either 1 or 0.

\n\n

It's only calculating the weights for the indices of x that are 1.

\n\n

w[i] * 1 = w[i]

", "rawMarkdown": "[quote=Robert;89310]\r\n\r\nHi if anyone is still reading this thread I could use some help understanding the predict method in the script. I was wondering if you could explain the reasoning behind the way wTx inner product is being implemented in the predict method. From reading the code it seems that the x features that have been hashed are not being used in the computation of the weight vector. Shouldn't the the inner product of the predict function be wTx = w[i] * i, which would follow along the lines of a typical logistic regression computation? Because currently it looks like it is just summing the computed weights, with no affect from the actual feature vector.\r\n\r\n[/quote]\r\n\r\n`x` is either 1 or 0.\r\n\r\nIt's only calculating the weights for the indices of `x` that are 1. \r\n\r\n`w[i] * 1 = w[i]`", "votes": null }, { "id": "89319", "postDate": "08/13/2015 20:43:33", "content": "

[quote=inversion;89315]

\n\n

[quote=Robert;89310]

\n\n

Hi if anyone is still reading this thread I could use some help understanding the predict method in the script. I was wondering if you could explain the reasoning behind the way wTx inner product is being implemented in the predict method. From reading the code it seems that the x features that have been hashed are not being used in the computation of the weight vector. Shouldn't the the inner product of the predict function be wTx = w[i] * i, which would follow along the lines of a typical logistic regression computation? Because currently it looks like it is just summing the computed weights, with no affect from the actual feature vector.

\n\n

[/quote]

\n\n

x is either 1 or 0.

\n\n

It's only calculating the weights for the indices of x that are 1.

\n\n

w[i] * 1 = w[i]

\n\n

[/quote]

\n\n

I thought x was a vector that contained the hashed values, which are indexed values from 1 to D. And the predict method is looping through the x vector with i. Or is x 0 or 1 due to the one hot encoding? Thanks for the reply by the way.

", "rawMarkdown": "[quote=inversion;89315]\r\n\r\n[quote=Robert;89310]\r\n\r\nHi if anyone is still reading this thread I could use some help understanding the predict method in the script. I was wondering if you could explain the reasoning behind the way wTx inner product is being implemented in the predict method. From reading the code it seems that the x features that have been hashed are not being used in the computation of the weight vector. Shouldn't the the inner product of the predict function be wTx = w[i] * i, which would follow along the lines of a typical logistic regression computation? Because currently it looks like it is just summing the computed weights, with no affect from the actual feature vector.\r\n\r\n[/quote]\r\n\r\n`x` is either 1 or 0.\r\n\r\nIt's only calculating the weights for the indices of `x` that are 1. \r\n\r\n`w[i] * 1 = w[i]`\r\n\r\n[/quote]\r\n\r\n\r\nI thought x was a vector that contained the hashed values, which are indexed values from 1 to D. And the predict method is looping through the x vector with i. Or is x 0 or 1 due to the one hot encoding? Thanks for the reply by the way.", "votes": null }, { "id": "89357", "postDate": "08/14/2015 12:03:02", "content": "

[quote=Robert;89319]

\n\n

I thought x was a vector that contained the hashed values, which are indexed values from 1 to D. And the predict method is looping through the x vector with i. Or is x 0 or 1 due to the one hot encoding? Thanks for the reply by the way.

\n\n

[/quote]

\n\n

I should have been more clear. Yes, x is the hashed values, the hashed valued being the index position of the 1's. (Due to the OneHotEncoding, as you point out.) So x is not really x; rather, it is a sparse representation of x, e.g.,

\n\n

\nx = [2,5]\n

\n\n

represents

\n\n

\nx = [0, 0, 1, 0, 0, 1]\n

", "rawMarkdown": "[quote=Robert;89319]\r\n\r\nI thought x was a vector that contained the hashed values, which are indexed values from 1 to D. And the predict method is looping through the x vector with i. Or is x 0 or 1 due to the one hot encoding? Thanks for the reply by the way. \r\n\r\n[/quote]\r\n\r\nI should have been more clear. Yes, `x` is the hashed values, the hashed valued being the index position of the 1's. (Due to the OneHotEncoding, as you point out.) So `x` is not really `x`; rather, it is a sparse representation of `x`, e.g.,\r\n\r\n```\r\nx = [2,5]\r\n```\r\n\r\nrepresents\r\n\r\n```\r\nx = [0, 0, 1, 0, 0, 1]\r\n```", "votes": null }, { "id": "96647", "postDate": "10/19/2015 12:45:48", "content": "

This may sound like a naive question but in previous version here the update rule was:

\n\n
\n

w[i] -= (p - y) * alpha / (sqrt(n[i]) + 1.)

\n
\n\n

But in new this version it has changed to:

\n\n
\n

g = p - y

\n \n

z[i] += g - sigma * w[i]

\n
\n\n

Why is the correction being added instead of subtracted now? Please tell me what I missed.

", "rawMarkdown": "This may sound like a naive question but in previous version [here][1] the update rule was:\r\n\r\n> w[i] -= (p - y) * alpha / (sqrt(n[i]) + 1.)\r\n\r\nBut in new this version it has changed to:\r\n\r\n> g = p - y \r\n\r\n> z[i] += g - sigma * w[i]\r\n\r\nWhy is the correction being added instead of subtracted now? Please tell me what I missed.\r\n\r\n [1]: https://www.kaggle.com/c/criteo-display-ad-challenge/forums/t/10322/beat-the-benchmark-with-less-then-200mb-of-memory", "votes": null }, { "id": "383591", "postDate": "09/09/2018 03:31:44", "content": "

Thanks!

", "rawMarkdown": "Thanks!", "votes": null }, { "id": "585879", "postDate": "07/28/2019 08:39:19", "content": "

Hi\nI had the same question. Then I found some commends which may help.

\n\n

\"\"\"Predict for features.\n Parameters:\n ----------\n x (list of int): a list of index of non-zero features\n Outputs:\n ----------\n p (float): prediction for input features\n \"\"\"

", "rawMarkdown": "Hi\nI had the same question. Then I found some commends which may help.\n\n\n \"\"\"Predict for features.\n Parameters:\n ----------\n x (list of int): a list of index of non-zero features\n Outputs:\n ----------\n p (float): prediction for input features\n \"\"\"", "votes": null }, { "id": "585880", "postDate": "07/28/2019 08:40:09", "content": "

Hi,\nI had the same question. Then I found some commends which may help.

\n\n

\"\"\"Predict for features.\n Parameters:\n ----------\n x (list of int): a list of index of non-zero features\n Outputs:\n ----------\n p (float): prediction for input features\n \"\"\"

", "rawMarkdown": "Hi,\nI had the same question. Then I found some commends which may help.\n\n\n \"\"\"Predict for features.\n Parameters:\n ----------\n x (list of int): a list of index of non-zero features\n Outputs:\n ----------\n p (float): prediction for input features\n \"\"\"", "votes": null }, { "id": "1104283", "postDate": "12/06/2020 19:15:03", "content": "

who knows where is file with code?

", "rawMarkdown": "who knows where is file with code?", "votes": null } ], "comments": [ { "id": 1104283, "author_name": "sandystep", "author_url": "", "post_date": "12/06/2020 19:15:03", "content": "

who knows where is file with code?

", "votes": null, "replies": [] }, { "id": 58055, "author_name": "jiweiliu", "author_url": "", "post_date": "11/14/2014 06:02:21", "content": "

I wish I could vote up more than once. 

\n

This code is classic. Thank you!

\n\n

And 1MB ? really ? :P

", "votes": null, "replies": [] }, { "id": 58056, "author_name": "ggglhf", "author_url": "", "post_date": "11/14/2014 06:03:32", "content": "

[quote=rcarson;58055]

\n

I wish I could vote up more than once. 

\n

This code is classic. Thank you!

\n

And 1MB ? really ? :P

\n

[/quote]

\n

1MB is too much for beating the benchmark. It is hard not to beat the benchmark for this one.

", "votes": null, "replies": [] }, { "id": 58057, "author_name": "herimanitra", "author_url": "", "post_date": "11/14/2014 06:56:29", "content": "

Hi all,

\n

How one can check memory usage during model training in Linux?

", "votes": null, "replies": [] }, { "id": 58058, "author_name": "ggglhf", "author_url": "", "post_date": "11/14/2014 08:08:44", "content": "

[quote=Herimanitra;58057]

\n

Hi all,

\n

How one can check memory usage during model training in Linux?

\n

[/quote]

\n

top

\n

https://duckduckgo.com/?q=top+linux&t=canonical

", "votes": null, "replies": [] }, { "id": 58061, "author_name": "martely", "author_url": "", "post_date": "11/14/2014 10:14:39", "content": "

Classy!

\n

Congratulations & many thanks!

", "votes": null, "replies": [] }, { "id": 58062, "author_name": "martely", "author_url": "", "post_date": "11/14/2014 10:21:05", "content": "

Revised version, with more control capabilities from command line.

\n

This is really now big fun!

", "votes": null, "replies": [] }, { "id": 58063, "author_name": "ggglhf", "author_url": "", "post_date": "11/14/2014 10:44:12", "content": "

[quote=Yannick Martel;58062]

\n

Revised version, with more control capabilities from command line.

\n

This is really now big fun!

\n

[/quote]

\n

Should I create a github repo so that people, like you, can also contribute and make the script better?

", "votes": null, "replies": [] }, { "id": 58064, "author_name": "domcastro", "author_url": "", "post_date": "11/14/2014 10:53:07", "content": "

You don't want it becoming a burden on you

", "votes": null, "replies": [] }, { "id": 58065, "author_name": "ggglhf", "author_url": "", "post_date": "11/14/2014 10:54:19", "content": "

[quote=ACS69;58064]

\n

You don't want it becoming a burden on you

\n

[/quote]

\n

OK, giving up right now, I'll just continue to share on the forums :P

\n

BTW, nice Homer thumbnail.

", "votes": null, "replies": [] }, { "id": 58066, "author_name": "domcastro", "author_url": "", "post_date": "11/14/2014 10:56:08", "content": "

lol! Your code is too beautiful to become a burden!

\n\n

edit: Homer in a West Ham kit

", "votes": null, "replies": [] }, { "id": 58069, "author_name": "rohanrao", "author_url": "", "post_date": "11/14/2014 13:02:50", "content": "

The subject should've been "Beat the benchmark in future with less than 1MB of memory" !

", "votes": null, "replies": [] }, { "id": 58072, "author_name": "mmueller", "author_url": "", "post_date": "11/14/2014 13:52:53", "content": "

[quote=tinrtgu;58063]

\n

[quote=Yannick Martel;58062]

\n

Revised version, with more control capabilities from command line.

\n

This is really now big fun!

\n

[/quote]

\n

Should I create a github repo so that people, like you, can also contribute and make the script better?

\n

[/quote]  

\n

great idea .. let's create a pykaggle lib. :)

", "votes": null, "replies": [] }, { "id": 58083, "author_name": "martely", "author_url": "", "post_date": "11/14/2014 18:06:40", "content": "

[quote=Faron;58072]

\n

[quote=tinrtgu;58063]

\n

[quote=Yannick Martel;58062]

\n

Revised version, with more control capabilities from command line.

\n

This is really now big fun!

\n

[/quote]

\n

Should I create a github repo so that people, like you, can also contribute and make the script better?

\n

[/quote]  

\n

great idea .. let's create a pykaggle lib. :)

\n

[/quote]

\n

Yes, it would be a really great idea.

", "votes": null, "replies": [] }, { "id": 58089, "author_name": "fchollet", "author_url": "", "post_date": "11/14/2014 21:17:37", "content": "

Awesome! Thank you for sharing this. The method is pretty much state of the art, and the code quality is excellent! Huge thumb up.

\n

If you guys haven't already, be sure to check out the Google paper that describes this method. Hint: it contains useful info on how to improve the algo :) 

\n

http://www.eecs.tufts.edu/~dsculley/papers/ad-click-prediction.pdf

", "votes": null, "replies": [] }, { "id": 58091, "author_name": "paweljankiewicz", "author_url": "", "post_date": "11/14/2014 21:26:03", "content": "

@tinrtgu Great job! Also I like your optimism. You already assumed that there will be a 2nd revision of the dataset :).

\n

train = 'train_rev2'
test = 'test_rev2'

\n

I will try to rewrite this in Cython to see what speed-up I can get.

", "votes": null, "replies": [] }, { "id": 58092, "author_name": "inversion", "author_url": "", "post_date": "11/14/2014 21:53:46", "content": "

[quote=Pawel;58091]

\n

@tinrtgu Great job! Also I like your optimism. You already assumed that there will be a 2nd revision of the dataset :).

\n

train = 'train_rev2'
test = 'test_rev2'

\n

I will try to rewrite this in Cython to see what speed-up I can get.

\n

[/quote]

\n

Actually, we'll be on rev3 once the competition starts back up. Rev2 was also broken.

", "votes": null, "replies": [] }, { "id": 58093, "author_name": "paweljankiewicz", "author_url": "", "post_date": "11/14/2014 21:59:21", "content": "

Oh. I must have missed that revision :).

", "votes": null, "replies": [] }, { "id": 58095, "author_name": "ggglhf", "author_url": "", "post_date": "11/14/2014 22:42:12", "content": "

[quote=Pawel;58091]

\n

I will try to rewrite this in Cython to see what speed-up I can get.

\n

[/quote]

\n

I think you are able to get a fair amount of speed up, since I was trying to go for maximum readability the script is actually lacking a bit in the run speed and memory utilization department.

", "votes": null, "replies": [] }, { "id": 58108, "author_name": "paweljankiewicz", "author_url": "", "post_date": "11/15/2014 11:38:10", "content": "

Around line 268 there is this 

\n

for key in sorted(row): # sort is for preserving feature ordering

\n

I think this sort is unnecessary. The results are the same if you skip it.

\n

for key in row:

", "votes": null, "replies": [] }, { "id": 58110, "author_name": "paweljankiewicz", "author_url": "", "post_date": "11/15/2014 12:11:00", "content": "

I'm trying to wrap my head around this problem. I don't understand why the lazy weights (self.w) are changed in the predict method. IMO model shouldn't be changed during prediction. It would be more logical to me if the predict method loop looked like this:

\n

wTx = 0.
for i in self._indices(x):
    wTx += w[i]

\n

And move all the updates of the model in the actual update method. It does change slightly the results because the updates are made after seeing the observation

\n

See this gist:

\n

https://gist.github.com/pjankiewicz/8ab7094d263bf0d4cfb8

", "votes": null, "replies": [] }, { "id": 58111, "author_name": "ggglhf", "author_url": "", "post_date": "11/15/2014 12:30:39", "content": "

[quote=Pawel;58108]

\n

Around line 268 there is this 

\n

for key in sorted(row): # sort is for preserving feature ordering

\n

I think this sort is unnecessary. The results are the same if you skip it.

\n

for key in row:

\n

[/quote]

\n

Yep, you are right and thanks for pointing this out. But that is only because I used the following hashing scheme for feature interactions, which does not depends on the order of i and j.

\n

for i in xrange(1, L): # skip bias term, so we start at 1
    for j in xrange(i+1, L):
        yield (i * j) % D

\n

I actually have thought about this, to leave in the sort of the take it out. In the end I've decided to leave in the sort since I think there is a high chance people might want to change the hashing method and this would be a hard to catch bug for them.

\n

Since you know what you are doing please feel free to remove it.

", "votes": null, "replies": [] }, { "id": 58112, "author_name": "ggglhf", "author_url": "", "post_date": "11/15/2014 12:34:34", "content": "

[quote=Pawel]

\n

I'm trying to wrap my head around this problem. I don't understand why the lazy weights (self.w) are changed in the predict method. IMO model shouldn't be changed during prediction. It would be more logical to me if the predict method loop looked like this:

\n

wTx = 0.
for i in self._indices(x):
    wTx += w[i]

\n

And move all the updates of the model in the actual update method. It does change slightly the results because the updates are made after seeing the observation

\n

See this gist:

\n

https://gist.github.com/pjankiewicz/8ab7094d263bf0d4cfb8

\n

[/quote]

\n

You are right again, geez you are good at this!

\n

Calculating w at prediction time is there for the support of memory saving mode. If you try to search for "# use this for memory usage reduction" in the code you can actually find out that we actually don't need the full w all the time. We can remove (the full) w and save up to 30% of the memory consumption.

", "votes": null, "replies": [] }, { "id": 58118, "author_name": "rohanrao", "author_url": "", "post_date": "11/15/2014 18:07:45", "content": "

tinrtgu: You should start a series of "Beating the benchmark with xxx score using less than yyy memory in just zzz lines of code!" and give Abhishek a run for his money!

\n

I won't be surprised if some day an innocent but wise Kaggler wins a competition by ensembling all these wonderful BeatingTheBenchmark codes from the forums! ;-)

", "votes": null, "replies": [] }, { "id": 58130, "author_name": "ggglhf", "author_url": "", "post_date": "11/16/2014 08:10:53", "content": "

[quote=Rohan Rao;58118]

\n

tinrtgu: You should start a series of "Beating the benchmark with xxx score using less than yyy memory in just zzz lines of code!" and give Abhishek a run for his money!

\n

[/quote]

\n

I'm hoping more of a hackathon competition where there are 100 datasets and 100 benchmarks, the winner is the one who can beat the most in 24 hours.

", "votes": null, "replies": [] }, { "id": 58131, "author_name": "rohanrao", "author_url": "", "post_date": "11/16/2014 11:40:04", "content": "

[quote=tinrtgu;58130]

\n

I'm hoping more of a hackathon competition where there are 100 datasets and 100 benchmarks, the winner is the one who can beat the most in 24 hours.

\n

[/quote]

\n

Hack-a-ton!

", "votes": null, "replies": [] }, { "id": 58158, "author_name": "paweljankiewicz", "author_url": "", "post_date": "11/16/2014 22:22:56", "content": "

@tinrtgu I got another thing ;). Much more subtle this time.

\n

The way you introduced interactions:

\n

yield (i * j) % D

\n

is somehow faulty because it produces non-uniform hashes. For example 2 random numbers are much more likely to produce 0 as the last digit.

\n

A small histogram of hashes for ((i * j) % D) % 10, where i,j are random numbers from range D

\n

0 *******************
1 ***
2 ******
3 ***
4 *********
5 ***
6 ******
7 ***
8 ************
9 ***

\n

Interactions should be introduced before hashing with something like to prevent this:

\n

hash(col1 + '_' + value1 + '_' + col2 + '_' + value2) % D

\n

EDIT: There may be a simpler solution actually 

\n

yield hash(str(i) + '_' + str(j)) % D

", "votes": null, "replies": [] }, { "id": 58160, "author_name": "ggglhf", "author_url": "", "post_date": "11/16/2014 23:07:18", "content": "

[quote=Pawel;58158]

\n

The way you introduced interactions:

\n

yield (i * j) % D

\n

is somehow faulty because it produces non-uniform hashes. For example 2 random numbers are much more likely to produce 0 as the last digit.

\n

Interactions should be introduced before hashing with something like to prevent this:

\n

hash(col1 + '_' + value1 + '_' + col2 + '_' + value2) % D

\n

EDIT: There may be a simpler solution actually 

\n

yield hash(str(i) + '_' + str(j)) % D

\n

[/quote]

\n

Yep, it is a trade-off between speed and accuracy, and this is also the reason that I left in the sorted feature list. I really appreciate all your feedback. Do you mind if I add your feedback into the script so that stuff is more clear to others?

", "votes": null, "replies": [] }, { "id": 58161, "author_name": "paweljankiewicz", "author_url": "", "post_date": "11/16/2014 23:12:07", "content": "

Sure. You can add it to the script.

", "votes": null, "replies": [] }, { "id": 58182, "author_name": "mmueller", "author_url": "", "post_date": "11/17/2014 07:07:49", "content": "

great work is going on here .. thumps up!

", "votes": null, "replies": [] }, { "id": 58220, "author_name": "fchollet", "author_url": "", "post_date": "11/17/2014 19:50:22", "content": "

[quote=tinrtgu;58054]

\n

I would say it should easily beat 0.40

\n

[/quote]

\n

Anyone tried training this code on the first 9 days and testing on the 10th? Which would be a similar setup to the leaderboard. 

\n

Doing this gets me a score of 0.45 after one epoch, which does not beat a constant CTR prediction (~0.43). Some parameter optimization gets me down to 0.40, but that is still quite high compared to what I would expect from this approach. 

\n

In general, the holdout validation will show very low scores because the samples have strong local correlation (temporally). It is much harder to get a model that generalizes to the next day. That is made even more prominent by using high learning rates, since then the model becomes locally specialized (overfit) to whatever segment of time it is currently learning.

\n

Some ideas to fix it (hope that will help others):

\n

- shuffling the training data (difficult to do in a memory-conscious way)

\n

- prediction calibration

", "votes": null, "replies": [] }, { "id": 58221, "author_name": "konradb", "author_url": "", "post_date": "11/17/2014 20:23:55", "content": "

@fchollet: one approach to shuffling which kinda seems to work is shuffle the data as a separate task before commencing actual modelling:

\n

- load the data in R using the ff package (super low memory footprint compared to the file size - direct read.table fails miserably)

\n

- shuffle using ffindexget / ffindexorder

\n

- store using ff wrapper around write.csv

\n\n

Admittedly not very clean, but it does get the job done.

", "votes": null, "replies": [] }, { "id": 58223, "author_name": "paweljankiewicz", "author_url": "", "post_date": "11/17/2014 20:31:28", "content": "

Another idea is to create a shell script that reads the standard input, buffers it (it is a parameter), shuffles and prints to standard output. I've seen such script in one of the Criteo solutions.

", "votes": null, "replies": [] }, { "id": 58277, "author_name": "ggglhf", "author_url": "", "post_date": "11/18/2014 04:45:07", "content": "

[quote=fchollet;58220]

\n

[quote=tinrtgu;58054]

\n

I would say it should easily beat 0.40

\n

[/quote]

\n

Anyone tried training this code on the first 9 days and testing on the 10th? Which would be a similar setup to the leaderboard. 

\n

Doing this gets me a score of 0.45 after one epoch, which does not beat a constant CTR prediction (~0.43). Some parameter optimization gets me down to 0.40, but that is still quite high compared to what I would expect from this approach. 

\n

[/quote]

\n

Weird, I tried to do the same thing as you stated, train with day 1 to 9 and test on day 10, and I can get logloss=0.392 on day 10 with just one epoch.

\n

Below are the parameters that I used.
alpha = .1
beta = 1.
L1 = 1.
L2 = 1.
D = 2 ** 20

\n

EDIT

\n

I have added my code (fast_solution_v3.py) to the original post, without changing any parameters it should train on day 1 to 9 for one epoch and give a validation score of 0.392 on day 10.

", "votes": null, "replies": [] }, { "id": 58334, "author_name": "abhishek", "author_url": "", "post_date": "11/18/2014 18:48:58", "content": "

v3. without change: LB: 0.6931472

", "votes": null, "replies": [] }, { "id": 58335, "author_name": "domcastro", "author_url": "", "post_date": "11/18/2014 18:54:40", "content": "

strange - are you sure it's not writing the submission file wrong? as it's the sample submission score

", "votes": null, "replies": [] }, { "id": 58336, "author_name": "abhishek", "author_url": "", "post_date": "11/18/2014 19:01:27", "content": "

[quote=ACS69;58335]

\n

strange - are you sure it's not writing the submission file wrong? as it's the sample submission score

\n

[/quote]

\n

sure. all predictions generated by v3 are 0.5

", "votes": null, "replies": [] }, { "id": 58341, "author_name": "fchollet", "author_url": "", "post_date": "11/18/2014 19:59:52", "content": "

[quote=Abhishek;58336]

\n

[quote=ACS69;58335]

\n

strange - are you sure it's not writing the submission file wrong? as it's the sample submission score

\n

[/quote]

\n

sure. all predictions generated by v3 are 0.5

\n

[/quote]

\n

No, your predictions are 0.5 because you are running an untrained model (all predictions are 0, and sigmoid(0) = 0.5 with the sigmoid function used in the code). And you are running an untrained model because of the old date-based code switch that controls when to train the model.

\n

Since the dates in the data have changed, you need to update the code to take into account the new dates.

\n

Quick fix: replace line ~230

\n

date = int(row['hour'][4:6])

\n

with 

\n

date = int(row['hour'][4:6]) - 20

", "votes": null, "replies": [] }, { "id": 58343, "author_name": "jianminsun", "author_url": "", "post_date": "11/18/2014 20:15:30", "content": "

or you can change 

\n

holdafter = 9

\n

to 

\n

holdafter = 29

\n

[quote=fchollet;58341]

\n

[quote=Abhishek;58336]

\n

[quote=ACS69;58335]

\n

strange - are you sure it's not writing the submission file wrong? as it's the sample submission score

\n

[/quote]

\n

sure. all predictions generated by v3 are 0.5

\n

[/quote]

\n

No, your predictions are 0.5 because you are running an untrained model (all predictions are 0, and sigmoid(0) = 0.5 with the sigmoid function used in the code). And you are running an untrained model because of the old date-based code switch that controls when to train the model.

\n

Since the dates in the data have changed, you need to update the code to take into account the new dates.

\n

Quick fix: replace line ~230

\n

date = int(row['hour'][4:6])

\n

with 

\n

date = int(row['hour'][4:6]) - 20

\n

[/quote]

", "votes": null, "replies": [] }, { "id": 58352, "author_name": "jiweiliu", "author_url": "", "post_date": "11/18/2014 21:50:33", "content": "

Version 3

\n

hold after 29: validation loss 0.399625, LB 0.3977417

\n

set D to 2**24

", "votes": null, "replies": [] }, { "id": 58366, "author_name": "jialunhe", "author_url": "", "post_date": "11/19/2014 03:54:09", "content": "

Here is the record of my testing on a workstation with Xeon CPU:

\n

python fast_solution_v3.py
Epoch 0 finished, validation logloss: 0.399873, elapsed time: 0:36:26.541000
Epoch 1 finished, validation logloss: 0.400307, elapsed time: 1:12:45.315000
Epoch 2 finished, validation logloss: 0.401073, elapsed time: 1:48:59.706000
Epoch 3 finished, validation logloss: 0.402093, elapsed time: 2:25:13.176000
Epoch 4 finished, validation logloss: 0.403243, elapsed time: 3:01:27.582000
Epoch 5 finished, validation logloss: 0.404274, elapsed time: 3:37:40.116000
Epoch 6 finished, validation logloss: 0.405214, elapsed time: 4:13:53.289000
Epoch 7 finished, validation logloss: 0.406166, elapsed time: 4:50:06.210000
Epoch 8 finished, validation logloss: 0.407127, elapsed time: 5:26:19.867000
Epoch 9 finished, validation logloss: 0.408056, elapsed time: 6:02:41.431000

", "votes": null, "replies": [] }, { "id": 58380, "author_name": "phunter", "author_url": "", "post_date": "11/19/2014 07:09:50", "content": "

@Birchwood try pypy? it is much faster

", "votes": null, "replies": [] }, { "id": 58448, "author_name": "phanisrikanth", "author_url": "", "post_date": "11/19/2014 21:56:45", "content": "

Here is the log of my attempt with tinrtgu's script and a slight tweaking of hyperparameters!

\n

alpha = .15

\n

beta = 1.1

\n

L1 = 1.1

\n

L2 = 1.1

\n

D = 2 ** 22

\n

$ pypy fast_solution_v3.py
Epoch 0 finished, validation logloss: 0.399544, elapsed time: 0:42:58.529000
Epoch 1 finished, validation logloss: 0.399377, elapsed time: 1:35:50.060000

\n

LB score: 0.3992365

", "votes": null, "replies": [] }, { "id": 58464, "author_name": "phunter", "author_url": "", "post_date": "11/20/2014 00:35:50", "content": "

[quote=binga;58448]

\n

Here is the log of my attempt with tinrtgu's script and a slight tweaking of hyperparameters!

\n

alpha = .15

\n

beta = 1.1

\n

L1 = 1.1

\n

L2 = 1.1

\n

D = 2 ** 22

\n

$ pypy fast_solution_v3.py
Epoch 0 finished, validation logloss: 0.399544, elapsed time: 0:42:58.529000
Epoch 1 finished, validation logloss: 0.399377, elapsed time: 1:35:50.060000

\n

LB score: 0.3992365

\n

[/quote]

\n

Just a friendly reminder: it is not a good idea of sharing fine-tuned parameters before the end of the competition.

", "votes": null, "replies": [] }, { "id": 58497, "author_name": "phanisrikanth", "author_url": "", "post_date": "11/20/2014 07:11:20", "content": "

[quote=phunter;58464]

\n

[quote=binga;58448]

\n

Here is the log of my attempt with tinrtgu's script and a slight tweaking of hyperparameters!

\n

alpha = .15

\n

beta = 1.1

\n

L1 = 1.1

\n

L2 = 1.1

\n

D = 2 ** 22

\n

$ pypy fast_solution_v3.py
Epoch 0 finished, validation logloss: 0.399544, elapsed time: 0:42:58.529000
Epoch 1 finished, validation logloss: 0.399377, elapsed time: 1:35:50.060000

\n

LB score: 0.3992365

\n

[/quote]

\n

Just a friendly reminder: it is not a good idea of sharing fine-tuned parameters before the end of the competition.

\n

[/quote]

\n

I got that score by just running the code, which I believe anybody who is interested in solving could do! As we have a lot of time to the end of competition, this score will really not matter, as I am somewhere at 60th on the LB. Anyways, if the guys here are still insecure about it, I'll keep good things to myself. Never mind.

", "votes": null, "replies": [] }, { "id": 58514, "author_name": "inversion", "author_url": "", "post_date": "11/20/2014 11:49:39", "content": "

It's going to take a lot more than just optimizing the parameters to get into the top 10.

", "votes": null, "replies": [] }, { "id": 58522, "author_name": "gertjac", "author_url": "", "post_date": "11/20/2014 13:45:32", "content": "

I just downloaded and tried pypy, it really is a lot faster!

\n

on my machine (v3, D = 2 ** 24, holdafter=29)

\n

pypy: Epoch 0 finished, validation logloss: 0.399601, elapsed time: 0:16:30.555000

\n

python: Epoch 0 finished, validation logloss: 0.399601, elapsed time: 0:56:32.461000

", "votes": null, "replies": [] }, { "id": 59144, "author_name": "martely", "author_url": "", "post_date": "11/28/2014 08:13:38", "content": "

Hello all,

\n

I have worked on the fast_solution great piece of code and added a few features:

\n

- parameter control from command line

\n

- dropout: where you randomly drop features out of consideration - prevents overfitting and features adaptation

\n

- inclusion of the # of samples seen per device_ip / device_id as a feature

\n

- inclusion of weedkay as a feature

\n

I am down to roughly 0.3948 on the LB with a lot of tuning, by running on all test samples - with a few hours of processing using pypy on an i7, using a few GB's of RAM. LB is very close to my own cross-validation (run on all days except 30, test on 30).

\n

The code probably would benefit from some optimization. I have written the first version in Julia - it was faster than C-Python, but 2-3 times slower than pypy, and I quickly reverted to Python.

\n

I would be glad of any constructive feedback.

\n

Yannick

", "votes": null, "replies": [] }, { "id": 59146, "author_name": "ggglhf", "author_url": "", "post_date": "11/28/2014 08:22:37", "content": "

@Yannick Martel

\n

Glad to see you adding new features to the benchmark code, and I really like your idea of using dropout.

", "votes": null, "replies": [] }, { "id": 59149, "author_name": "nthanhtam", "author_url": "", "post_date": "11/28/2014 09:50:04", "content": "

[quote=Yannick Martel;59144]

\n

- inclusion of the # of samples seen per device_ip / device_id as a feature

\n

[/quote]

\n

For the counters, I think we should use defaultdict, then we don't have to catch KeyError exception. For instance, just replace:

\n

device_ip_counter = {}
device_id_counter = {}

\n

by

\n

device_ip_counter = defaultdict(int)
device_id_counter =  defaultdict(int)

", "votes": null, "replies": [] }, { "id": 59200, "author_name": "drpatrickchan", "author_url": "", "post_date": "11/29/2014 04:19:39", "content": "

Thanks Yannick.. curious to know how the dropout is impacting your accuracy...

\n

[quote=Yannick Martel;59144]

\n

Hello all,

\n

I have worked on the fast_solution great piece of code and added a few features:

\n

- parameter control from command line

\n

- dropout: where you randomly drop features out of consideration - prevents overfitting and features adaptation

\n

- inclusion of the # of samples seen per device_ip / device_id as a feature

\n

- inclusion of weedkay as a feature

\n

I am down to roughly 0.3948 on the LB with a lot of tuning, by running on all test samples - with a few hours of processing using pypy on an i7, using a few GB's of RAM. LB is very close to my own cross-validation (run on all days except 30, test on 30).

\n

The code probably would benefit from some optimization. I have written the first version in Julia - it was faster than C-Python, but 2-3 times slower than pypy, and I quickly reverted to Python.

\n

I would be glad of any constructive feedback.

\n

Yannick

\n

[/quote]

", "votes": null, "replies": [] }, { "id": 59210, "author_name": "herimanitra", "author_url": "", "post_date": "11/29/2014 09:04:25", "content": "

@Yannick

\n

Could you give an example (not your optimal parameter) on how to run this script. I'm stuck with the command line parameters.

\n

I already have all the stuff installed...

\n

Thks

\n

[quote=Yannick Martel;59144]

\n

Hello all,

\n

I have worked on the fast_solution great piece of code and added a few features:

\n

- parameter control from command line

\n

- dropout: where you randomly drop features out of consideration - prevents overfitting and features adaptation

\n

- inclusion of the # of samples seen per device_ip / device_id as a feature

\n

- inclusion of weedkay as a feature

\n

I am down to roughly 0.3948 on the LB with a lot of tuning, by running on all test samples - with a few hours of processing using pypy on an i7, using a few GB's of RAM. LB is very close to my own cross-validation (run on all days except 30, test on 30).

\n

The code probably would benefit from some optimization. I have written the first version in Julia - it was faster than C-Python, but 2-3 times slower than pypy, and I quickly reverted to Python.

\n

I would be glad of any constructive feedback.

\n

Yannick

\n

[/quote]

", "votes": null, "replies": [] }, { "id": 59216, "author_name": "martely", "author_url": "", "post_date": "11/29/2014 10:40:15", "content": "

@all

\n

Yes, the script is not as user-friendly as I would have liked :-)

\n

You can have some help output by running:

\n

python fast_solution_plus.py --help

\n

Then to learn you do something like:

\n

  python fast_solution_plus.py train --train <training csv file> -o <output model file>

\n

[ like in: python fast_solution_plus.py train --train train.csv -o first.model.gz ]

\n

Once your model is trained, you can predict:

\n

  python fast_solution_plus.py predict --test test.csv -i first.model.gz -p my.submission.csv.gz

\n

You can:

\n

- usually provide a .csv.gz instead of .csv only - it decompress on the fly

\n

- use pypy instead of python

\n

- provide additionnal tuning parameters: --L1 1.0 --L2 0.1 --dropout 0.4, and so on (you get the list when running with -h or --help)

\n

Have fun!

\n

@ Patrick: on my experiment, dropout does reduce indeed overfit and improve logloss. I get very similar figures between my own cross-validation and the LB.

\n

With dropout, I recommend to run multiple passes (--n_epochs) to avoid "loosing" some pieces of samples, depending on the dropout factor.

\n\n

Yannick

", "votes": null, "replies": [] }, { "id": 59235, "author_name": "grigorydymov", "author_url": "", "post_date": "11/29/2014 18:07:42", "content": "

Did anyone have any luck with feature interactions? It seems so attractive to add them, but howevery I try, they do not give any improvement.

", "votes": null, "replies": [] }, { "id": 59252, "author_name": "ngutten", "author_url": "", "post_date": "11/30/2014 02:34:18", "content": "

I'm not sure I understand how the dropout parameter is working in this new code. It seems to me that meaningful values should be between 0 and 1, but when I was doing parameter tuning I found that the logloss changed continuously even for values above dropout=1 (I went as high as 1.4, and the logloss just kept smoothly changing).

\n

When I looked at the code, it seems that 1 is treated as a special case, but that anything above 1 should be identical (drop all features, I guess, so I should get nonsense or at least just the logloss associated with the mean ctr).

\n

Any idea what's going on here?

", "votes": null, "replies": [] }, { "id": 59253, "author_name": "alexistaugeron", "author_url": "", "post_date": "11/30/2014 03:28:57", "content": "

Here's my understanding:

\n\n

If dropout > 1 you don't drop anything (random.random() > dropout is always False), but you do divide wTx by dropout because the dropped list is [] instead of None (if dropped != None: wTx /= dropout).

\n\n

Thus you are using your regular model, except you're tweaking wTx on each prediction.

", "votes": null, "replies": [] }, { "id": 59256, "author_name": "ngutten", "author_url": "", "post_date": "11/30/2014 05:25:43", "content": "

Ah, I see, that makes sense then, thanks!

", "votes": null, "replies": [] }, { "id": 59283, "author_name": "kolobok007", "author_url": "", "post_date": "11/30/2014 19:01:16", "content": "

Can you, guys, please explain how you do cross-validation with the code? As I understood the code calculates logloss for each 100th (default setting) sample eg 100,200,.... At the end it prints something like average logloss for all samles that were not used for training. How accurate is it, I mean how close is it to LB score? What should I change to train it on let's say all days except the 30th day and then test it  on the 30th day (as was described by Yannick) eg change holdout pattern from samples to days? My PC is very old but I am trying to do my best with it. It runs 1 epoch for several hours with PYPY. Maybe I could also try to reduce training size eg train on just 5-10 days and test it on any single day that was not used for training.

\n

Thanks

", "votes": null, "replies": [] }, { "id": 59289, "author_name": "alexistaugeron", "author_url": "", "post_date": "11/30/2014 19:29:37", "content": "

If you are using the original script posted by tinrtgu, you need to set the holdafter parameter to wherever you want to end your training (eg. 29), and set holdout to None.

\n\n

If your computer is old, you might want to split the training data in one file per day and run the script on a subset of the training data. This won't be as accurate but being able to run a lot of experiment quickly is definitely a big advantage.

\n\n

If your computer has enough RAM, you might also want to run several experiments in parallel.

\n\n

Personally I currently train on 21-27, then independently validate on 28, 29 and 30, and print the logloss on each validation day and the average. It's nowhere close to my LB score because my classifier's score varies widely depending on the CTR of the day, and I still haven't figured out how to fix that. But overall if an experiment improves my validation score, it also improves my LB score.

", "votes": null, "replies": [] }, { "id": 59372, "author_name": "tanopereira", "author_url": "", "post_date": "12/02/2014 03:28:47", "content": "

I've been able to go down to 0.376 in validation with interactions, but LB=0.41 ...

", "votes": null, "replies": [] }, { "id": 59441, "author_name": "ggglhf", "author_url": "", "post_date": "12/03/2014 05:21:57", "content": "

[quote=TanoPereira;59372]

\n

I've been able to go down to 0.376 in validation with interactions, but LB=0.41 ...

\n

[/quote]

\n

Just a hint, overfitting is more severe with interactions so it would be a good idea to increase regularizations.

", "votes": null, "replies": [] }, { "id": 59472, "author_name": "adjgiulio", "author_url": "", "post_date": "12/03/2014 16:06:40", "content": "

I've run this code with pypy on Ubuntu. I wanted to run it on my laptop, W7 64-bit with Python 64 bit, but it seems like pypy will work only with 32-bit Python...

\n

Has anyone ran this on a 64 bit Python on a Windows machine?

", "votes": null, "replies": [] }, { "id": 59474, "author_name": "inversion", "author_url": "", "post_date": "12/03/2014 16:11:56", "content": "

[quote=Giulio;59472]

\n

I've run this code with pypy on Ubuntu. I wanted to run it on my laptop, W7 64-bit with Python 64 bit, but it seems like pypy will work only with 32-bit Python...

\n

Has anyone ran this on a 64 bit Python on a Windows machine?

\n

[/quote]

\n

That's why I switched to Numba (using the Anaconda distribution). It's at least as fast, but also gives me access to my entire system memory.

", "votes": null, "replies": [] }, { "id": 59483, "author_name": "tanopereira", "author_url": "", "post_date": "12/03/2014 17:35:10", "content": "

Can you provide pointers on how to use numba?

\n

I have tried but get compilation errors all over the place...

\n

Thanks!

", "votes": null, "replies": [] }, { "id": 59488, "author_name": "inversion", "author_url": "", "post_date": "12/03/2014 18:30:50", "content": "

[quote=TanoPereira;59483]

\n

Can you provide pointers on how to use numba?

\n

I have tried but get compilation errors all over the place...

\n

Thanks!

\n

[/quote]

\n

I had the same issue. I had to "de-object-orient" the code to make it work. In other words, I took the code out of the Class definition.

\n

Sometimes it can be tricky to get the decorators right. You can use these to get you started. (You'll need to modify depending on how you define your dtypes.)

\n

from numba import int32, float32, float64, void, jit

\n

@jit(float64(int32[:],float32[:],float32[:],float32[:]))
def predict(x, w, z, n):
...

\n

@jit(void(int32[:],float32[:],float32[:],float32[:],float64,float64))
def update(x, w, z, n, p, y):
...
@jit(float64(float64,float64))
def logloss(p, y):

\n\n

Also, you need to change your lists (x, w, z, n) to arrays, e.g.,

\n

w = np.zeros(D, dtype=np.float32)

\n\n

Let me know if you have any other issues. I can assure you, it's worth the time to figure it out.

", "votes": null, "replies": [] }, { "id": 59493, "author_name": "martely", "author_url": "", "post_date": "12/03/2014 20:40:02", "content": "

For my information, how efficient is numba as compared to Pypy? How long does it take for one pass over the full training set?

\n

On my own i7 Linux PC with Pypy, the code provided on the forum takes around 2:30.

", "votes": null, "replies": [] }, { "id": 59497, "author_name": "inversion", "author_url": "", "post_date": "12/03/2014 21:05:26", "content": "

[quote=Yannick Martel;59493]

\n

For my information, how efficient is numba as compared to Pypy? How long does it take for one pass over the full training set?

\n

On my own i7 Linux PC with Pypy, the code provided on the forum takes around 2:30.

\n

[/quote]

\n

I don't have a paired comparison for this code, but I ran the (very similar) benchmark code in the Tradeshift challenge.

\n

Results were:

\n", "votes": null, "replies": [] }, { "id": 59499, "author_name": "jialunhe", "author_url": "", "post_date": "12/03/2014 21:21:51", "content": "

Can you run the python script with numba in command line: python predict.py?

", "votes": null, "replies": [] }, { "id": 59501, "author_name": "inversion", "author_url": "", "post_date": "12/03/2014 22:01:08", "content": "

[quote=Birchwood;59499]

\n

Can you run the python script with numba in command line: python predict.py?

\n

[/quote]

\n

No, you need the special Numba IDE. 

", "votes": null, "replies": [] }, { "id": 59511, "author_name": "tanopereira", "author_url": "", "post_date": "12/04/2014 01:21:11", "content": "

Thank you inversion!

\n

I was able to get the code out of the class, however I get no speed up at all :(

\n

I must be doing something really wrong. My Python abilities are really scarcer than I'd thought.

", "votes": null, "replies": [] }, { "id": 59515, "author_name": "philculliton", "author_url": "", "post_date": "12/04/2014 03:22:41", "content": "

Thanks for the pointers, inversion!

\n

I also had issues with Numba/NumbaPro - I've found that it negatively impacted performance.  I'm using the 64-bit version of PyPy, however, so the comparison is likely different.  So far it's been significantly faster than a numpy rewrite and Numba/NumbaPro.  I'm considering moving the code to Theano or C++ just for fun... we'll see how it compares.

", "votes": null, "replies": [] }, { "id": 59523, "author_name": "herimanitra", "author_url": "", "post_date": "12/04/2014 09:41:09", "content": "

If 'train.csv' is not comma separated, It raises an error. Any hints?

\n

[quote=Yannick Martel;59216]

\n

@all

\n

....

\n

[ like in: python fast_solution_plus.py train --train train.csv -o first.model.gz ]

\n

...

\n

Yannick

\n

[/quote]

", "votes": null, "replies": [] }, { "id": 59526, "author_name": "abhishek", "author_url": "", "post_date": "12/04/2014 09:46:49", "content": "

[quote=Herimanitra;59523]

\n

If 'train.csv' is not comma separated, It raises an error. Any hints?

\n

[quote=Yannick Martel;59216]

\n

@all

\n

....

\n

[ like in: python fast_solution_plus.py train --train train.csv -o first.model.gz ]

\n

...

\n

Yannick

\n

[/quote]

\n

[/quote]

\n\n

Use: csv.DictReader(f, delimiter='\\t', quoting=csv.QUOTE_NONE)

", "votes": null, "replies": [] }, { "id": 59530, "author_name": "herimanitra", "author_url": "", "post_date": "12/04/2014 10:36:47", "content": "

Thanks Abishek,

\n

In my case, DictReader(f_train, delimiter=';')  works!

\n

The script should allow user to bring their own features  into the algorithm, one hot encoding should be an option :)

\n

[quote=Abhishek;59526]

\n

[quote=Herimanitra;59523]

\n

If 'train.csv' is not comma separated, It raises an error. Any hints?

\n

[quote=Yannick Martel;59216]

\n

@all

\n

....

\n

[ like in: python fast_solution_plus.py train --train train.csv -o first.model.gz ]

\n

...

\n

Yannick

\n

[/quote]

\n

[/quote]

\n

Use: csv.DictReader(f, delimiter='\\t', quoting=csv.QUOTE_NONE)

\n

[/quote]

", "votes": null, "replies": [] }, { "id": 59534, "author_name": "martely", "author_url": "", "post_date": "12/04/2014 11:41:47", "content": "

Yes, there are very strong assumptions on the file being similar to the train.csv and test.csv provided for the competition: comma separated, with "ID", "click" and "hour" columns.

\n

All features are accepted, and they are all treated as categorical via feature hashing.

", "votes": null, "replies": [] }, { "id": 59538, "author_name": "herimanitra", "author_url": "", "post_date": "12/04/2014 12:04:58", "content": "

should be an option then as people like me :) may already have numerical features ready

\n

for training.

\n

Something like:

\n

pypy fast_solution_plus.py train --train train.csv -o train_model.gz  --train_as_is

\n

[quote=Yannick Martel;59534]

\n

All features are accepted, and they are all treated as categorical via feature hashing.

\n

[/quote]

", "votes": null, "replies": [] }, { "id": 59542, "author_name": "martely", "author_url": "", "post_date": "12/04/2014 13:09:02", "content": "

Yes, you are right, it would be a nice enhancement, to use with some new features - engineered features probably as the raw features are more categorical to me.

\n

We would need to have as argument to --train_as_is the list of features, as you probably want to mix:

\n

pypy fast_solution_plus.py train --train train.csv -o train_model.gz --train_as_is feature1,feature2

", "votes": null, "replies": [] }, { "id": 59554, "author_name": "manas6", "author_url": "", "post_date": "12/04/2014 17:37:32", "content": "

Does anyone know how I might be able to get this to work in terminal on a Mac with both Python 2.7.8 and Pypy installed? I've been trying but all I've been able to do is get the 'id' and 'click' columns in my submission file. (I'm fairly new to python, so any assistance would be greatly appreciated! Just trying to learn!)

\n

I've been noticing a lot of indentation errors come up in terminal. Could this warrant any changes that might help be create a submission file?

", "votes": null, "replies": [] }, { "id": 59595, "author_name": "dataying", "author_url": "", "post_date": "12/05/2014 04:07:11", "content": "

I am wondering why the values of self.n and self.z are not updated? 

\n

are they updated automatically after n and z are updated?

\n\n\n

[quote=tinrtgu;58054]

\n

Notice (11/18/2014): added the third version of the script fast_solution_v3.py

\n

Notice (11/20/2014): here is a post on how to get 0.3977417 on the public leaderboard

\n

-

\n

Dear fellow Kagglers, tin-ert-gu here.

\n

-

\n

This is a python implementation of adaptive learning rate L1 and L2 regularized logistic regression using hash-trick one-hot encoding. This algorithm is also used by Google for their online CTR prediction system.
All credits of this script goes to their hard work of research.

\n

-

\n

How to run?

\n

In your terminal, simply type

\n

python fast_solution.py

\n

The script is also python3 compatible, so you can use python3 as well

\n

python3 fast_solution.py

\n

However, since the code only depends on native modules, it is highly suggested that you run the script with pypy for a huge speed-up. To use the script with pypy on Ubuntu, first type

\n

sudo apt-get install pypy

\n

Then use the pypy interpreter to run the script

\n

pypy fast_solution.py

\n

-

\n

Changelog over the previous version of the script

\n
    \n
  1. Added L1 regularization
    2. \n
  2. Added L2 regularization
    3. \n
  3. Build in multiple epoch training
    4. \n
  4. Build in holdout set validation support
    5. \n
  5. Build in support for poly 2 feature interactions
    6. \n
  6. Re-factored code for better extendibility
    7. \n
  7. EVEN MORE COMMENTS!
    8. \n
\n

-

\n

Performance

\n

Training time for a single epoch, on a single Intel(R) Xeon(R) CPU E5-2630L v2 @ 2.40GHz:

\n\n

Memory usage:

\n

This mainly depends on the D parameter in the script.

\n\n

-

\n

The entire algorithm is disclosed, while using only out of the box modules provided by Python, so it should also be an easy sandbox code to build new ideas upon.

\n

Good luck and have fun!

\n

-

\n

@Abhishek:

\n

Can't submit right now, so I don't know the leader-board score, but I would say it should easily beat 0.40

\n

-

\n

EDIT 1

\n

add fast_solution_v2.py to fix a null reference bug under memory saving mode

\n

EDIT 2

\n

add fast_solution_v3.py

\n

A great thanks for Paweł's feedback, the following changes are made as a result

\n\n

By considering fchollet's experiment

\n\n

This should be the last time I modify this benchmark for this competition. Hope the actual data will get release soon, and happy predicting!

\n

[/quote]

", "votes": null, "replies": [] }, { "id": 59635, "author_name": "kolobok007", "author_url": "", "post_date": "12/05/2014 18:40:47", "content": "

[quote=Ying;59595]

\n

I am wondering why the values of self.n and self.z are not updated? 

\n

are they updated automatically after n and z are updated?

\n

[/quote]

\n

It's Python. So n and z are just references to self.n and self.z. They are not copies! So when you update n you actually update self.n.

", "votes": null, "replies": [] }, { "id": 59680, "author_name": "herimanitra", "author_url": "", "post_date": "12/06/2014 15:49:18", "content": "

If I remove this piece of code:

\n

# build x
x = [0] # 0 is the index of the bias term
for key in row:
value = row[key]

\n

# one-hot encode everything with hash trick
index = abs(hash(key + '_' + value)) % D
x.append(index)

\n

Should I expect that  x will be treated as is?

", "votes": null, "replies": [] }, { "id": 59705, "author_name": "pboswell0", "author_url": "", "post_date": "12/07/2014 01:53:24", "content": "

I keep getting the error shown in the image attached. Is there a particular 'train' and 'test' file I should be downloading? Or is there a special place I should save them to?

", "votes": null, "replies": [] }, { "id": 59709, "author_name": "ggglhf", "author_url": "", "post_date": "12/07/2014 03:03:07", "content": "

[quote=PBoswell;59705]

\n

I keep getting the error shown in the image attached. Is there a particular 'train' and 'test' file I should be downloading? Or is there a special place I should save them to?

\n

[/quote]

\n

Modify line 32 to 34 of the script to the correct file path.

", "votes": null, "replies": [] }, { "id": 59710, "author_name": "ggglhf", "author_url": "", "post_date": "12/07/2014 03:06:56", "content": "

[quote=Herimanitra;59680]

\n

If I remove this piece of code:

\n

# build x
x = [0] # 0 is the index of the bias term
for key in row:
value = row[key]

\n

# one-hot encode everything with hash trick
index = abs(hash(key + '_' + value)) % D
x.append(index)

\n

Should I expect that  x will be treated as is?

\n

[/quote]

\n

I think an exception will be raised.

", "votes": null, "replies": [] }, { "id": 59715, "author_name": "herimanitra", "author_url": "", "post_date": "12/07/2014 04:25:09", "content": "

seems I figure out ( the script run without problem, x seems to be treated as is):

\n

# turn hour really into hour, it was originally YYMMDDHH

\n

#date = row['hour'][0:6]
#row['hour'] = row['hour'][6:]

\n

# stderr.write("_%s_" % date)

\n

#if dayfilter != None and not date in dayfilter:
#continue

\n

#if dayfeature:
## extract date
#row['wd'] = str(int(date) % 7)
#row['wd_hour'] = "%s_%s" % (row['wd'], row['hour'])

\n

#if counters:
#d_ip = row['device_ip']
#d_id = row ["device_id"]
#try:
#device_ip_counter [d_ip] += 1
#device_id_counter [d_id] += 1
#except KeyError:
#device_ip_counter [d_ip] = 1
#device_id_counter [d_id] = 1
#row["ipc"] = str(min(device_ip_counter[d_ip], 8))
#row["idc"] = str(min(device_id_counter[d_id], 8))

\n

# build x
x = [0] # 0 is the index of the bias term
for key in row:
value = row[key]

\n

## one-hot encode everything with hash trick
#index = abs(hash(key + '_' + value)) % D
#x.append(index)

\n

Could you confirm?

", "votes": null, "replies": [] }, { "id": 59724, "author_name": "pboswell0", "author_url": "", "post_date": "12/07/2014 06:36:39", "content": "

can someone point me in the right direction for resources on some of these parameters (i.e. regularization, beta, D parameter).

\n

Also what is the verdict on learning rate. Is it truly the (s)lower the better?

", "votes": null, "replies": [] }, { "id": 59738, "author_name": "jasonwang592", "author_url": "", "post_date": "12/07/2014 17:15:52", "content": "

This paper might help:

\n

L1 L2 Paper

\n

The beta parameter to my understanding doesn't need much tuning since it becomes nearly inconsequential after many samples. It's really just used for smoothing in the first few occurrences of a value. As for D, generally larger is better since you get fewer collisions but a smaller D also serves as dimension reduction so that will need some tinkering around with.

\n

As for the learning rate, like you're implying, a lower rate slows convergence but can prevent oscillation of the weight vector when you choose a larger learning rate. Again, probably another parameter to tinker. From what I've seen in literature, 0.1 is a good place to start.

\n

Cheers!

", "votes": null, "replies": [] }, { "id": 59753, "author_name": "sneakyfox", "author_url": "", "post_date": "12/08/2014 00:57:23", "content": "

@tinrtgu - Thank you for writing this beautiful piece of code - very clean & and is easy to read.

\n

I have been using the code, and tweaking the hyper-parameters manually.  I am curious if there is a method to going about doing this - does one select the hyper-parameters by brute force, or is there some intuition/methodology that goes behind selecting the optimal parameters?  

\n

If there are any papers on this, I would love to read them!

\n

--update:  I didn't see the post above this one.

", "votes": null, "replies": [] }, { "id": 59760, "author_name": "ggglhf", "author_url": "", "post_date": "12/08/2014 04:35:37", "content": "

[quote=sneakyfox;59753]

\n

@tinrtgu - Thank you for writing this beautiful piece of code - very clean & and is easy to read.

\n

I have been using the code, and tweaking the hyper-parameters manually.  I am curious if there is a method to going about doing this - does one select the hyper-parameters by brute force, or is there some intuition/methodology that goes behind selecting the optimal parameters?  

\n

If there are any papers on this, I would love to read them!

\n

--update:  I didn't see the post above this one.

\n

[/quote]

\n

Yeah, the post above you is pretty nice.

\n

You don't really need to tune \\beta, just leave it alone at 1 is good enough. From my experience, so please take it with a grain of salt, without feature interactions the most important parameter is the learning rate, and with interactions L1 and L2 are important.

\n

As how to find the optimal parameters, I just use the old fashion grid search with validation.

", "votes": null, "replies": [] }, { "id": 59771, "author_name": "pboswell0", "author_url": "", "post_date": "12/08/2014 09:51:11", "content": "

[quote=Yannick Martel;59144]

\n

Hello all,

\n

I have worked on the fast_solution great piece of code and added a few features:

\n

- parameter control from command line

\n

- dropout: where you randomly drop features out of consideration - prevents overfitting and features adaptation

\n

- inclusion of the # of samples seen per device_ip / device_id as a feature

\n

- inclusion of weedkay as a feature

\n

I am down to roughly 0.3948 on the LB with a lot of tuning, by running on all test samples - with a few hours of processing using pypy on an i7, using a few GB's of RAM. LB is very close to my own cross-validation (run on all days except 30, test on 30).

\n

The code probably would benefit from some optimization. I have written the first version in Julia - it was faster than C-Python, but 2-3 times slower than pypy, and I quickly reverted to Python.

\n

I would be glad of any constructive feedback.

\n

Yannick

\n

[/quote]

\n\n

Using Yannick's script, I get the error "too few arguments". Any ideas why?

", "votes": null, "replies": [] }, { "id": 59782, "author_name": "martely", "author_url": "", "post_date": "12/08/2014 13:04:05", "content": "

You need to provide one argument "train" or "predict" to determine what the script will do, plus some more in the forme -o model.gz for the arguments.

\n

For instance:

\n

> pypy fast_solution_plus.py train --train train.csv.gz -o model.gz

\n

> pypy fast_solution_plus.py predict --test test.csv.gz -i model.gz -p predict.gz

\n

You can also do

\n

> pypy fast_solution_plus.py --help

\n

to get more information.

", "votes": null, "replies": [] }, { "id": 59800, "author_name": "pboswell0", "author_url": "", "post_date": "12/08/2014 17:06:07", "content": "

Yannick,

\n

Thank you for the quick response. A couple more questions, if you don't mind. Is this model training on 9 days and validating on 10th? I noticed the fast_solution_v3.py script includes a D parameter; do you have an equivalent or what is the default? Also, what does the 'holdout' feature do (in the other script it is defaulted at 0?

\n\n

Thanks!

", "votes": null, "replies": [] }, { "id": 59801, "author_name": "pboswell0", "author_url": "", "post_date": "12/08/2014 17:10:50", "content": "

On another note, I keep getting the following error when running Yannick's 'fast_solution_plus.py' script (attached photo)

", "votes": null, "replies": [] }, { "id": 59802, "author_name": "martely", "author_url": "", "post_date": "12/08/2014 17:15:34", "content": "

@PBoswell:

\n

On your first question, the logic is a bit different from fast_solution_v3.py: I validate on 1 per 100 (with holdout = 100), and not on a separate day. I recommend to train on n days and validate on n+1, but you have to script if outside of the python executable.

\n

The D parameters is indirect: you can specify --bits 23 on the command line to get D=2**23.

\n

On the memory problem, it seems that gzip is very memory hungry when you save the model. Maybe try first on a .model file, not a .model.gz. I have 16GB on my PC, so it works, but it might not on a smaller machine.

\n

Any idea to get an efficient serialisation plus compression of Python data structure on disk is welcome! Plucker plus gzip is reasonably simple, but not so fast nor so efficient on disk.

", "votes": null, "replies": [] }, { "id": 59806, "author_name": "pboswell0", "author_url": "", "post_date": "12/08/2014 17:53:50", "content": "

You are awesome, Yannick!

", "votes": null, "replies": [] }, { "id": 59808, "author_name": "pboswell0", "author_url": "", "post_date": "12/08/2014 18:09:09", "content": "

More of a high level discussion I want to start here:

\n\n

Has anyone been able to beat .397 with these scripts? That is, is the underlying modeling used here limited? Or do you really believe with parameter tuning you could reach #1 on LB?

\n\n

From my experience, increasing L1 has consistently improved my LB score, but with diminishing returns. Changing learning rate either way does not seem to help, but maybe I need to change learning rate ALONG WITH something else?

", "votes": null, "replies": [] }, { "id": 59809, "author_name": "adjgiulio", "author_url": "", "post_date": "12/08/2014 18:14:15", "content": "

[quote=PBoswell;59808]

\n

Has anyone been able to beat .397 with these scripts? 

\n

[/quote]

\n

The default parameters give .395 if you train on the whole data.

", "votes": null, "replies": [] }, { "id": 59811, "author_name": "pboswell0", "author_url": "", "post_date": "12/08/2014 18:23:35", "content": "

you mean train on all data by making holdafter = None/10?

", "votes": null, "replies": [] }, { "id": 59812, "author_name": "inversion", "author_url": "", "post_date": "12/08/2014 19:15:25", "content": "

[quote=PBoswell;59808]

\n

Has anyone been able to beat .397 with these scripts? 

\n

[/quote]

\n

Using this model, I've reached a 0.3915 LB score, and still making fairly steady progress.

", "votes": null, "replies": [] }, { "id": 59813, "author_name": "pboswell0", "author_url": "", "post_date": "12/08/2014 19:27:27", "content": "

Well I don't know what you guys are doing, because the default parameters yield me 0.3976439 when not changing any parameters.

", "votes": null, "replies": [] }, { "id": 59821, "author_name": "pboswell0", "author_url": "", "post_date": "12/08/2014 20:18:00", "content": "

How do you train on the whole data?

", "votes": null, "replies": [] }, { "id": 59826, "author_name": "clustifier", "author_url": "", "post_date": "12/08/2014 20:30:38", "content": "

[quote=inversion;59812]

\n

[quote=PBoswell;59808]

\n

Has anyone been able to beat .397 with these scripts? 

\n

[/quote]

\n

Using this model, I've reached a 0.3915 LB score, and still making fairly steady progress.

\n

[/quote]

\n

Without any feature engineering? 

", "votes": null, "replies": [] }, { "id": 59828, "author_name": "pboswell0", "author_url": "", "post_date": "12/08/2014 20:36:22", "content": "

[quote=clustifier;59826]

\n

[quote=inversion;59812]

\n

[quote=PBoswell;59808]

\n

Has anyone been able to beat .397 with these scripts? 

\n

[/quote]

\n

Using this model, I've reached a 0.3915 LB score, and still making fairly steady progress.

\n

[/quote]

\n

Without any feature engineering? 

\n

[/quote]

\n

Also just to clarify, are we talking about the original 'fast_solution_v3' script, or Yannick's 'fast_solution_plus'?

", "votes": null, "replies": [] }, { "id": 59829, "author_name": "inversion", "author_url": "", "post_date": "12/08/2014 20:44:47", "content": "

[quote=PBoswell;59828]

\n

[quote=clustifier;59826]

\n

[quote=inversion;59812]

\n

[/quote]

\n

Using this model, I've reached a 0.3915 LB score, and still making fairly steady progress.

\n

Without any feature engineering? 

\n

[/quote]

\n

Also just to clarify, are we talking about the original 'fast_solution_v3' script, or Yannick's 'fast_solution_plus'?

\n

[/quote]

\n

I'm pretty much using v1 (I had to modify it to make it work with numba), I'm not doing anything special with feature engineering other than brute-force interactions.

", "votes": null, "replies": [] }, { "id": 59830, "author_name": "pboswell0", "author_url": "", "post_date": "12/08/2014 20:45:43", "content": "

so you think v1 is better than v3?

", "votes": null, "replies": [] }, { "id": 59831, "author_name": "inversion", "author_url": "", "post_date": "12/08/2014 20:49:43", "content": "

[quote=PBoswell;59830]

\n

so you think v1 is better than v3?

\n

[/quote]

\n

I don't know what to think. Probably best to try different methods. There are lots of ways to approach the problem.

", "votes": null, "replies": [] }, { "id": 59851, "author_name": "ngutten", "author_url": "", "post_date": "12/09/2014 00:49:41", "content": "

With no feature engineering I'm at 0.3931 LB with Yannick's version of the script, but the runs take about 6 hours and maybe 4-6GB of RAM. Looks like I've still got some parameter optimization to do.

", "votes": null, "replies": [] }, { "id": 59858, "author_name": "pboswell0", "author_url": "", "post_date": "12/09/2014 02:08:57", "content": "

[quote=Nicholas Guttenberg;59851]

\n

With no feature engineering I'm at 0.3931 LB with Yannick's version of the script, but the runs take about 6 hours and maybe 4-6GB of RAM. Looks like I've still got some parameter optimization to do.

\n

[/quote]

\n\n

do you mind sharing what droupout you used?

", "votes": null, "replies": [] }, { "id": 59859, "author_name": "ngutten", "author_url": "", "post_date": "12/09/2014 02:30:45", "content": "

I haven't actually optimized the dropout yet, so I'm just using whatever the default is.

", "votes": null, "replies": [] }, { "id": 59861, "author_name": "pboswell0", "author_url": "", "post_date": "12/09/2014 04:33:19", "content": "

I don't know what is going on, but I continuously get a 'MemoryError' when running 'fast_solution_plus.py' when trying to save the output model.

\n

I am running on 8GB of RAM. I monitored the usage during process and never went above 50% memory usage.

\n

Anyone have any ideas?

", "votes": null, "replies": [] }, { "id": 59862, "author_name": "phanisrikanth", "author_url": "", "post_date": "12/09/2014 04:44:42", "content": "

[quote=PBoswell;59861]

\n

I don't know what is going on, but I continuously get a 'MemoryError' when running 'fast_solution_plus.py' when trying to save the output model.

\n

I am running on 8GB of RAM. I monitored the usage during process and never went above 50% memory usage.

\n

Anyone have any ideas?

\n

[/quote]

\n

As Yannick mentioned, one suggestion could be to save the model without gzip. I guess it should work.

", "votes": null, "replies": [] }, { "id": 59863, "author_name": "pboswell0", "author_url": "", "post_date": "12/09/2014 04:47:44", "content": "

[quote=binga;59862]

\n

[quote=PBoswell;59861]

\n

I don't know what is going on, but I continuously get a 'MemoryError' when running 'fast_solution_plus.py' when trying to save the output model.

\n

I am running on 8GB of RAM. I monitored the usage during process and never went above 50% memory usage.

\n

Anyone have any ideas?

\n

[/quote]

\n

As Yannick mentioned, one suggestion could be to save the model without gzip. I guess it should work.

\n

[/quote]

\n

I have tried both. Both yield the same error.

", "votes": null, "replies": [] }, { "id": 59864, "author_name": "phanisrikanth", "author_url": "", "post_date": "12/09/2014 05:00:09", "content": "

It just means that 8GB RAM isnt sufficient for the operation. You could keep an eye on system memory and check which chunk of code is consuming all the RAM

", "votes": null, "replies": [] }, { "id": 59865, "author_name": "pboswell0", "author_url": "", "post_date": "12/09/2014 05:11:48", "content": "

so it seems that despite the error, it is saving 'model.gz' to my directory. However, when I try to predict I get:

\n

'IOerror: CRC Check failed'

\n

What does that mean!?

", "votes": null, "replies": [] }, { "id": 59869, "author_name": "phanisrikanth", "author_url": "", "post_date": "12/09/2014 07:26:45", "content": "

It means that when you tried saving your model during the train operation, if you have encountered the memory error, even then the model.gz is saved to your disk but it is corrupted. And, that is why you encounter a CRC check failed error while unpacking the model.gz during predict operation.

", "votes": null, "replies": [] }, { "id": 59872, "author_name": "phanisrikanth", "author_url": "", "post_date": "12/09/2014 07:59:35", "content": "

Here's a turnaround. You could save the file without gzip-ping it.

\n

Modify write_learner function to - with open(model_save, "wb") as model_file:

\n

& modify the load_learner function to - with open(model_save, "rb") as model_file:

\n

It should work now!

", "votes": null, "replies": [] }, { "id": 59889, "author_name": "lukasdrapal", "author_url": "", "post_date": "12/09/2014 15:20:35", "content": "

@inversion I am trying to rewrite the script for numba. I am a bit confused how to treat _indices: how to modify it and call it from the predict function. Could You please give an advice?

", "votes": null, "replies": [] }, { "id": 59891, "author_name": "inversion", "author_url": "", "post_date": "12/09/2014 15:52:00", "content": "

[quote=Matfyzak;59889]

\n

@inversion I am trying to rewrite the script for numba. I am a bit confused how to treat _indices: how to modify it and call it from the predict function. Could You please give an advice?

\n

[/quote]

\n

Oh, right, this tripped me up to. I just ended up getting rid of it and pulling it into the data subroutine (see attached). And then, instead of

\n

for i in self._indices(x):

\n

I use

\n

for i in x:

\n\n

Make sense?

", "votes": null, "replies": [] }, { "id": 59899, "author_name": "sneakyfox", "author_url": "", "post_date": "12/09/2014 19:50:09", "content": "

I am pretty new to machine-learning, and I am curious if anybody has a method for developing a robust validation framework to guard against over-fitting.

\n

In the original code sample given by tinrtgu, there is a field that allows us to validate on one of the days, or specify a holdout.  There is also a field for "epochs" or the number of passes through the dataset.

\n

My concerns - why are epochs useful, and how do we interpret it?  If I were to set it to be 10, and get 10 sets of log-losses from it, do I just take the average and have that to be my overall log-loss?

\n

Currently, the way I am approaching the problem is to shuffle the whole training dataset, and setting the holdout to be various numbers (usually 10 or 100).  My reasoning for shuffling the dataset is to get rid of any serial dependencies the entries may have had upon one another.  Is this a solid approach?

\n

We are getting very suspicious of the results that the validation set that kaggle is spitting out, because it is only on a sub-set.  We feel that we are over-fitting to that set specifically by making the submissions.

", "votes": null, "replies": [] }, { "id": 59907, "author_name": "liamhuber", "author_url": "", "post_date": "12/09/2014 21:07:26", "content": "

[quote=sneakyfox;59899]

\n

I am pretty new to machine-learning, and I am curious if anybody has a method for developing a robust validation framework to guard against over-fitting.[/quote]

\n

The most widely-used and broadly-applicable and all-around-awesome method is simply cross validation (CV)--test on something you didn't train on. Since you're "holding out" every nth sample or full days, you're already doing this :) 

\n

[quote=sneakyfox;59899]

\n

 why are epochs useful, and how do we interpret it?[/quote]

\n

Epochs are the number of times you pass through the same data; training then re-training on it. In an online learner, you typically use a small "learning rate", so that the information stored in a single data point isn't all jammed into your model at once. For instance, if you train on a data point, then immediately predict that exact same point, you won't get the true value. This is actually very desirable, since you don't want your model to simply reproduce the most recent point perfectly, but rather take the big picture into account.

\n

By using multiple epochs, you allow each data point to contribute to the model more than once. If you have too little data, this can be really useful. But be careful, because if you use too many epochs, you run the risk of over-fitting to your training data and may lose generalizability.

\n

Train with however many epochs you want, then use that trained model to predict on some test data. Use the number of epochs that gives you a good CV score.

\n

[quote=sneakyfox;59899]

\n

Currently, the way I am approaching the problem is to shuffle the whole training dataset, and setting the holdout to be various numbers (usually 10 or 100).  My reasoning for shuffling the dataset is to get rid of any serial dependencies the entries may have had upon one another.  Is this a solid approach?[/quote]

\n

Good question. I don't understand the subtleties of the shuffle/don't shuffle argument very well.

\n

[quote=sneakyfox;59899]

\n

We are getting very suspicious of the results that the validation set that kaggle is spitting out, because it is only on a sub-set.  We feel that we are over-fitting to that set specifically by making the submissions.[/quote]

\n

This one has a simple solution: ignore the leader board. It's fun, and it can, potentially, give you some information--but you should trust in your CV. Usually at the end of the competition you're allowed to mark multiple submissions for final evaluation, so usually you can choose your best one or two based on CV and still submit your best LB version as well :)

", "votes": null, "replies": [] }, { "id": 59909, "author_name": "sneakyfox", "author_url": "", "post_date": "12/09/2014 21:23:48", "content": "

Thanks for the informative answer @LiamHuber! Would give 2 upvotes if I could!

\n

I had some quick follow up questions - for cross-validation, I have read in some places that it still provides an optimistic estimate for the generalization error.  Should we be just weary of this fact when applying it?  

\n

Secondly, do you have and good references for online-learning model?  I would like to learn more about the topic.

\n

Another approach that we were thinking of is averaging a bunch of models together, in order to reduce the variance - if we take our best 5 models according to CV score, and do an unweighted average, it may yield a more representative log-loss.  Just a thought though, may be completely wrong.

", "votes": null, "replies": [] }, { "id": 59931, "author_name": "pboswell0", "author_url": "", "post_date": "12/10/2014 04:18:49", "content": "

So what is the major difference between 'fast_solution_v(1,2, or 3)' and Yannick's 'fast_solution_plus'?

\n\n

That is, using the same parameters between the two, why would I get different LB scores?

", "votes": null, "replies": [] }, { "id": 59936, "author_name": "cowabungadood", "author_url": "", "post_date": "12/10/2014 05:44:46", "content": "

I am having some trouble running fast_solution_v3.py on my terminal.  Here is the  error I receive:

\n\n

Traceback (most recent call last):
File "fast_solution_v3.py", line 260, in <module>
for t, date, ID, x, y in data(train, D): # data is a generator
File "fast_solution_v3.py", line 216, in data
for t, row in enumerate(DictReader(open(path))):
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/csv.py", line 107, in next
self.fieldnames
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/csv.py", line 90, in fieldnames
self._fieldnames = self.reader.next()
_csv.Error: line contains NULL byte

\n\n

Any help would be appreciated. Thanks.

", "votes": null, "replies": [] }, { "id": 59938, "author_name": "liamhuber", "author_url": "", "post_date": "12/10/2014 06:04:08", "content": "

[quote=sneakyfox;59909]

\n

Thanks for the informative answer @LiamHuber! Would give 2 upvotes if I could!

\n

I had some quick follow up questions - for cross-validation, I have read in some places that it still provides an optimistic estimate for the generalization error.  Should we be just weary of this fact when applying it?  

\n

Secondly, do you have and good references for online-learning model?

\n

Another approach that we were thinking of is averaging a bunch of models together...

\n

[/quote]

\n\n

Glad it was helpful!

\n

I think how optimistic your CV is will depend on what you're drawing it from. e.g. contrast CV by holding out every 100th step vs CV by holding out an entire day. In the former case, there is likely to be more similarity between the train and test data points than in the later case (perhaps because the train day is a weekday but test day is a weekend, or any other factor.) Since our goal is to predict a whole new day, I recommend CV by separate days. But even this isn't a panacea since the test set has many feature values totally missing in the training set (there's a really nice post somewhere showing a breakdown of that on the old data, but I can't find it.)

\n

I don't have any great advice for online learners. This is my first time exploring online learning as well. The script in this thread is obviously really helpful, and I've also been having moderate luck (CV ~0.41) with a fairly simple neural net.

\n

Your last idea is usually called bagging, and is an excellent idea!

", "votes": null, "replies": [] }, { "id": 59951, "author_name": "lukasdrapal", "author_url": "", "post_date": "12/10/2014 15:56:33", "content": "

@inversion: It makes sense - thanks a lot! I don't have now time to complete the script, but this part makes sense! ;)

", "votes": null, "replies": [] }, { "id": 60037, "author_name": "davis3", "author_url": "", "post_date": "12/11/2014 18:51:56", "content": "

Just wondering how people are getting .392 with Yannick's code? Are you using the default parameters and how many epochs did you use? We are stuck at 0.3947749 and just trying to get below .394. Thanks!

", "votes": null, "replies": [] }, { "id": 60038, "author_name": "runeoedegaard", "author_url": "", "post_date": "12/11/2014 18:53:20", "content": "

Hey guys,

\n

Thanks for posting these codes, great learning!

\n

One question, how are people able to get 0.392 with Yannick's code? If you are, are you changing the parameters or using the default settings, and how many epochs do you use? 

\n

We really just want to see if we can get below 0.394 using this code...

\n

Thanks!

", "votes": null, "replies": [] }, { "id": 60040, "author_name": "pboswell0", "author_url": "", "post_date": "12/11/2014 19:24:36", "content": "

[quote=Nicholas Guttenberg;59851]

\n

With no feature engineering I'm at 0.3931 LB with Yannick's version of the script, but the runs take about 6 hours and maybe 4-6GB of RAM. Looks like I've still got some parameter optimization to do.

\n

[/quote]

\n

how many epochs are you running?

", "votes": null, "replies": [] }, { "id": 60061, "author_name": "ngutten", "author_url": "", "post_date": "12/12/2014 00:28:56", "content": "

I'm not actually sure how much specific parameter information is kosher for me to share (at least, I'd like an opinion on that from the more senior Kagglers before giving too much detail, since someone was gently reminded of this for sharing their # of bits earlier in the thread).

", "votes": null, "replies": [] }, { "id": 60062, "author_name": "jfkingiii", "author_url": "", "post_date": "12/12/2014 00:36:15", "content": "

[quote=Nicholas Guttenberg;60061]

\n

I'm not actually sure how much specific parameter information is kosher for me to share (at least, I'd like an opinion on that from the more senior Kagglers before giving too much detail, since someone was gently reminded of this for sharing their # of bits earlier in the thread).

\n

[/quote]

\n

The rules allow you to share anything you want, but if you post the exact parameters that give a 0.3931 score you'll make some people mad. It's entirely up to you how much or how little you share. Remember there will be a robust discussion where a lot of people explain everything when the competition is over.

", "votes": null, "replies": [] }, { "id": 60144, "author_name": "dlsiii", "author_url": "", "post_date": "12/13/2014 13:07:08", "content": "

Hi Tinrtgu, thanks for posting this, it's very interesting and helpful!  

\n

I have a question about the hashing: if I'm reading the code correctly, all the features have their hashed values mapped into the same list of indices (x).  Does that mean that there is a risk of collision even between the values of different features?  It seems to me like there is (even with a fairly large D) although I'm not sure it's something I should be concerned about.

", "votes": null, "replies": [] }, { "id": 60145, "author_name": "ggglhf", "author_url": "", "post_date": "12/13/2014 13:19:45", "content": "

[quote=dlsiii;60144]

\n

Does that mean that there is a risk of collision even between the values of different features?

\n

[/quote]

\n

Yes, you are correct and I did it that way to ease the implementation. If you are really concerned you can maintain an index look-up dictionary so that there is 0 chance of collision, it should give you some improvements, although I feel it would only be marginal.

", "votes": null, "replies": [] }, { "id": 60148, "author_name": "dlsiii", "author_url": "", "post_date": "12/13/2014 15:37:51", "content": "

[quote=tinrtgu;60145]

\n

Yes, you are correct and I did it that way to ease the implementation. If you are really concerned you can maintain an index look-up dictionary so that there is 0 chance of collision, it should give you some improvements, although I feel it would only be marginal.

\n

[/quote]

\n

Thanks for the quick response, you're probably right, when I think about it seems like the chance of two values with reasonably high frequency colliding are pretty low for a large enough D.

", "votes": null, "replies": [] }, { "id": 60155, "author_name": "ngutten", "author_url": "", "post_date": "12/13/2014 18:32:34", "content": "

One way to test to see if collisions are making a difference would be to do two (or more, if you have time) sets of predictions where you modify the category names in different ways before passing them into the hash function. If the log-loss varies significantly, then something important might be colliding (e.g. a frequently occurring pair of highly predictive categorical labels are getting hashed to the same thing). 

", "votes": null, "replies": [] }, { "id": 60162, "author_name": "qbert64", "author_url": "", "post_date": "12/13/2014 20:34:18", "content": "

Sorry to bug people with what's probably a simple I/O issue, but I get this error when running fast_solution_v3:

\n\n

--------------------------------------------------------------------------
Error Traceback (most recent call last)
<ipython-input-16-9f047f2c965d> in <module>()
4 count = 0
5
----> 6 for t, date, ID, x, y in data(train, D): # data is a generator
7 # t: just a instance counter
8 # date: you know what this is<ipython-input-8-ed0fc799fe79> in data(path, D)
29 '''
30
---> 31 for t, row in enumerate(DictReader(open(path))):
32 # process id
33 ID = row['id']/Users/David/anaconda/lib/python2.7/csv.pyc in next(self)
105 if self.line_num == 0:
106 # Used only for its side effect.
--> 107 self.fieldnames
108 row = self.reader.next()
109 self.line_num = self.reader.line_num/Users/David/anaconda/lib/python2.7/csv.pyc in fieldnames(self)
88 if self._fieldnames is None:
89 try:
---> 90 self._fieldnames = self.reader.next()
91 except StopIteration:
92 pass

\n\n

Error: new-line character seen in unquoted field - do you need to open the file in universal-newline mode?

\n\n

Does anyone know what the solution is?

\n\n

Thanks in advance...

", "votes": null, "replies": [] }, { "id": 60175, "author_name": "pavitrakumar78", "author_url": "", "post_date": "12/14/2014 05:44:17", "content": "

Hi,

\n

I've been playing around with the epoch values and I've noticed at after a certain number of iteration the validation log-loss seems to increase - this is because of the model is overfitting the training data right? and one possible way to avoid that is to increase regularization?

\n

(Sorry if its a noob question, I'm still a beginner)

", "votes": null, "replies": [] }, { "id": 60180, "author_name": "ggglhf", "author_url": "", "post_date": "12/14/2014 07:34:46", "content": "

[quote=Pavitrakumar;60175]

\n

Hi,

\n

I've been playing around with the epoch values and I've noticed at after a certain number of iteration the validation log-loss seems to increase - this is because of the model is overfitting the training data right? and one possible way to avoid that is to increase regularization?

\n

(Sorry if its a noob question, I'm still a beginner)

\n

[/quote]

\n

The question is actually great, and I think it is one of the key to this competition - how to do robust validation and keep overfitting in check.

\n

As to your question, yep, it means that the model is overfitting to the training set, increasing L1 and/or L2 regularization is one way to deal with it, decreasing the learning rate will also have some positive effect, another way is to not use so much epochs to begin with.

", "votes": null, "replies": [] }, { "id": 60185, "author_name": "pavitrakumar78", "author_url": "", "post_date": "12/14/2014 11:44:51", "content": "

[quote=tinrtgu;60180]

\n

[quote=Pavitrakumar;60175]

\n

Hi,

\n

I've been playing around with the epoch values and I've noticed at after a certain number of iteration the validation log-loss seems to increase - this is because of the model is overfitting the training data right? and one possible way to avoid that is to increase regularization?

\n

(Sorry if its a noob question, I'm still a beginner)

\n

[/quote]

\n

The question is actually great, and I think it is one of the key to this competition - how to do robust validation and keep overfitting in check.

\n

As to your question, yep, it means that the model is overfitting to the training set, increasing L1 and/or L2 regularization is one way to deal with it, decreasing the learning rate will also have some positive effect, another way is to not use so much epochs to begin with.

\n

[/quote]

\n

Thank you for the reply, tinrtgu.

\n

As you said, I am getting better validation log-loss after tuning my L1,L2 and epoch parameters, but I am still not able to reach my existing LB score obtained using VW.

\n

I am not able to get a validation log-loss lesser than 0.398x - Can you please tell me what part am I missing here? 

", "votes": null, "replies": [] }, { "id": 60190, "author_name": "dopelearner", "author_url": "", "post_date": "12/14/2014 15:59:34", "content": "

Hi tinrtgu,

\n

I can't seem to download the fast_solution files that you attached. Can you please take a look at it?

", "votes": null, "replies": [] }, { "id": 60197, "author_name": "pavitrakumar78", "author_url": "", "post_date": "12/14/2014 18:13:49", "content": "

[quote=Saurav Gupta;60190]

\n

Hi tinrtgu,

\n

I can't seem to download the fast_solution files that you attached. Can you please take a look at it?

\n

[/quote]

\n

I can download it properly, what is the problem?

", "votes": null, "replies": [] }, { "id": 60199, "author_name": "dopelearner", "author_url": "", "post_date": "12/14/2014 19:29:09", "content": "

I don't know, it says that the server can't be found because DNS lookup failed. Can you please send it to me at sauravguptaaa@gmail.com ?

\n

Thanks!

\n

Edit: Fixed it. No need to email. Using Google's DNS servers now :)

", "votes": null, "replies": [] }, { "id": 60231, "author_name": "xinliu", "author_url": "", "post_date": "12/15/2014 13:24:48", "content": "

[quote=dlsiii;60144]

\n

Hi Tinrtgu, thanks for posting this, it's very interesting and helpful!  

\n

I have a question about the hashing: if I'm reading the code correctly, all the features have their hashed values mapped into the same list of indices (x).  Does that mean that there is a risk of collision even between the values of different features?  It seems to me like there is (even with a fairly large D) although I'm not sure it's something I should be concerned about.

\n

[/quote]

\n

its hash the key_value combination. e.g. C1_123456, SITE_ID_123456

\n

ln240:  index = abs(hash(key + '_' + value)) % D

", "votes": null, "replies": [] }, { "id": 60316, "author_name": "alkhwarizmi", "author_url": "", "post_date": "12/16/2014 03:07:16", "content": "

[quote=tinrtgu;58054]

\n

Notice (11/18/2014): added the third version of the script fast_solution_v3.py

\n

Notice (11/20/2014): here is a post on how to get 0.3977417 on the public leaderboard

\n

-

\n

Dear fellow Kagglers, tin-ert-gu here.

\n

-

\n

This is a python implementation of adaptive learning rate L1 and L2 regularized logistic regression using hash-trick one-hot encoding. This algorithm is also used by Google for their online CTR prediction system.
All credits of this script goes to their hard work of research.

\n

-

\n

How to run?

\n

In your terminal, simply type

\n

python fast_solution.py

\n

The script is also python3 compatible, so you can use python3 as well

\n

python3 fast_solution.py

\n

However, since the code only depends on native modules, it is highly suggested that you run the script with pypy for a huge speed-up. To use the script with pypy on Ubuntu, first type

\n

sudo apt-get install pypy

\n

Then use the pypy interpreter to run the script

\n

pypy fast_solution.py

\n

-

\n

Changelog over the previous version of the script

\n
    \n
  1. Added L1 regularization
    2. \n
  2. Added L2 regularization
    3. \n
  3. Build in multiple epoch training
    4. \n
  4. Build in holdout set validation support
    5. \n
  5. Build in support for poly 2 feature interactions
    6. \n
  6. Re-factored code for better extendibility
    7. \n
  7. EVEN MORE COMMENTS!
    8. \n
\n

-

\n

Performance

\n

Training time for a single epoch, on a single Intel(R) Xeon(R) CPU E5-2630L v2 @ 2.40GHz:

\n\n

Memory usage:

\n

This mainly depends on the D parameter in the script.

\n\n

-

\n

The entire algorithm is disclosed, while using only out of the box modules provided by Python, so it should also be an easy sandbox code to build new ideas upon.

\n

Good luck and have fun!

\n

-

\n

@Abhishek:

\n

Can't submit right now, so I don't know the leader-board score, but I would say it should easily beat 0.40

\n

-

\n

EDIT 1

\n

add fast_solution_v2.py to fix a null reference bug under memory saving mode

\n

EDIT 2

\n

add fast_solution_v3.py

\n

A great thanks for Paweł's feedback, the following changes are made as a result

\n\n

By considering fchollet's experiment

\n\n

This should be the last time I modify this benchmark for this competition. Hope the actual data will get release soon, and happy predicting!

\n

[/quote]

\n\n

Thanks for the code. Every time I run it, regardless of the parameter values, I get a prediction of 0.5 for every record in the test set. Am I missing something? Using fast_solution_v3.py.

", "votes": null, "replies": [] }, { "id": 60317, "author_name": "herimanitra", "author_url": "", "post_date": "12/16/2014 03:10:27", "content": "

@AlKhwarizmi

\n

you should read previous page:

\n

https://www.kaggle.com/c/avazu-ctr-prediction/forums/t/10927/beat-the-benchmark-with-less-than-1mb-of-memory?page=2

", "votes": null, "replies": [] }, { "id": 60318, "author_name": "herimanitra", "author_url": "", "post_date": "12/16/2014 03:37:08", "content": "

I got the following error for some of my datasets:

\n

Traceback (most recent call last):
File "app_main.py", line 72, in run_toplevel
File "fast_solution_v3.py", line 253, in <module>
for t, date, ID, x, y in data(train, D): # data is a generator
File "fast_solution_v3.py", line 222, in data
date = int(row['hour'][4:6])-20
ValueError: invalid literal for int() with base 10: ''

\n

Any hints to solve it?

\n

Thks

\n

Edit: there are some NA values in my 'hour'

\n

will fix it :)

", "votes": null, "replies": [] }, { "id": 60460, "author_name": "phanisrikanth", "author_url": "", "post_date": "12/17/2014 10:25:56", "content": "

The attached graph tells a lot about how people have used your code @tinrtgu !

\n

2 humps at 0.398 and 0.395!\"Avazu

", "votes": null, "replies": [] }, { "id": 61621, "author_name": "sushize", "author_url": "", "post_date": "12/21/2014 16:13:48", "content": "

[quote=tinrtgu;59441]

\n

[quote=TanoPereira;59372]

\n

I've been able to go down to 0.376 in validation with interactions, but LB=0.41 ...

\n

[/quote]

\n

Just a hint, overfitting is more severe with interactions so it would be a good idea to increase regularizations.

\n

[/quote]

\n

Dear friends,

\n

I have questions about CV score and LB score for the interactions-version code. I use day 21-28 for training, and day 29-30 for validation. For the no-interaction version code, it works fine for me(i.e., in most case, improvement in CV also results in improvement in LB score).

\n

But for the interaction version code, it seems to me that such 2-day validation approach totally corrupts. To be specific, with interactions, my 2-day validation score is about 0.387, while the LB score is 0.399! (In comparison, my best 2-day validation score of no-interaction code is about 0.391, and the corresponding LB score is 0.394).

\n

Note that,  in my interaction-version code for this 0.387 validation result, I already follows the suggestion of tinrtgu and significantly increase the regularizations to (L1=40,L2=40), which is about10 times larger than the regularizations used in my best-results of the no-interaction version.

\n

Do I miss anything here, or any explanation or hints?

\n

Thanks in advance, and have a great day!

\n

Best wishes,

\n

Shize

", "votes": null, "replies": [] }, { "id": 61642, "author_name": "pboswell0", "author_url": "", "post_date": "12/22/2014 00:33:25", "content": "

1) Would somebody be kind enough to explain/comment how lines 65-98 actually work? For example, how does "self.w = [0.] * D" work? Doesn't multiplying by 0 just make it 0? 

\n

2) Is anyone touching the bounded sigmoid function at line 83?

", "votes": null, "replies": [] }, { "id": 61646, "author_name": "triskelion", "author_url": "", "post_date": "12/22/2014 04:28:11", "content": "

[quote=PBoswell;61642]

\n

For example, how does "self.w = [0.] * D" work? Doesn't multiplying by 0 just make it 0? 

\n

[/quote]

\n

[0] * 5 in python makes [0,0,0,0,0]

", "votes": null, "replies": [] }, { "id": 61700, "author_name": "inspector", "author_url": "", "post_date": "12/22/2014 21:05:47", "content": "

[quote=tinrtgu;58054]

\n

Notice (11/18/2014): added the third version of the script fast_solution_v3.py

\n

Notice (11/20/2014): here is a post on how to get 0.3977417 on the public leaderboard

\n

-

\n

Dear fellow Kagglers, tin-ert-gu here.

\n

-

\n

This is a python implementation of adaptive learning rate L1 and L2 regularized logistic regression using hash-trick one-hot encoding. This algorithm is also used by Google for their online CTR prediction system.
All credits of this script goes to their hard work of research.

\n

[/quote]

\n\n

Hey Tinrtgu -

\n

Curious for the reason why the absolute value of the hash value is taken before the mod operation? Does this help with collisions?

\n

Thanks!

", "votes": null, "replies": [] }, { "id": 61708, "author_name": "ivanlobov", "author_url": "", "post_date": "12/22/2014 23:06:25", "content": "

[quote=Inspector;61700]

\n

Hey Tinrtgu -

\n

Curious for the reason why the absolute value of the hash value is taken before the mod operation? Does this help with collisions?

\n

Thanks!

\n

[/quote]

\n

Absolute value is taken, because hash values are later used to reference to different weights in w[i] where i is a hash value. And a negative value cannot be a proper list index in Python.

\n

Abs doesn't help collisions, it makes it twice worse (twice less possible values for hash to take), but these are the downsides of super fast hash trick.

", "votes": null, "replies": [] }, { "id": 61715, "author_name": "inspector", "author_url": "", "post_date": "12/23/2014 00:40:10", "content": "

[quote=Ivan Lobov;61708]

\n

[quote=Inspector;61700]

\n

Hey Tinrtgu -

\n

Curious for the reason why the absolute value of the hash value is taken before the mod operation? Does this help with collisions?

\n

Thanks!

\n

[/quote]

\n

Absolute value is taken, because hash values are later used to reference to different weights in w[i] where i is a hash value. And a negative value cannot be a proper list index in Python.

\n

Abs doesn't help collisions, it makes it twice worse (twice less possible values for hash to take), but these are the downsides of super fast hash trick.

\n

[/quote]

\n

If you are using the hashed value (after %D) as input into a one-hot encoding, it would make sense to not take the abs() first - correct?

", "votes": null, "replies": [] }, { "id": 61717, "author_name": "anujamkelkar", "author_url": "", "post_date": "12/23/2014 00:49:17", "content": "

I don't know why, I end up getting equally probable for all ads, with this solution. 0.5 for all. Could someone kindly point out what I am doing wrong. I realize this might be an extremely stupid question to ask. 

", "votes": null, "replies": [] }, { "id": 61718, "author_name": "sushize", "author_url": "", "post_date": "12/23/2014 00:58:10", "content": "

[quote=Dee Dee;61717]

\n

I don't know why, I end up getting equally probable for all ads, with this solution. 0.5 for all. Could someone kindly point out what I am doing wrong. I realize this might be an extremely stupid question to ask. 

\n

[/quote]

\n

Hi Dee,

\n

I guess you didn't change the parameter holdafter=9 to holdafter=29? There are different versions of the code, and for v3, if you use holdafter=9, it simply means that no data were used for training (i.e., all data are used for validation), and this is why you get all 0.5 prediction.

\n

If you set the parameter holdafter=29 and D=2^24, you would get LB score about 0.376.

\n\n

Hope this might be helpful for you.

\n

Good night!

\n

Best wishes,

\n

Shize

", "votes": null, "replies": [] }, { "id": 61720, "author_name": "ggglhf", "author_url": "", "post_date": "12/23/2014 01:11:19", "content": "

[quote=Inspector;61715]

\n

If you are using the hashed value (after %D) as input into a one-hot encoding, it would make sense to not take the abs() first - correct?

\n

[/quote]

\n

Yeah, feel free to remove it. I added that abs() without giving it a thought, thanks for pointing it out. 

", "votes": null, "replies": [] }, { "id": 61722, "author_name": "inspector", "author_url": "", "post_date": "12/23/2014 01:22:55", "content": "

[quote=tinrtgu;61720]

\n

[quote=Inspector;61715]

\n

If you are using the hashed value (after %D) as input into a one-hot encoding, it would make sense to not take the abs() first - correct?

\n

[/quote]

\n

Yeah, feel free to remove it. I added that abs() without giving it a thought, thanks for pointing it out. 

\n

[/quote]

\n\n

OK, so it is not actually a required part of the program? I didn't see where it was actually needed. Cant thank you enough for sharing this - super cool

", "votes": null, "replies": [] }, { "id": 61774, "author_name": "pboswell0", "author_url": "", "post_date": "12/23/2014 19:52:50", "content": "

In version 3 of tngrtu's script, does setting holdafter to, say, 28 mean validation on every day after 28 or just the next day? If it is the former, how would I specify a non-consecutive series of days?

", "votes": null, "replies": [] }, { "id": 61777, "author_name": "sushize", "author_url": "", "post_date": "12/23/2014 20:19:13", "content": "

[quote=PBoswell;61774]

\n

In version 3 of tngrtu's script, does setting holdafter to, say, 28 mean validation on every day after 28 or just the next day? If it is the former, how would I specify a non-consecutive series of days?

\n

[/quote]

\n\n

Hi PBoswell,

\n\n

In v3 code, setting holdafter=28 means validation on everyday after 28 (i.e., 29 and 30). 

\n\n

To specify a non-consecutive series of days for validation, a simple and intuitive way is that you can change the validation condition line in the code file from " if (holdafter and date > holdafter) or (holdout and t % holdout == 0):" to any conditions you want, such as " if (holdafter and (date ==30 or date==24 or date==26)) or (holdout and t % holdout == 0):". 

\n\n

But still one issue associated with the validation approach specified by " if (holdafter and (date ==30 or date==24 or date==26)) or (holdout and t % holdout == 0):" is that the validation loss is not computed after that the model training  is finished. To be specific, the model start training on data day 21 -23 and compute logloss on day 24, and then continue model training on day 25, and then compute logloss on day 26, and then continue training on day 27-29, and then compute logloss on day 30. It is easy to see that such validation loss is not accurate enough to evaluate the trained model's efficiency (for accurate validation loss computation, ideally, it should first train the model on day 21-23, day 25, day 27-29 data, and then start to compute the validation loss on day 24, 26 and 30). Therefore, for more accurate validation loss, I think you could use one small script to output the validation data into a separate csv file, and then you could compute the validation logloss after the whole training procedure finished. (Currently I am still using day 29 and day 30 for validation, so I didn't use such small script for writing separate validation set. But I believe it is easy to write such small script if you want.)

\n\n

Hope this might be helpful for you.

\n\n

I am also interested to know if anybody else has a much easier way to do nonconsecutive-series of days for validation.

\n\n

Thanks, and have a good day!

\n\n

Best wishes,

\n

Shize

", "votes": null, "replies": [] }, { "id": 62122, "author_name": "jdxyw2004", "author_url": "", "post_date": "12/31/2014 07:12:18", "content": "

Thanks for the sharing. 

\n\n

I have a question about the bias term. Generally, the bias term should represent the average score of a model. But I found the bias generated by it is near zero. Anyone know why? 

", "votes": null, "replies": [] }, { "id": 62128, "author_name": "martely", "author_url": "", "post_date": "12/31/2014 10:30:53", "content": "

[quote=jdxyw;62122]

\n

Thanks for the sharing. 

\n

I have a question about the bias term. Generally, the bias term should represent the average score of a model. But I found the bias generated by it is near zero. Anyone know why? 

\n

[/quote]

\n

That's an excellent question, and maybe a key to improving the algorithm. I am wondering if we don't get an over-weight on some features which might block the bias term and maybe some others, and thus give an under-performing logistic regression.

\n

Any other insight?

", "votes": null, "replies": [] }, { "id": 62195, "author_name": "jialunhe", "author_url": "", "post_date": "01/02/2015 00:09:50", "content": "

#split training data into several days

\n

from datetime import datetime

\n

#++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
#main program starts here
start = datetime.now()

\n

train = 'data/train.csv' # path to training file
target_date = '30'
save_file = 'data/date' + target_date + '.csv'
str_label = 'id,click,hour,C1,banner_pos,site_id,site_domain,site_category,app_id,app_domain,app_category,device_id,device_ip,device_model,device_type,device_conn_type,C14,C15,C16,C17,C18,C19,C20,C21'

\n

with open(save_file, 'w') as outfile:
outfile.write('%s\\n' % str_label)

f=open(train)
f.readline()
for line in f:
xx=line.split(',')
date = xx[2][4:6]
if date == target_date :
outfile.write('%s\\n' % line)

\n

f.close()

print('saved: %s, etime: %s' % (save_file, str(datetime.now() - start)))

", "votes": null, "replies": [] }, { "id": 62203, "author_name": "herrahuu", "author_url": "", "post_date": "01/02/2015 06:28:09", "content": "

[quote=Yannick Martel;62128]

\n

That's an excellent question, and maybe a key to improving the algorithm. I am wondering if we don't get an over-weight on some features which might block the bias term and maybe some others, and thus give an under-performing logistic regression.

\n

Any other insight?

\n

[/quote]

\n

I think the reason is that the model, as it stands, is basically unidentified. To illustrate what I mean by this, let's say we have just one binary attribute and we will do the one-hot encoding. So, the linear function of the model can be written as: bias + (param1 or param2). Now the problem is that if {bias=0, param1=1, param2=2} is the "correct answer", then so is {bias=1, param1=0, param2=1} and {bias=10, param1=-9, param2=-8} etc. Ie. one of the parameters can be set to arbitrarily value.

\n

To fix this, the bias term or one of the other parameters should be set to value 0. However, due to regularization I don't think it would make a big difference though.

", "votes": null, "replies": [] }, { "id": 62409, "author_name": "justwajdi", "author_url": "", "post_date": "01/07/2015 17:03:36", "content": "

Thanks for sharing this but why did I get a "clean" Submission with 0.5 all over the ads list?

", "votes": null, "replies": [] }, { "id": 62425, "author_name": "josefernandozazo", "author_url": "", "post_date": "01/07/2015 21:34:08", "content": "

[quote=Farhani Wajdi;62409]

\n

Thanks for sharing this but why did I get a "clean" Submission with 0.5 all over the ads list?

\n

[/quote]

\n\n

This question has been answered before in the thread. You have to specify the train set by setting the holdafter variable to a correct value.

\n\n

For instance holdafter=28 means training over all the days until the 28th and apply CV over the data corresponding to the 29 and 30 day.

", "votes": null, "replies": [] }, { "id": 62447, "author_name": "justwajdi", "author_url": "", "post_date": "01/08/2015 08:27:20", "content": "

[quote=josefernando.zazo;62425]

\n

This question has been answered before in the thread. You have to specify the train set by setting the holdafter variable to a correct value.

\n

For instance holdafter=28 means training over all the days until the 28th and apply CV over the data corresponding to the 29 and 30 day.

\n

[/quote]

\n

That's look fine, Thanks

", "votes": null, "replies": [] }, { "id": 62454, "author_name": "clustifier", "author_url": "", "post_date": "01/08/2015 14:01:11", "content": "

[quote=fchollet;58089]

\n

Awesome! Thank you for sharing this. The method is pretty much state of the art, and the code quality is excellent! Huge thumb up.

\n

If you guys haven't already, be sure to check out the Google paper that describes this method. Hint: it contains useful info on how to improve the algo :) 

\n

http://www.eecs.tufts.edu/~dsculley/papers/ad-click-prediction.pdf

\n

[/quote]

\n

@fchollet, can you please hint further on what part of the paper you were referring to when saying "it contains useful info on how to improve the algo". I've read it for the second time today but I still not sure what you were referring to.

\n

Thanks,

\n

C 

", "votes": null, "replies": [] }, { "id": 62462, "author_name": "mmueller", "author_url": "", "post_date": "01/08/2015 16:32:40", "content": "

[quote=clustifier;62454]

\n

[quote=fchollet;58089]

\n

Awesome! Thank you for sharing this. The method is pretty much state of the art, and the code quality is excellent! Huge thumb up.

\n

If you guys haven't already, be sure to check out the Google paper that describes this method. Hint: it contains useful info on how to improve the algo :) 

\n

http://www.eecs.tufts.edu/~dsculley/papers/ad-click-prediction.pdf

\n

[/quote]

\n

@fchollet, can you please hint further on what part of the paper you were referring to when saying "it contains useful info on how to improve the algo". I've read it for the second time today but I still not sure what you were referring to.

\n

Thanks,

\n

C 

\n

[/quote]

\n

I guess he is most likely reffering to "CALIBRATING PREDICTIONS"

", "votes": null, "replies": [] }, { "id": 62519, "author_name": "vvarkey", "author_url": "", "post_date": "01/09/2015 18:35:16", "content": "

the paper seems to suggest that the only reason to use FTRL-proximal is to have a sparse weight matrix. we're not using this sparsity, we store the whole weight matrix. so it should be better to use plain Logistic Regression. Atleast that's what i got, please correct me.

", "votes": null, "replies": [] }, { "id": 62577, "author_name": "", "author_url": "", "post_date": "01/10/2015 18:37:25", "content": "

Some of the features we are using are maybe irrelevant for the prediction, more probably the noise is superior to the signal for one or both of these reasons:
- A lot of noise, i.e. the feature have a small count.
- A weak signal, i.e. the feature have a very small predictive value.

\n

So as I understand it, the weight vector is sparse in order to get rid of this kind of features, hopefully improving the generalisation performance, by avoiding fitting the noise.

\n

I don't see much value in storing a sparse weight vector, it is small anyway.

\n

And for the record, the FTRL-proximal is just a plain Logistic Regression with an elastic net regularisation (L1 + L2). The FTRL's advantage is to optimize with a small number of passes over the data, Google's paper gave me the impression that it is optimized for one pass.

", "votes": null, "replies": [] }, { "id": 62600, "author_name": "jyt274463", "author_url": "", "post_date": "01/11/2015 08:18:10", "content": "

This is magic! I'm still figuring out how it works, but it's just fascinating! 

", "votes": null, "replies": [] }, { "id": 62767, "author_name": "jiweiliu", "author_url": "", "post_date": "01/15/2015 21:46:43", "content": "

hi, could someone enlighten me why FTRL induces sparsity? compared to traditional SGD?

\n

Thank you very much!

", "votes": null, "replies": [] }, { "id": 62903, "author_name": "mikhailtrofimov", "author_url": "", "post_date": "01/19/2015 11:56:01", "content": "

As I understood, FTRL "activate" coordinate weight (makes it non-zero) only if this weight have enough information (samples). And L1-penalty introduce some sparsity.

\n

Feel free to correct me if I'm wrong =)

", "votes": null, "replies": [] }, { "id": 62905, "author_name": "jiweiliu", "author_url": "", "post_date": "01/19/2015 14:32:40", "content": "

[quote=Mikhail Trofimov;62903]

\n

As I understood, FTRL "activate" coordinate weight (makes it non-zero) only if this weight have enough information (samples). And L1-penalty introduce some sparsity.

\n

Feel free to correct me if I'm wrong =)

\n

[/quote]

\n

Thank you very much. It makes good sense to me.

", "votes": null, "replies": [] }, { "id": 64213, "author_name": "inspector", "author_url": "", "post_date": "02/14/2015 04:17:36", "content": "

[quote=Yannick Martel;59144]

\n

Hello all,

\n

I have worked on the fast_solution great piece of code and added a few features:

\n

- parameter control from command line

\n

- dropout: where you randomly drop features out of consideration - prevents overfitting and features adaptation

\n

- inclusion of the # of samples seen per device_ip / device_id as a feature

\n

- inclusion of weedkay as a feature

\n

I am down to roughly 0.3948 on the LB with a lot of tuning, by running on all test samples - with a few hours of processing using pypy on an i7, using a few GB's of RAM. LB is very close to my own cross-validation (run on all days except 30, test on 30).

\n

The code probably would benefit from some optimization. I have written the first version in Julia - it was faster than C-Python, but 2-3 times slower than pypy, and I quickly reverted to Python.

\n

I would be glad of any constructive feedback.

\n

Yannick

\n

[/quote]

\n

Yannick,

\n

I am curious about the dropout code. It looks like you are dividing the linear predictor wTx by 'dropout' when predicting an example (or during test time). I thought typically the prediction was multiplied by the dropout rate. Curious about this method. Thanks!

\n

for j, i in enumerate(self._indices(x)):

if dropped != None and dropped[j]:
continue

wTx += w[i]

if dropped != None: wTx /= dropout

\n

# bounded sigmoid function, this is the probability estimation
return 1. / (1. + exp(-max(min(wTx, 35.), -35.)))

", "votes": null, "replies": [] }, { "id": 64220, "author_name": "martely", "author_url": "", "post_date": "02/14/2015 07:27:25", "content": "

[quote=Inspector;64213]

\n

[quote=Yannick Martel;59144]

\n

Hello all,

\n

I have worked on the fast_solution great piece of code and added a few features:

\n

- parameter control from command line

\n

- dropout: where you randomly drop features out of consideration - prevents overfitting and features adaptation

\n

- inclusion of the # of samples seen per device_ip / device_id as a feature

\n

- inclusion of weedkay as a feature

\n

I am down to roughly 0.3948 on the LB with a lot of tuning, by running on all test samples - with a few hours of processing using pypy on an i7, using a few GB's of RAM. LB is very close to my own cross-validation (run on all days except 30, test on 30).

\n

The code probably would benefit from some optimization. I have written the first version in Julia - it was faster than C-Python, but 2-3 times slower than pypy, and I quickly reverted to Python.

\n

I would be glad of any constructive feedback.

\n

Yannick

\n

[/quote]

\n

Yannick,

\n

I am curious about the dropout code. It looks like you are dividing the linear predictor wTx by 'dropout' when predicting an example (or during test time). I thought typically the prediction was multiplied by the dropout rate. Curious about this method. Thanks!

\n

for j, i in enumerate(self._indices(x)):

if dropped != None and dropped[j]:
continue

wTx += w[i]

if dropped != None: wTx /= dropout

\n

# bounded sigmoid function, this is the probability estimation
return 1. / (1. + exp(-max(min(wTx, 35.), -35.)))

\n

[/quote]

\n

Dear Inspector,

\n

Yes, my implementation is a bit more intricate than the way you are describing, but it is supposed to work the same.

\n

The "dropout" variable is incorrectly named: it is the rate of features that I am keeping: 1. for keeping everybody, .8 for keeping 80%, .5 for keeping only half of them.

\n

I wanted to apply all of dropout at update time, not prediction time, so I put the weight adjustment in the predict code so it is applied only when updating (predict call with dropped != None).

\n

Dividing wTx by dropout is equivalent to multiplying the features by 1/dropout - making them stronger when they are missing (1-dropout) * N of their buddies, when they should have been N without dropout.

\n

Then, in a sense, instead of reducing weights at prediction time by multiplying by the dropout rate (or more correctly:  1 - rate of dropout), I increase the weights (or features) at learning time by dividing by the same rate.

\n

It should be somewhat the same, except maybe that the impact of the alpha and beta is different.

\n

I have also tried a variant: divide not by the "dropout" variable, but by the ratio of features we are keeping for this particular sample:

\n

if wTx != 0: wTx *= len(dropped) * 1.0 / (len(dropped) - sum(dropped))

\n

It is slightly more complex, a bit longer to compute and I have not found it to improve performance.

\n

I hope the explanation helps, please tell me if not!

", "votes": null, "replies": [] }, { "id": 64234, "author_name": "inspector", "author_url": "", "post_date": "02/14/2015 17:14:40", "content": "

[quote=Yannick Martel;64220]

\n

Dear Inspector,

\n

Yes, my implementation is a bit more intricate than the way you are describing, but it is supposed to work the same.

\n

The "dropout" variable is incorrectly named: it is the rate of features that I am keeping: 1. for keeping everybody, .8 for keeping 80%, .5 for keeping only half of them.

\n

I wanted to apply all of dropout at update time, not prediction time, so I put the weight adjustment in the predict code so it is applied only when updating (predict call with dropped != None).

\n

Dividing wTx by dropout is equivalent to multiplying the features by 1/dropout - making them stronger when they are missing (1-dropout) * N of their buddies, when they should have been N without dropout.

\n

Then, in a sense, instead of reducing weights at prediction time by multiplying by the dropout rate (or more correctly:  1 - rate of dropout), I increase the weights (or features) at learning time by dividing by the same rate.

\n

It should be somewhat the same, except maybe that the impact of the alpha and beta is different.

\n

I have also tried a variant: divide not by the "dropout" variable, but by the ratio of features we are keeping for this particular sample:

\n

if wTx != 0: wTx *= len(dropped) * 1.0 / (len(dropped) - sum(dropped))

\n

It is slightly more complex, a bit longer to compute and I have not found it to improve performance.

\n

I hope the explanation helps, please tell me if not!

\n

[/quote]

\n

I am still confused I am afraid. I think it might be that I am thinking about this for offline learning.

\n

I. First, is my understanding correct that the dropout adjustment (generally, not in this type of situation necessarily) is done because when training a model, (dropout rate)% features are removed, thus we expect the remaining (1-dropout rate)% coefficients to be greater in magnitude than otherwise (they are compensating for the loss of the other features). So for example, if we keep 75% of the features, the weights will be on average too large (25% too large). Attached is an example of this and the adjustment.

\n

II. Assuming this is correct, is FTRL case different? Will the weights not adjust to compensate for the missing (dropped) ones, so that each weight will end up being too small and that is why you divide the sum wTx by dropout?

\n

III. Do you divide by dropout weight when predicting the test set as well? I assume so, but just checking.

\n

Thanks!

", "votes": null, "replies": [] }, { "id": 64235, "author_name": "martely", "author_url": "", "post_date": "02/14/2015 19:03:16", "content": "

I think that we are quite close in understanding, but let me try to explain in another way, with a concrete example: if I (randomly) remove, let's say, 20% of features, then to predict correctly I must increase the weights by 1.25 = 1/0.8, in order to keep the sum of weights coherent.

\n

With dropout, we remove features only when learning, not when predicting. During learning, there are two steps:

\n
    \n
  1. predict for the value of x for which we want to learn -> with dropout we predict with a randomly selected of features, let's say we keep only [1, 2, 4, 5] out of [1, 2, 3, 4, 5];
    2. \n
  2. update the weights for the features we have seen based on the difference between the the expected y and the p_hat we get - so here we update weights for features [1, 2, 4, 5].
    3. \n
\n

In step 1, I predict with weights multiplied by 1.25 (with 80% dropout rate, or 20% of features removed). In step 2, I update weights for [1, 2, 4, 5], with no dropout factor.

\n

When predicting, I would predict with all of [1, 2, 3, 4, 5], with no weight adjustment.

\n\n

If we assume all weights are 1, then the sum of weights is:

\n\n\n

Does this make more sense to you?

", "votes": null, "replies": [] }, { "id": 64241, "author_name": "inspector", "author_url": "", "post_date": "02/14/2015 20:25:39", "content": "

[quote=Yannick Martel;64235]

\n

I think that we are quite close in understanding, but let me try to explain in another way, with a concrete example: if I (randomly) remove, let's say, 20% of features, then to predict correctly I must increase the weights by 1.25 = 1/0.8, in order to keep the sum of weights coherent.

\n

With dropout, we remove features only when learning, not when predicting. During learning, there are two steps:

\n
    \n
  1. predict for the value of x for which we want to learn -> with dropout we predict with a randomly selected of features, let's say we keep only [1, 2, 4, 5] out of [1, 2, 3, 4, 5];
    2. \n
  2. update the weights for the features we have seen based on the difference between the the expected y and the p_hat we get - so here we update weights for features [1, 2, 4, 5].
    3. \n
\n

In step 1, I predict with weights multiplied by 1.25 (with 80% dropout rate, or 20% of features removed). In step 2, I update weights for [1, 2, 4, 5], with no dropout factor.

\n

When predicting, I would predict with all of [1, 2, 3, 4, 5], with no weight adjustment.

\n

If we assume all weights are 1, then the sum of weights is:

\n\n

Does this make more sense to you?

\n

[/quote]

\n

Thank you for your continued explanation!

\n

Here is my understanding of the standard way (what is discussed by Hinton) and how you are doing it here.It seems that VW uses the "standard way": http://permalink.gmane.org/gmane.comp.ai.machine-learning.vowpal-wabbit/163

\n\n

Please tell me I understand properly :)

\n\n

* N features are available
* p = probability a feature wont drop out (1-p it will)

\n

The Standard Way:
*A training example is seen and p*N of its features are used to predict the value of the output
*The update to these p*N weights is larger than it would normally be if all N features had been present. This is because the error (predicted - actual) is larger.
*Thus, on average, all the weights will be larger by 1/(1-p) and so we multiply the final weights by p at test time, when all N features will be present. Otherwise we would have trained a model with weights too large.

\n

Non-Standard Way:
*A training example is seen and p*N of its features are used to predict the value of the output (just like the standard way)
* We divide the predicted value by p, which means on average the update to the p*N weights will be what they would have been without dropout because the error was reduced (predicted / p - actual).
*At test time, no need to do anything - use the final weights as is and include all the N features for every example

", "votes": null, "replies": [] }, { "id": 64245, "author_name": "martely", "author_url": "", "post_date": "02/14/2015 21:00:06", "content": "

Congratulations, I believe that you have nailed it. Your explanation seems to me both correct and very clear!

\n

The "standard way" is what I have seen explaned in the litterature on the topic. I have implemented the "non standard way" because I wanted to keep the prediction time cleaner - no dropout coefficients to adjust.

\n

As I have mentionned above, I have tested the option to correct at learning time by the actual proportion of features which have been selected for this particular sample - but it has not changed anything. Maybe it would make a difference with a dropout rate closer to 0.5 than what I used.

\n

I have also tested with no adjustment to weights, and it performed very poorly, as you could expect...

\n

Another difference to the standard way: I apply dropout also to the bias term - not 100% sure this is a smart thing to do.

", "votes": null, "replies": [] }, { "id": 66062, "author_name": "martely", "author_url": "", "post_date": "03/12/2015 21:09:38", "content": "

Hi to all,

\n

I have created a small github repository with the latest release of my version of fast_solution.py, heavily modified with both Google's FTRL Proximal and Microsoft's AdPredictor. Here it is: http://bit.ly/GHFASTDG. My team came to the 22th rank by using an ensembling of FTRL, AdPredictor and Vowpal Wabbit models.

\n

Many thanks to tinrtgu for providing the original code and the inspiration!

\n

You can also read some write-up's on the competition at http://bit.ly/CTRK1 and http://bit.ly/CBLOGBOOK1.

", "votes": null, "replies": [] }, { "id": 89310, "author_name": "cowabungadood", "author_url": "", "post_date": "08/13/2015 17:54:13", "content": "

Hi if anyone is still reading this thread I could use some help understanding the predict method in the script. I was wondering if you could explain the reasoning behind the way wTx inner product is being implemented in the predict method. From reading the code it seems that the x features that have been hashed are not being used in the computation of the weight vector. Shouldn't the the inner product of the predict function be wTx = w[i] * i, which would follow along the lines of a typical logistic regression computation? Because currently it looks like it is just summing the computed weights, with no affect from the actual feature vector.

", "votes": null, "replies": [ { "id": 585879, "author_name": "pannana", "author_url": "", "post_date": "07/28/2019 08:39:19", "content": "

Hi\nI had the same question. Then I found some commends which may help.

\n\n

\"\"\"Predict for features.\n Parameters:\n ----------\n x (list of int): a list of index of non-zero features\n Outputs:\n ----------\n p (float): prediction for input features\n \"\"\"

", "votes": null, "replies": [] } ] }, { "id": 89315, "author_name": "inversion", "author_url": "", "post_date": "08/13/2015 18:56:16", "content": "

[quote=Robert;89310]

\n\n

Hi if anyone is still reading this thread I could use some help understanding the predict method in the script. I was wondering if you could explain the reasoning behind the way wTx inner product is being implemented in the predict method. From reading the code it seems that the x features that have been hashed are not being used in the computation of the weight vector. Shouldn't the the inner product of the predict function be wTx = w[i] * i, which would follow along the lines of a typical logistic regression computation? Because currently it looks like it is just summing the computed weights, with no affect from the actual feature vector.

\n\n

[/quote]

\n\n

x is either 1 or 0.

\n\n

It's only calculating the weights for the indices of x that are 1.

\n\n

w[i] * 1 = w[i]

", "votes": null, "replies": [ { "id": 585880, "author_name": "pannana", "author_url": "", "post_date": "07/28/2019 08:40:09", "content": "

Hi,\nI had the same question. Then I found some commends which may help.

\n\n

\"\"\"Predict for features.\n Parameters:\n ----------\n x (list of int): a list of index of non-zero features\n Outputs:\n ----------\n p (float): prediction for input features\n \"\"\"

", "votes": null, "replies": [] } ] }, { "id": 89319, "author_name": "cowabungadood", "author_url": "", "post_date": "08/13/2015 20:43:33", "content": "

[quote=inversion;89315]

\n\n

[quote=Robert;89310]

\n\n

Hi if anyone is still reading this thread I could use some help understanding the predict method in the script. I was wondering if you could explain the reasoning behind the way wTx inner product is being implemented in the predict method. From reading the code it seems that the x features that have been hashed are not being used in the computation of the weight vector. Shouldn't the the inner product of the predict function be wTx = w[i] * i, which would follow along the lines of a typical logistic regression computation? Because currently it looks like it is just summing the computed weights, with no affect from the actual feature vector.

\n\n

[/quote]

\n\n

x is either 1 or 0.

\n\n

It's only calculating the weights for the indices of x that are 1.

\n\n

w[i] * 1 = w[i]

\n\n

[/quote]

\n\n

I thought x was a vector that contained the hashed values, which are indexed values from 1 to D. And the predict method is looping through the x vector with i. Or is x 0 or 1 due to the one hot encoding? Thanks for the reply by the way.

", "votes": null, "replies": [] }, { "id": 89357, "author_name": "inversion", "author_url": "", "post_date": "08/14/2015 12:03:02", "content": "

[quote=Robert;89319]

\n\n

I thought x was a vector that contained the hashed values, which are indexed values from 1 to D. And the predict method is looping through the x vector with i. Or is x 0 or 1 due to the one hot encoding? Thanks for the reply by the way.

\n\n

[/quote]

\n\n

I should have been more clear. Yes, x is the hashed values, the hashed valued being the index position of the 1's. (Due to the OneHotEncoding, as you point out.) So x is not really x; rather, it is a sparse representation of x, e.g.,

\n\n

\nx = [2,5]\n

\n\n

represents

\n\n

\nx = [0, 0, 1, 0, 0, 1]\n

", "votes": null, "replies": [] }, { "id": 96647, "author_name": "swapniel", "author_url": "", "post_date": "10/19/2015 12:45:48", "content": "

This may sound like a naive question but in previous version here the update rule was:

\n\n
\n

w[i] -= (p - y) * alpha / (sqrt(n[i]) + 1.)

\n
\n\n

But in new this version it has changed to:

\n\n
\n

g = p - y

\n \n

z[i] += g - sigma * w[i]

\n
\n\n

Why is the correction being added instead of subtracted now? Please tell me what I missed.

", "votes": null, "replies": [] }, { "id": 383591, "author_name": "cyfei1106", "author_url": "", "post_date": "09/09/2018 03:31:44", "content": "

Thanks!

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "58054": "", "58055": "", "58056": "", "58057": "", "58058": "", "58061": "", "58062": "", "58063": "", "58064": "", "58065": "", "58066": "", "58069": "", "58072": "", "58083": "", "58089": "", "58091": "", "58092": "", "58093": "", "58095": "", "58108": "", "58110": "", "58111": "", "58112": "", "58118": "", "58130": "", "58131": "", "58158": "", "58160": "", "58161": "", "58182": "", "58220": "", "58221": "", "58223": "", "58277": "", "58334": "", "58335": "", "58336": "", "58341": "", "58343": "", "58352": "", "58366": "", "58380": "", "58448": "", "58464": "", "58497": "", "58514": "", "58522": "", "59144": "", "59146": "", "59149": "", "59200": "", "59210": "", "59216": "", "59235": "", "59252": "", "59253": "", "59256": "", "59283": "", "59289": "", "59372": "", "59441": "", "59472": "", "59474": "", "59483": "", "59488": "", "59493": "", "59497": "", "59499": "", "59501": "", "59511": "", "59515": "", "59523": "", "59526": "", "59530": "", "59534": "", "59538": "", "59542": "", "59554": "", "59595": "", "59635": "", "59680": "", "59705": "", "59709": "", "59710": "", "59715": "", "59724": "", "59738": "", "59753": "", "59760": "", "59771": "", "59782": "", "59800": "", "59801": "", "59802": "", "59806": "", "59808": "", "59809": "", "59811": "", "59812": "", "59813": "", "59821": "", "59826": "", "59828": "", "59829": "", "59830": "", "59831": "", "59851": "", "59858": "", "59859": "", "59861": "", "59862": "", "59863": "", "59864": "", "59865": "", "59869": "", "59872": "", "59889": "", "59891": "", "59899": "", "59907": "", "59909": "", "59931": "", "59936": "", "59938": "", "59951": "", "60037": "", "60038": "", "60040": "", "60061": "", "60062": "", "60144": "", "60145": "", "60148": "", "60155": "", "60162": "", "60175": "", "60180": "", "60185": "", "60190": "", "60197": "", "60199": "", "60231": "", "60316": "", "60317": "", "60318": "", "60460": "", "61621": "", "61642": "", "61646": "", "61700": "", "61708": "", "61715": "", "61717": "", "61718": "", "61720": "", "61722": "", "61774": "", "61777": "", "62122": "", "62128": "", "62195": "", "62203": "", "62409": "", "62425": "", "62447": "", "62454": "", "62462": "", "62519": "", "62577": "", "62600": "", "62767": "", "62903": "", "62905": "", "64213": "", "64220": "", "64234": "", "64235": "", "64241": "", "64245": "", "66062": "", "89310": "Hi if anyone is still reading this thread I could use some help understanding the predict method in the script. I was wondering if you could explain the reasoning behind the way wTx inner product is being implemented in the predict method. From reading the code it seems that the x features that have been hashed are not being used in the computation of the weight vector. Shouldn't the the inner product of the predict function be wTx = w[i] * i, which would follow along the lines of a typical logistic regression computation? Because currently it looks like it is just summing the computed weights, with no affect from the actual feature vector.", "89315": "[quote=Robert;89310]\r\n\r\nHi if anyone is still reading this thread I could use some help understanding the predict method in the script. I was wondering if you could explain the reasoning behind the way wTx inner product is being implemented in the predict method. From reading the code it seems that the x features that have been hashed are not being used in the computation of the weight vector. Shouldn't the the inner product of the predict function be wTx = w[i] * i, which would follow along the lines of a typical logistic regression computation? Because currently it looks like it is just summing the computed weights, with no affect from the actual feature vector.\r\n\r\n[/quote]\r\n\r\n`x` is either 1 or 0.\r\n\r\nIt's only calculating the weights for the indices of `x` that are 1. \r\n\r\n`w[i] * 1 = w[i]`", "89319": "[quote=inversion;89315]\r\n\r\n[quote=Robert;89310]\r\n\r\nHi if anyone is still reading this thread I could use some help understanding the predict method in the script. I was wondering if you could explain the reasoning behind the way wTx inner product is being implemented in the predict method. From reading the code it seems that the x features that have been hashed are not being used in the computation of the weight vector. Shouldn't the the inner product of the predict function be wTx = w[i] * i, which would follow along the lines of a typical logistic regression computation? Because currently it looks like it is just summing the computed weights, with no affect from the actual feature vector.\r\n\r\n[/quote]\r\n\r\n`x` is either 1 or 0.\r\n\r\nIt's only calculating the weights for the indices of `x` that are 1. \r\n\r\n`w[i] * 1 = w[i]`\r\n\r\n[/quote]\r\n\r\n\r\nI thought x was a vector that contained the hashed values, which are indexed values from 1 to D. And the predict method is looping through the x vector with i. Or is x 0 or 1 due to the one hot encoding? Thanks for the reply by the way.", "89357": "[quote=Robert;89319]\r\n\r\nI thought x was a vector that contained the hashed values, which are indexed values from 1 to D. And the predict method is looping through the x vector with i. Or is x 0 or 1 due to the one hot encoding? Thanks for the reply by the way. \r\n\r\n[/quote]\r\n\r\nI should have been more clear. Yes, `x` is the hashed values, the hashed valued being the index position of the 1's. (Due to the OneHotEncoding, as you point out.) So `x` is not really `x`; rather, it is a sparse representation of `x`, e.g.,\r\n\r\n```\r\nx = [2,5]\r\n```\r\n\r\nrepresents\r\n\r\n```\r\nx = [0, 0, 1, 0, 0, 1]\r\n```", "96647": "This may sound like a naive question but in previous version [here][1] the update rule was:\r\n\r\n> w[i] -= (p - y) * alpha / (sqrt(n[i]) + 1.)\r\n\r\nBut in new this version it has changed to:\r\n\r\n> g = p - y \r\n\r\n> z[i] += g - sigma * w[i]\r\n\r\nWhy is the correction being added instead of subtracted now? Please tell me what I missed.\r\n\r\n [1]: https://www.kaggle.com/c/criteo-display-ad-challenge/forums/t/10322/beat-the-benchmark-with-less-then-200mb-of-memory", "383591": "Thanks!", "585879": "Hi\nI had the same question. Then I found some commends which may help.\n\n\n \"\"\"Predict for features.\n Parameters:\n ----------\n x (list of int): a list of index of non-zero features\n Outputs:\n ----------\n p (float): prediction for input features\n \"\"\"", "585880": "Hi,\nI had the same question. Then I found some commends which may help.\n\n\n \"\"\"Predict for features.\n Parameters:\n ----------\n x (list of int): a list of index of non-zero features\n Outputs:\n ----------\n p (float): prediction for input features\n \"\"\"", "1104283": "who knows where is file with code?" }, "source": "meta" }