{ "id": 12478, "title": "Kaggler - Python Package for Kaggler", "url": "/competitions/avazu-ctr-prediction/discussion/12478", "author_name": "jeong", "post_date": "2015-02-11T09:17:14.550000", "votes": 51, "comment_count": 35, "views": 11323, "content": "

Hi all,

\n

It was another great competition with tons of learning - special thanks to tinrtgu and fellow Kagglers who shared their insights, and big congratulations to the winners!

\n

Here I'd like to share what I've put together for online learning as a Python package - named Kaggler.

\n

You can install it with pip as follows:

\n

    $ pip install -U Kaggler

\n

then, import algorithm classes as follows:

\n

    from kaggler.online_model import SGD, FTRL, FM, NN, NN_H2

\n

Currently it supports 4 online learning algorithms - SGD, FTRL, FM (original version, not 3 idiot's yet), NN (1 or 2 ReLU hidden layers).

\n

It uses the liblinear style sparse input format - It is chosen so that the same input file can be used across other popular tools such as XGBoost, VW (with slight modification), libFM, SVMLight, etc.

\n

Code and examples are available at https://github.com/jeongyoonlee/Kaggler, and package documentation is available at http://pythonhosted.org//Kaggler/.

\n

I plan to update it as I participate more competitions and learn more - supporting the CSV input format and adding FM-FTRL by Daniel Yoo will be my next update.

\n

Hope you guys find this useful, and see you at other competitions! :)

", "messages": [ { "id": 64009, "postDate": "2015-02-11T09:17:14.550Z", "content": "

Hi all,

\n

It was another great competition with tons of learning - special thanks to tinrtgu and fellow Kagglers who shared their insights, and big congratulations to the winners!

\n

Here I'd like to share what I've put together for online learning as a Python package - named Kaggler.

\n

You can install it with pip as follows:

\n

    $ pip install -U Kaggler

\n

then, import algorithm classes as follows:

\n

    from kaggler.online_model import SGD, FTRL, FM, NN, NN_H2

\n

Currently it supports 4 online learning algorithms - SGD, FTRL, FM (original version, not 3 idiot's yet), NN (1 or 2 ReLU hidden layers).

\n

It uses the liblinear style sparse input format - It is chosen so that the same input file can be used across other popular tools such as XGBoost, VW (with slight modification), libFM, SVMLight, etc.

\n

Code and examples are available at https://github.com/jeongyoonlee/Kaggler, and package documentation is available at http://pythonhosted.org//Kaggler/.

\n

I plan to update it as I participate more competitions and learn more - supporting the CSV input format and adding FM-FTRL by Daniel Yoo will be my next update.

\n

Hope you guys find this useful, and see you at other competitions! :)

", "votes": 51 }, { "id": 365296, "postDate": "2018-08-02T09:35:14.450Z", "content": "

Thanks for sharing.\nwhen i use clf.fit(x_train,y_train)\ni get a problem:ValueError: Buffer dtype mismatch, expected 'int' but got 'long'\nis this can only run on 32bit mechine?

", "rawMarkdown": "Thanks for sharing.\nwhen i use clf.fit(x_train,y_train)\ni get a problem:ValueError: Buffer dtype mismatch, expected 'int' but got 'long'\nis this can only run on 32bit mechine?", "votes": 1 }, { "id": 64090, "postDate": "2015-02-12T03:59:50.233Z", "content": "

Thanks Jeong-Yoon Lee for sharing your code.

\n

I tried using "pip install Kaggler" and got the following error:

\n

IOError: [Errno 2] No such file or directory: 'README.md'

\n

It seems like you forgot to include this file onto the PyPI repository.

\n\n\n

EDIT: Oops! Someone just beat me to raising the question. I did not notice that when posting...

", "votes": 1 }, { "id": 64086, "postDate": "2015-02-12T03:26:32.707Z", "content": "

Thanks for sharing!

\n

I encounter the installation error when I try "pip install Kaggler"

\n

IOError: [Errno 2] No such file or directory: 'README.md'

\n

Any hint to resolve it?

", "votes": 1 }, { "id": 64043, "postDate": "2015-02-11T15:45:55.973Z", "content": "

do you have the dev version of libgfortran3?

", "votes": 1 }, { "id": 64315, "postDate": "2015-02-16T05:26:49.273Z", "content": "

[quote=CodingNeo;64304]

\n

@Jeong-Yoon Lee, I fixed the issue which is caused by the wrong value of input parameters. I am trying the factorization machine in this package using the train data of this competition. I did the one hot encoding by hash trick as the input data. The factorization machine give me very bad result. I wonder something I did wrong. So consult you with two following question:

\n
    \n
  1. I simply convert the data by one-hot encoding before sending to FM. Is it correct?
    2. \n
  2. Can you share the parameter values you used for FM for this problem?
    3. \n
\n

[/quote]

\n

I did as follows:

\n

1. Extracting the weekday and hour-of-day variables from the hour column

\n

2. One-hot-encoding all columns with the cut-off of 100 (unique values of which frequency are less than 100 are combined into one category)

\n

3. Random shuffling input stream with the shuffling window of 1,000,000 - which is not part of the FM class yet, but there is a function, shuf_file() in kaggler.io you can take a look.

\n

4. Parameters used are: FM(n=60000, dim=16, a=0.02) - n was small because after one-hot-encoding the number of features was less than 60K.

\n

5. Then iterate with training data 10 times.

\n

This scored 0.3978 and 0.3959 on the public and private leaderboards respectively.  It was my only submission using kaggler.online_model.FM.

", "votes": 2 }, { "id": 64451, "postDate": "2015-02-18T01:23:49.397Z", "content": "

[quote=Jeong-Yoon Lee;64347]

\n

[quote=CodingNeo;64343]

\n

[quote=Jeong-Yoon Lee;64341]

\n

[quote=CodingNeo;64319]

\n

Thanks Jeong-Yoon Lee. You share with me a lot of information. One more question to ask you for help. How can I get the probability of the prediction instead of 0/1 prediction? I need the probability to compute the log loss.

\n

[/quote]

\n

It always outputs the probability.

\n

[/quote]

\n

I got all prediction with the values of 1.00000/0.00000

\n

[/quote]

\n

It seems that the learning rate is too high and makes the output of the sigmoid function (1/(1 + exp(-x))) saturated

\n

[/quote]

\n\n

Thank you very much! Reducing the learning rate resolves the issue.

" }, { "id": 64347, "postDate": "2015-02-16T15:59:15.597Z", "content": "

[quote=CodingNeo;64343]

\n

[quote=Jeong-Yoon Lee;64341]

\n

[quote=CodingNeo;64319]

\n

Thanks Jeong-Yoon Lee. You share with me a lot of information. One more question to ask you for help. How can I get the probability of the prediction instead of 0/1 prediction? I need the probability to compute the log loss.

\n

[/quote]

\n

It always outputs the probability.

\n

[/quote]

\n

I got all prediction with the values of 1.00000/0.00000

\n

[/quote]

\n

It seems that the learning rate is too high and makes the output of the sigmoid function (1/(1 + exp(-x))) saturated

" }, { "id": 64343, "postDate": "2015-02-16T15:05:22.110Z", "content": "

[quote=Jeong-Yoon Lee;64341]

\n

[quote=CodingNeo;64319]

\n

Thanks Jeong-Yoon Lee. You share with me a lot of information. One more question to ask you for help. How can I get the probability of the prediction instead of 0/1 prediction? I need the probability to compute the log loss.

\n

[/quote]

\n

It always outputs the probability.

\n

[/quote]

\n

I got all prediction with the values of 1.00000/0.00000

" }, { "id": 64342, "postDate": "2015-02-16T15:04:43.153Z", "content": "

I got all 1.00000/0.00000 in prediction with the parameters (100000, 4, 1.0). Why?

" }, { "id": 64341, "postDate": "2015-02-16T14:57:44.287Z", "content": "

[quote=CodingNeo;64319]

\n

Thanks Jeong-Yoon Lee. You share with me a lot of information. One more question to ask you for help. How can I get the probability of the prediction instead of 0/1 prediction? I need the probability to compute the log loss.

\n

[/quote]

\n

It always outputs the probability.

" }, { "id": 64332, "postDate": "2015-02-16T10:17:37.923Z", "content": "

[quote=Jeong-Yoon Lee;64270]

\n

[quote=Jiming Ye;64266]

\n

 I tried to learn how to implement those online algorithm from your code, and I have a question.

\n

In your FM model and NN, you have put counters in. And you use them like:

\n

w1[h] -= dl_dy / (sqrt(c) + 1) + l2 * w1[h]

\n

I haven't seen those things before. Actually when I implemented FM and NN, I didn't use them(Maybe that's the reason my FM model performs badly)

\n

Can you explain a little bit about these counters? I know this question might be somewhat dumb, but I just cannot figure it out by myself :(

\n

Thanks!

\n

[/quote]

\n

It's again from tinrtgu's code based on Google's FTRL paper at the Criteo competition.  Actually 1 / (sqrt(c) + 1) is not what was used in the paper, but - I guess - his approximation of 1 / (sqrt(sum(g * g)) + 1).

\n

In later updates, I will try 1/(sqrt(sum(g*g)) + 1).

\n

EDIT:

\n

1 / sqrt(n) is common in stochastic gradient learning, and it is called per-coordinate learning rate if the i-th weight, w[i] is updated using c[i] only if it's associated with non-zero input x[i].

\n

To do so, 1 / (sqrt(c[i]) + 1) is straightforward extension of 1 / sqrt(n).  However, as in the paper, 1 / (sqrt(sum(g[i] * g[i])) + 1) is a better choice, where g[i] is per-coordinate gradient for the i-th weight.

\n

[/quote]

\n

Thank you so much!

\n

Now I understand :)

" }, { "id": 64319, "postDate": "2015-02-16T06:33:08.107Z", "content": "

Thanks Jeong-Yoon Lee. You share with me a lot of information. One more question to ask you for help. How can I get the probability of the prediction instead of 0/1 prediction? I need the probability to compute the log loss.

" }, { "id": 64304, "postDate": "2015-02-16T02:36:37.163Z", "content": "

@Jeong-Yoon Lee, I fixed the issue which is caused by the wrong value of input parameters. I am trying the factorization machine in this package using the train data of this competition. I did the one hot encoding by hash trick as the input data. The factorization machine give me very bad result. I wonder something I did wrong. So consult you with two following question:

\n
    \n
  1. I simply convert the data by one-hot encoding before sending to FM. Is it correct?
    2. \n
  2. Can you share the parameter values you used for FM for this problem?
    3. \n
" }, { "id": 64270, "postDate": "2015-02-15T14:35:45.517Z", "content": "

[quote=Jiming Ye;64266]

\n

 I tried to learn how to implement those online algorithm from your code, and I have a question.

\n

In your FM model and NN, you have put counters in. And you use them like:

\n

w1[h] -= dl_dy / (sqrt(c) + 1) + l2 * w1[h]

\n

I haven't seen those things before. Actually when I implemented FM and NN, I didn't use them(Maybe that's the reason my FM model performs badly)

\n

Can you explain a little bit about these counters? I know this question might be somewhat dumb, but I just cannot figure it out by myself :(

\n

Thanks!

\n

[/quote]

\n

It's again from tinrtgu's code based on Google's FTRL paper at the Criteo competition.  Actually 1 / (sqrt(c) + 1) is not what was used in the paper, but - I guess - his approximation of 1 / (sqrt(sum(g * g)) + 1).

\n

In later updates, I will try 1/(sqrt(sum(g*g)) + 1).

\n

EDIT:

\n

1 / sqrt(n) is common in stochastic gradient learning, and it is called per-coordinate learning rate if the i-th weight, w[i] is updated using c[i] only if it's associated with non-zero input x[i].

\n

To do so, 1 / (sqrt(c[i]) + 1) is straightforward extension of 1 / sqrt(n).  However, as in the paper, 1 / (sqrt(sum(g[i] * g[i])) + 1) is a better choice, where g[i] is per-coordinate gradient for the i-th weight.

" }, { "id": 64269, "postDate": "2015-02-15T14:05:55.100Z", "content": "

[quote=CodingNeo;64214]

\n

I try to convert the train/test data into sparse format for using this package. But I got an zero division error in fm.read_sparse function. Any hints to resolve this issue?

\n

[/quote]

\n

I just tested SGD, FM, FTRL, NN and all worked fine on my side.  Can you provide with more details that I can reproduce the error?

\n

Also, since I've updated the package quite a bit, please try the latest version.

" }, { "id": 64266, "postDate": "2015-02-15T12:37:21.313Z", "content": "

 I tried to learn how to implement those online algorithm from your code, and I have a question.

\n

In your FM model and NN, you have put counters in. And you use them like:

\n

w1[h] -= dl_dy / (sqrt(c) + 1) + l2 * w1[h]

\n

I haven't seen those things before. Actually when I implemented FM and NN, I didn't use them(Maybe that's the reason my FM model performs badly)

\n

Can you explain a little bit about these counters? I know this question might be somewhat dumb, but I just cannot figure it out by myself :(

\n

Thanks!

" }, { "id": 64214, "postDate": "2015-02-14T04:25:53.340Z", "content": "

I try to convert the train/test data into sparse format for using this package. But I got an zero division error in fm.read_sparse function. Any hints to resolve this issue?

" }, { "id": 64131, "postDate": "2015-02-12T17:57:20.103Z", "content": "

[quote=binga;64099]

\n

[quote=Charles;64090]

\n

Thanks Jeong-Yoon Lee for sharing your code.

\n

I tried using "pip install Kaggler" and got the following error:

\n

IOError: [Errno 2] No such file or directory: 'README.md'

\n

It seems like you forgot to include this file onto the PyPI repository.

\n

EDIT: Oops! Someone just beat me to raising the question. I did not notice that when posting...

\n

[/quote]

\n

I got the same error by installing through pip. However, by downloading the source and running

\n

python setup.py install

\n

everything went smooth.

\n

[/quote]

\n

Hi @binga.  Sorry for the error.

\n

I noticed that "pip install" without "-U" will fail if there was a previous unsuccessful installation attempt.  

\n

I tried "pip install -U Kaggler" on Mac and Ubuntu, and was able to install the package successfully.

\n

Can anyone else confirm either successful or unsuccessful installation by using pip?

\n

Thanks!

" }, { "id": 64105, "postDate": "2015-02-12T10:16:29.433Z", "content": "

[quote=Charles;64104]

\n

python setup.py build_ext --inplace

\n

[/quote]

\n

Oh, yes. You're right! Same here.

" }, { "id": 64104, "postDate": "2015-02-12T10:11:46.160Z", "content": "

[quote=binga;64099]

\n

[quote=Charles;64090]

\n

Thanks Jeong-Yoon Lee for sharing your code.

\n

I tried using "pip install Kaggler" and got the following error:

\n

IOError: [Errno 2] No such file or directory: 'README.md'

\n

It seems like you forgot to include this file onto the PyPI repository.

\n

EDIT: Oops! Someone just beat me to raising the question. I did not notice that when posting...

\n

[/quote]

\n

I got the same error by installing through pip. However, by downloading the source and running

\n

python setup.py install

\n

everything went smooth.

\n

[/quote]

\n

I ended up also downloading the source and building from source. But I think you'd need to first run (as documented on the README page) 

\n

python setup.py build_ext --inplace

\n

before running

\n

python setup.py install

\n

for the underlying C code to compile correctly. At least for me, the installation would not work without running the first command...

" }, { "id": 64099, "postDate": "2015-02-12T07:38:25.153Z", "content": "

[quote=Charles;64090]

\n

Thanks Jeong-Yoon Lee for sharing your code.

\n

I tried using "pip install Kaggler" and got the following error:

\n

IOError: [Errno 2] No such file or directory: 'README.md'

\n

It seems like you forgot to include this file onto the PyPI repository.

\n

EDIT: Oops! Someone just beat me to raising the question. I did not notice that when posting...

\n

[/quote]

\n

I got the same error by installing through pip. However, by downloading the source and running

\n

python setup.py install

\n

everything went smooth.

" }, { "id": 64092, "postDate": "2015-02-12T05:24:15.263Z", "content": "

Thanks @CodingNeo and @Charles for the issue report, and sorry for the error.

\n

I fixed it and uploaded a new version - it must work now.  Please try:

\n

$ pip install -U Kaggler

\n\n

I just learned a couple of things about Python packaging by sharing it with you guys:

\n\n

Sharing is learning. So awesome! :D

" }, { "id": 64085, "postDate": "2015-02-12T03:13:54.520Z", "content": "

[quote=Jiming Ye;64081]

\n

Thanks for sharing!

\n

I didn't see any L1 or L2 term in the FM model. I don't know whether this is how original FM works

\n

[/quote]

\n

Hi Jiming, Yes, in the original FM paper by Steffen, there was neither L1 nor L2 term.  However since it's trivial, I'll consider adding them to the model soon. :)

" }, { "id": 64081, "postDate": "2015-02-12T01:31:20.580Z", "content": "

Thanks for sharing!

\n

I didn't see any L1 or L2 term in the FM model. I don't know whether this is how original FM works

" }, { "id": 64063, "postDate": "2015-02-11T20:07:35.093Z", "content": "

Thank you for sharing this!!

\n

It is a great addition to everybody's toolbox.

" }, { "id": 64046, "postDate": "2015-02-11T17:24:05.687Z", "content": "

[quote=rcarson;64041]

\n

Thank you for sharing. I tried to build from source code, however I got this error:

\n

error: Setup script exited with error: library dfftpack has Fortran sources but no Fortran compiler found

\n

I use ubuntu 14.04 and libgfortran3 is installed. What else Fortran compiler do I need ? thank you.

\n

[/quote]

\n

Seems like it's one of SciPy dependencies.  Can you build it again after installing SciPy's dependencies?

\n

$ sudo apt-get install python-pip python-dev build-essential libatlas-base-dev gfortran

\n

Installing only gfortran might be sufficient.

\n

Please let me know if you find other dependency issues.

" }, { "id": 64045, "postDate": "2015-02-11T17:13:13.860Z", "content": "

[quote=Mikhail Trofimov;64020]

\n

Great!

\n

@Jeong-Yoon Lee, did you use this code in this contest? 

\n

[/quote]

\n

Yes. With one-hot-encoding (cutoff at 100) on original + weekday + hour features, FTRL, FM, NN scored around 0.395 on public leaderboard.

\n

Our team's final submission was FTRL + FM + NN + 3 idiots' FM + VW, and it scored 0.390 and 0.388 on the public and private leaderboard respectively.

" }, { "id": 64041, "postDate": "2015-02-11T15:43:05.837Z", "content": "

Thank you for sharing. I tried to build from source code, however I got this error:

\n

error: Setup script exited with error: library dfftpack has Fortran sources but no Fortran compiler found

\n

I use ubuntu 14.04 and libgfortran3 is installed. What else Fortran compiler do I need ? thank you.

" }, { "id": 64040, "postDate": "2015-02-11T15:37:06.080Z", "content": "

Thanks! for sharing.

" }, { "id": 64020, "postDate": "2015-02-11T12:39:54.497Z", "content": "

Great!

\n

@Jeong-Yoon Lee, did you use this code in this contest? 

" }, { "id": 64018, "postDate": "2015-02-11T11:32:47.177Z", "content": "

Thanks for sharing.

" }, { "id": 3187425, "postDate": "2025-04-26T02:23:54.297Z", "content": "

Thanks a lot, for your sharing!

", "rawMarkdown": "Thanks a lot, for your sharing!", "votes": 2 }, { "id": 2659723, "postDate": "2024-02-20T05:24:44.820Z", "content": "

Thanks for sharing.

", "rawMarkdown": "Thanks for sharing." }, { "id": 2049811, "postDate": "2022-11-30T11:01:19.610Z", "content": "

Thanks for sharing.

", "rawMarkdown": "Thanks for sharing." }, { "id": 3224128, "postDate": "2025-06-14T12:50:43.103Z", "rawMarkdown": "", "votes": 4, "isDeleted": true } ], "comments": [ { "id": 365296, "author_name": "qwerty", "author_url": "", "post_date": "2018-08-02T09:35:14.450000", "content": "

Thanks for sharing.\nwhen i use clf.fit(x_train,y_train)\ni get a problem:ValueError: Buffer dtype mismatch, expected 'int' but got 'long'\nis this can only run on 32bit mechine?

", "votes": 1, "replies": [] }, { "id": 64090, "author_name": "Charles", "author_url": "", "post_date": "2015-02-12T03:59:50.233000", "content": "

Thanks Jeong-Yoon Lee for sharing your code.

\n

I tried using "pip install Kaggler" and got the following error:

\n

IOError: [Errno 2] No such file or directory: 'README.md'

\n

It seems like you forgot to include this file onto the PyPI repository.

\n\n\n

EDIT: Oops! Someone just beat me to raising the question. I did not notice that when posting...

", "votes": 1, "replies": [] }, { "id": 64086, "author_name": "CodingNeo", "author_url": "", "post_date": "2015-02-12T03:26:32.707000", "content": "

Thanks for sharing!

\n

I encounter the installation error when I try "pip install Kaggler"

\n

IOError: [Errno 2] No such file or directory: 'README.md'

\n

Any hint to resolve it?

", "votes": 1, "replies": [] }, { "id": 64043, "author_name": "Abhishek Thakur", "author_url": "", "post_date": "2015-02-11T15:45:55.973000", "content": "

do you have the dev version of libgfortran3?

", "votes": 1, "replies": [] }, { "id": 64315, "author_name": "jeong", "author_url": "", "post_date": "2015-02-16T05:26:49.273000", "content": "

[quote=CodingNeo;64304]

\n

@Jeong-Yoon Lee, I fixed the issue which is caused by the wrong value of input parameters. I am trying the factorization machine in this package using the train data of this competition. I did the one hot encoding by hash trick as the input data. The factorization machine give me very bad result. I wonder something I did wrong. So consult you with two following question:

\n
    \n
  1. I simply convert the data by one-hot encoding before sending to FM. Is it correct?
    2. \n
  2. Can you share the parameter values you used for FM for this problem?
    3. \n
\n

[/quote]

\n

I did as follows:

\n

1. Extracting the weekday and hour-of-day variables from the hour column

\n

2. One-hot-encoding all columns with the cut-off of 100 (unique values of which frequency are less than 100 are combined into one category)

\n

3. Random shuffling input stream with the shuffling window of 1,000,000 - which is not part of the FM class yet, but there is a function, shuf_file() in kaggler.io you can take a look.

\n

4. Parameters used are: FM(n=60000, dim=16, a=0.02) - n was small because after one-hot-encoding the number of features was less than 60K.

\n

5. Then iterate with training data 10 times.

\n

This scored 0.3978 and 0.3959 on the public and private leaderboards respectively.  It was my only submission using kaggler.online_model.FM.

", "votes": 2, "replies": [] }, { "id": 64451, "author_name": "CodingNeo", "author_url": "", "post_date": "2015-02-18T01:23:49.397000", "content": "

[quote=Jeong-Yoon Lee;64347]

\n

[quote=CodingNeo;64343]

\n

[quote=Jeong-Yoon Lee;64341]

\n

[quote=CodingNeo;64319]

\n

Thanks Jeong-Yoon Lee. You share with me a lot of information. One more question to ask you for help. How can I get the probability of the prediction instead of 0/1 prediction? I need the probability to compute the log loss.

\n

[/quote]

\n

It always outputs the probability.

\n

[/quote]

\n

I got all prediction with the values of 1.00000/0.00000

\n

[/quote]

\n

It seems that the learning rate is too high and makes the output of the sigmoid function (1/(1 + exp(-x))) saturated

\n

[/quote]

\n\n

Thank you very much! Reducing the learning rate resolves the issue.

", "votes": 0, "replies": [] }, { "id": 64347, "author_name": "jeong", "author_url": "", "post_date": "2015-02-16T15:59:15.597000", "content": "

[quote=CodingNeo;64343]

\n

[quote=Jeong-Yoon Lee;64341]

\n

[quote=CodingNeo;64319]

\n

Thanks Jeong-Yoon Lee. You share with me a lot of information. One more question to ask you for help. How can I get the probability of the prediction instead of 0/1 prediction? I need the probability to compute the log loss.

\n

[/quote]

\n

It always outputs the probability.

\n

[/quote]

\n

I got all prediction with the values of 1.00000/0.00000

\n

[/quote]

\n

It seems that the learning rate is too high and makes the output of the sigmoid function (1/(1 + exp(-x))) saturated

", "votes": 0, "replies": [] }, { "id": 64343, "author_name": "CodingNeo", "author_url": "", "post_date": "2015-02-16T15:05:22.110000", "content": "

[quote=Jeong-Yoon Lee;64341]

\n

[quote=CodingNeo;64319]

\n

Thanks Jeong-Yoon Lee. You share with me a lot of information. One more question to ask you for help. How can I get the probability of the prediction instead of 0/1 prediction? I need the probability to compute the log loss.

\n

[/quote]

\n

It always outputs the probability.

\n

[/quote]

\n

I got all prediction with the values of 1.00000/0.00000

", "votes": 0, "replies": [] }, { "id": 64342, "author_name": "CodingNeo", "author_url": "", "post_date": "2015-02-16T15:04:43.153000", "content": "

I got all 1.00000/0.00000 in prediction with the parameters (100000, 4, 1.0). Why?

", "votes": 0, "replies": [] }, { "id": 64341, "author_name": "jeong", "author_url": "", "post_date": "2015-02-16T14:57:44.287000", "content": "

[quote=CodingNeo;64319]

\n

Thanks Jeong-Yoon Lee. You share with me a lot of information. One more question to ask you for help. How can I get the probability of the prediction instead of 0/1 prediction? I need the probability to compute the log loss.

\n

[/quote]

\n

It always outputs the probability.

", "votes": 0, "replies": [] }, { "id": 64332, "author_name": "Jiming Ye", "author_url": "", "post_date": "2015-02-16T10:17:37.923000", "content": "

[quote=Jeong-Yoon Lee;64270]

\n

[quote=Jiming Ye;64266]

\n

 I tried to learn how to implement those online algorithm from your code, and I have a question.

\n

In your FM model and NN, you have put counters in. And you use them like:

\n

w1[h] -= dl_dy / (sqrt(c) + 1) + l2 * w1[h]

\n

I haven't seen those things before. Actually when I implemented FM and NN, I didn't use them(Maybe that's the reason my FM model performs badly)

\n

Can you explain a little bit about these counters? I know this question might be somewhat dumb, but I just cannot figure it out by myself :(

\n

Thanks!

\n

[/quote]

\n

It's again from tinrtgu's code based on Google's FTRL paper at the Criteo competition.  Actually 1 / (sqrt(c) + 1) is not what was used in the paper, but - I guess - his approximation of 1 / (sqrt(sum(g * g)) + 1).

\n

In later updates, I will try 1/(sqrt(sum(g*g)) + 1).

\n

EDIT:

\n

1 / sqrt(n) is common in stochastic gradient learning, and it is called per-coordinate learning rate if the i-th weight, w[i] is updated using c[i] only if it's associated with non-zero input x[i].

\n

To do so, 1 / (sqrt(c[i]) + 1) is straightforward extension of 1 / sqrt(n).  However, as in the paper, 1 / (sqrt(sum(g[i] * g[i])) + 1) is a better choice, where g[i] is per-coordinate gradient for the i-th weight.

\n

[/quote]

\n

Thank you so much!

\n

Now I understand :)

", "votes": 0, "replies": [] }, { "id": 64319, "author_name": "CodingNeo", "author_url": "", "post_date": "2015-02-16T06:33:08.107000", "content": "

Thanks Jeong-Yoon Lee. You share with me a lot of information. One more question to ask you for help. How can I get the probability of the prediction instead of 0/1 prediction? I need the probability to compute the log loss.

", "votes": 0, "replies": [] }, { "id": 64304, "author_name": "CodingNeo", "author_url": "", "post_date": "2015-02-16T02:36:37.163000", "content": "

@Jeong-Yoon Lee, I fixed the issue which is caused by the wrong value of input parameters. I am trying the factorization machine in this package using the train data of this competition. I did the one hot encoding by hash trick as the input data. The factorization machine give me very bad result. I wonder something I did wrong. So consult you with two following question:

\n
    \n
  1. I simply convert the data by one-hot encoding before sending to FM. Is it correct?
    2. \n
  2. Can you share the parameter values you used for FM for this problem?
    3. \n
", "votes": 0, "replies": [] }, { "id": 64270, "author_name": "jeong", "author_url": "", "post_date": "2015-02-15T14:35:45.517000", "content": "

[quote=Jiming Ye;64266]

\n

 I tried to learn how to implement those online algorithm from your code, and I have a question.

\n

In your FM model and NN, you have put counters in. And you use them like:

\n

w1[h] -= dl_dy / (sqrt(c) + 1) + l2 * w1[h]

\n

I haven't seen those things before. Actually when I implemented FM and NN, I didn't use them(Maybe that's the reason my FM model performs badly)

\n

Can you explain a little bit about these counters? I know this question might be somewhat dumb, but I just cannot figure it out by myself :(

\n

Thanks!

\n

[/quote]

\n

It's again from tinrtgu's code based on Google's FTRL paper at the Criteo competition.  Actually 1 / (sqrt(c) + 1) is not what was used in the paper, but - I guess - his approximation of 1 / (sqrt(sum(g * g)) + 1).

\n

In later updates, I will try 1/(sqrt(sum(g*g)) + 1).

\n

EDIT:

\n

1 / sqrt(n) is common in stochastic gradient learning, and it is called per-coordinate learning rate if the i-th weight, w[i] is updated using c[i] only if it's associated with non-zero input x[i].

\n

To do so, 1 / (sqrt(c[i]) + 1) is straightforward extension of 1 / sqrt(n).  However, as in the paper, 1 / (sqrt(sum(g[i] * g[i])) + 1) is a better choice, where g[i] is per-coordinate gradient for the i-th weight.

", "votes": 0, "replies": [] }, { "id": 64269, "author_name": "jeong", "author_url": "", "post_date": "2015-02-15T14:05:55.100000", "content": "

[quote=CodingNeo;64214]

\n

I try to convert the train/test data into sparse format for using this package. But I got an zero division error in fm.read_sparse function. Any hints to resolve this issue?

\n

[/quote]

\n

I just tested SGD, FM, FTRL, NN and all worked fine on my side.  Can you provide with more details that I can reproduce the error?

\n

Also, since I've updated the package quite a bit, please try the latest version.

", "votes": 0, "replies": [] }, { "id": 64266, "author_name": "Jiming Ye", "author_url": "", "post_date": "2015-02-15T12:37:21.313000", "content": "

 I tried to learn how to implement those online algorithm from your code, and I have a question.

\n

In your FM model and NN, you have put counters in. And you use them like:

\n

w1[h] -= dl_dy / (sqrt(c) + 1) + l2 * w1[h]

\n

I haven't seen those things before. Actually when I implemented FM and NN, I didn't use them(Maybe that's the reason my FM model performs badly)

\n

Can you explain a little bit about these counters? I know this question might be somewhat dumb, but I just cannot figure it out by myself :(

\n

Thanks!

", "votes": 0, "replies": [] }, { "id": 64214, "author_name": "CodingNeo", "author_url": "", "post_date": "2015-02-14T04:25:53.340000", "content": "

I try to convert the train/test data into sparse format for using this package. But I got an zero division error in fm.read_sparse function. Any hints to resolve this issue?

", "votes": 0, "replies": [] }, { "id": 64131, "author_name": "jeong", "author_url": "", "post_date": "2015-02-12T17:57:20.103000", "content": "

[quote=binga;64099]

\n

[quote=Charles;64090]

\n

Thanks Jeong-Yoon Lee for sharing your code.

\n

I tried using "pip install Kaggler" and got the following error:

\n

IOError: [Errno 2] No such file or directory: 'README.md'

\n

It seems like you forgot to include this file onto the PyPI repository.

\n

EDIT: Oops! Someone just beat me to raising the question. I did not notice that when posting...

\n

[/quote]

\n

I got the same error by installing through pip. However, by downloading the source and running

\n

python setup.py install

\n

everything went smooth.

\n

[/quote]

\n

Hi @binga.  Sorry for the error.

\n

I noticed that "pip install" without "-U" will fail if there was a previous unsuccessful installation attempt.  

\n

I tried "pip install -U Kaggler" on Mac and Ubuntu, and was able to install the package successfully.

\n

Can anyone else confirm either successful or unsuccessful installation by using pip?

\n

Thanks!

", "votes": 0, "replies": [] }, { "id": 64105, "author_name": "binga", "author_url": "", "post_date": "2015-02-12T10:16:29.433000", "content": "

[quote=Charles;64104]

\n

python setup.py build_ext --inplace

\n

[/quote]

\n

Oh, yes. You're right! Same here.

", "votes": 0, "replies": [] }, { "id": 64104, "author_name": "Charles", "author_url": "", "post_date": "2015-02-12T10:11:46.160000", "content": "

[quote=binga;64099]

\n

[quote=Charles;64090]

\n

Thanks Jeong-Yoon Lee for sharing your code.

\n

I tried using "pip install Kaggler" and got the following error:

\n

IOError: [Errno 2] No such file or directory: 'README.md'

\n

It seems like you forgot to include this file onto the PyPI repository.

\n

EDIT: Oops! Someone just beat me to raising the question. I did not notice that when posting...

\n

[/quote]

\n

I got the same error by installing through pip. However, by downloading the source and running

\n

python setup.py install

\n

everything went smooth.

\n

[/quote]

\n

I ended up also downloading the source and building from source. But I think you'd need to first run (as documented on the README page) 

\n

python setup.py build_ext --inplace

\n

before running

\n

python setup.py install

\n

for the underlying C code to compile correctly. At least for me, the installation would not work without running the first command...

", "votes": 0, "replies": [] }, { "id": 64099, "author_name": "binga", "author_url": "", "post_date": "2015-02-12T07:38:25.153000", "content": "

[quote=Charles;64090]

\n

Thanks Jeong-Yoon Lee for sharing your code.

\n

I tried using "pip install Kaggler" and got the following error:

\n

IOError: [Errno 2] No such file or directory: 'README.md'

\n

It seems like you forgot to include this file onto the PyPI repository.

\n

EDIT: Oops! Someone just beat me to raising the question. I did not notice that when posting...

\n

[/quote]

\n

I got the same error by installing through pip. However, by downloading the source and running

\n

python setup.py install

\n

everything went smooth.

", "votes": 0, "replies": [] }, { "id": 64092, "author_name": "jeong", "author_url": "", "post_date": "2015-02-12T05:24:15.263000", "content": "

Thanks @CodingNeo and @Charles for the issue report, and sorry for the error.

\n

I fixed it and uploaded a new version - it must work now.  Please try:

\n

$ pip install -U Kaggler

\n\n

I just learned a couple of things about Python packaging by sharing it with you guys:

\n\n

Sharing is learning. So awesome! :D

", "votes": 0, "replies": [] }, { "id": 64085, "author_name": "jeong", "author_url": "", "post_date": "2015-02-12T03:13:54.520000", "content": "

[quote=Jiming Ye;64081]

\n

Thanks for sharing!

\n

I didn't see any L1 or L2 term in the FM model. I don't know whether this is how original FM works

\n

[/quote]

\n

Hi Jiming, Yes, in the original FM paper by Steffen, there was neither L1 nor L2 term.  However since it's trivial, I'll consider adding them to the model soon. :)

", "votes": 0, "replies": [] }, { "id": 64081, "author_name": "Jiming Ye", "author_url": "", "post_date": "2015-02-12T01:31:20.580000", "content": "

Thanks for sharing!

\n

I didn't see any L1 or L2 term in the FM model. I don't know whether this is how original FM works

", "votes": 0, "replies": [] }, { "id": 64063, "author_name": "TanoPereira", "author_url": "", "post_date": "2015-02-11T20:07:35.093000", "content": "

Thank you for sharing this!!

\n

It is a great addition to everybody's toolbox.

", "votes": 0, "replies": [] }, { "id": 64046, "author_name": "jeong", "author_url": "", "post_date": "2015-02-11T17:24:05.687000", "content": "

[quote=rcarson;64041]

\n

Thank you for sharing. I tried to build from source code, however I got this error:

\n

error: Setup script exited with error: library dfftpack has Fortran sources but no Fortran compiler found

\n

I use ubuntu 14.04 and libgfortran3 is installed. What else Fortran compiler do I need ? thank you.

\n

[/quote]

\n

Seems like it's one of SciPy dependencies.  Can you build it again after installing SciPy's dependencies?

\n

$ sudo apt-get install python-pip python-dev build-essential libatlas-base-dev gfortran

\n

Installing only gfortran might be sufficient.

\n

Please let me know if you find other dependency issues.

", "votes": 0, "replies": [] }, { "id": 64045, "author_name": "jeong", "author_url": "", "post_date": "2015-02-11T17:13:13.860000", "content": "

[quote=Mikhail Trofimov;64020]

\n

Great!

\n

@Jeong-Yoon Lee, did you use this code in this contest? 

\n

[/quote]

\n

Yes. With one-hot-encoding (cutoff at 100) on original + weekday + hour features, FTRL, FM, NN scored around 0.395 on public leaderboard.

\n

Our team's final submission was FTRL + FM + NN + 3 idiots' FM + VW, and it scored 0.390 and 0.388 on the public and private leaderboard respectively.

", "votes": 0, "replies": [] }, { "id": 64041, "author_name": "Jiwei Liu", "author_url": "", "post_date": "2015-02-11T15:43:05.837000", "content": "

Thank you for sharing. I tried to build from source code, however I got this error:

\n

error: Setup script exited with error: library dfftpack has Fortran sources but no Fortran compiler found

\n

I use ubuntu 14.04 and libgfortran3 is installed. What else Fortran compiler do I need ? thank you.

", "votes": 0, "replies": [] }, { "id": 64040, "author_name": "cup3", "author_url": "", "post_date": "2015-02-11T15:37:06.080000", "content": "

Thanks! for sharing.

", "votes": 0, "replies": [] }, { "id": 64020, "author_name": "", "author_url": "", "post_date": "2015-02-11T12:39:54.497000", "content": "", "votes": 0, "replies": [] }, { "id": 64018, "author_name": "", "author_url": "", "post_date": "2015-02-11T11:32:47.177000", "content": "", "votes": 0, "replies": [] }, { "id": 3187425, "author_name": "", "author_url": "", "post_date": "2025-04-26T02:23:54.297000", "content": "", "votes": 2, "replies": [] }, { "id": 2659723, "author_name": "", "author_url": "", "post_date": "2024-02-20T05:24:44.820000", "content": "", "votes": 0, "replies": [] }, { "id": 2049811, "author_name": "", "author_url": "", "post_date": "2022-11-30T11:01:19.610000", "content": "", "votes": 0, "replies": [] }, { "id": 3224128, "author_name": "", "author_url": "", "post_date": "2025-06-14T12:50:43.103000", "content": "", "votes": 4, "replies": [] } ], "raw_markdown_by_id": { "64009": "", "365296": "Thanks for sharing.\nwhen i use clf.fit(x_train,y_train)\ni get a problem:ValueError: Buffer dtype mismatch, expected 'int' but got 'long'\nis this can only run on 32bit mechine?", "64090": "", "64086": "", "64043": "", "64315": "", "64451": "", "64347": "", "64343": "", "64342": "", "64341": "", "64332": "", "64319": "", "64304": "", "64270": "", "64269": "", "64266": "", "64214": "", "64131": "", "64105": "", "64104": "", "64099": "", "64092": "", "64085": "", "64081": "", "64063": "", "64046": "", "64045": "", "64041": "", "64040": "", "64020": "", "64018": "", "3187425": "Thanks a lot, for your sharing!", "2659723": "Thanks for sharing.", "2049811": "Thanks for sharing.", "3224128": "" } }