Dear all,
\nWe have prepared our codes and documents.
\nFor the codes, please see here; and for the documents, please see here.
\nIn this competition and Avazu's CTR competition, we find field-aware factorization machines (FFM) are powerful models. We have implemented it in the package LIBFFM.
\nYour comments are very welcome. Thanks!
\n-- 3 Idiots
", "comments": [ { "id": 578934, "author_name": "kapil kumar", "author_url": "", "post_date": "2019-07-18T10:01:02.703000", "content": "Hey I have a little doubt, are the features belonging to same field mutually exclusive or can they occur together.\n for ex. can we have libffm data like 0 1:21:1 1:3:1 1:55:1 ... I tried looking it up online but couldn't find much info. Thanks for help.
", "votes": 1, "replies": [ { "id": 868746, "author_name": "alimeituan", "author_url": "", "post_date": "2020-05-31T13:07:17.803000", "content": "hi,\nI have the same question,\nI just want to some help to transform the CSV to FFM data format ,\ncould you pls help?\nthx
", "votes": 0, "replies": [] } ] }, { "id": 55888, "author_name": "Yuchin Juan", "author_url": "", "post_date": "2014-10-10T16:17:20.997000", "content": "Tianqi:
\nBecause I do not know how other people used FM, sorry I have no idea about the
"difference". I guess the following three settings might be helpful:
1. Transforming infrequent features into a special tag. Conceptually, infrequent
features should only include very little or even no information, so it
should be very hard for a model to extract those information. In fact, these
features can be very noisy. We got significant improvement (more than 0.0005)
by transforming these features into a special tag.
2. Transforming numerical features (I1-I13) to categorical features.
Empirically we observe using categorical features is always better than
using numerical features. However, too many features are generated if
numerical features are directly transformed into categorical features, so we
use
v <- floor(log(v)^2)
\nto reduce the number of features generated.
\n3. Data normalization. According to our empirical experience, instance-wise
data normalization makes the optimization problem easier to be solved.
For example, for the following vector x
(0, 0, 1, 1, 0, 1, 0, 1, 0),
\nthe standard data normalization divides each element of x by by the 2-norm
of x. The normalized x becomes
(0, 0, 0.5, 0.5, 0, 0.5, 0, 0.5, 0).
\nThe hashing trick do not contribute any improvement on the leaderboard. We
apply the hashing trick only because it makes our life easier to generate
features. :)
Amazing solution!! Some quick questions:
\n(1) What is the impact of using field aware FM and the global latent factor version?
\n(2) By LR and FM, do you use features generated by GBDT? Have you measure the impact of the GBDT generated feature?
", "votes": 9, "replies": [] }, { "id": 59345, "author_name": "jwjohnson314", "author_url": "", "post_date": "2014-12-01T17:13:32.243000", "content": "Very nice solution. I'm curious - how did you determine the average CTR's on the public/private leaderboards (slide 16)?
", "votes": 5, "replies": [] }, { "id": 182484, "author_name": "LinLongzhan", "author_url": "", "post_date": "2017-05-14T04:16:37.237000", "content": "我觉得ffm是个很好的主意
", "votes": 0, "replies": [] }, { "id": 65763, "author_name": "Yuchin Juan", "author_url": "", "post_date": "2015-03-09T08:25:50.893000", "content": "[quote=jwjohnson314;59345]
\nVery nice solution. I'm curious - how did you determine the average CTR's on the public/private leaderboards (slide 16)?
\n[/quote]
\nSuppose we have an impression whose label is 1. If the prediction for this
impression is 0, then we should get an infinite logloss. In practice this is
not desired, so on the submission system there is a ceiling (C) for logloss.
If the logloss for an impression is greater than this ceiling, then it will be
truncated. Note that we do not know the value of this ceiling.
Using this feature, we can hack the average CTR by two submissions.
\nThe first submission contains all ones, so the logloss we get is P1 =
(nr_non_click*C)/nr_instance.
The second submission contains all zeros, so the logloss we get is P2 =
(nr_click*C)/nr_instance.
We can then get the average CTR = P2/(P1+P2).
", "votes": 4, "replies": [] }, { "id": 56827, "author_name": "Yuchin Juan", "author_url": "", "post_date": "2014-10-27T05:59:10.610000", "content": "[quote=Torgos;56807]
\nThank you for posting your solution, but the linked explanation pdf currently returns the error "You don't have permission to access /~r01922136/kaggle-2014-criteo.pdf on this server."
\n[/quote]
\nIt should be back now. Please try again. Thanks.
", "votes": 3, "replies": [] }, { "id": 55932, "author_name": "Yuchin Juan", "author_url": "", "post_date": "2014-10-11T16:04:12.633000", "content": "You are very welcome!
\n(2) They are similar but not the same. The FM model in the paper you provided
is field unaware. The difference between equation 1 in the paper and the
formula on page 14 of our slide is that our w is not only indexed by j1 and
j2, but also indexed by f1 and f2. Consider the example on page 15, if
Rendle's FM is applied, it becomes:
w376^Tw248x376x248 + w376^Tw571x376x571 + w376^Tw942x376x942
+ w248^Tw571x248x571 + w248^Tw942x248x942
+ w571^Tw942x571x942
BTW, we use k = 4, not k = 2.
\nPlease let me know if you have more questions. :)
\n[quote=Inspector;55906]
\n1) I think this helped a lot. I was confused what the hashing trick was doing. I was thinking perhaps the value of a feature, say 5a9ed9b0, was REPLACED by an integer. So, I understand now that one-hot-encoding is still being used, it is just the indexing of the data which is improved (memory wise) when hashed.
\n2) So this is essentially the same model as shown in the Rendle paper http://www.ismll.uni-hildesheim.de/pub/pdfs/Rendle2010FM.pdf (equation 1) , where you used k=2 (the number of factors)? Is this correct?
\nThanks very much!!
\n[/quote]
", "votes": 1, "replies": [] }, { "id": 55898, "author_name": "Yuchin Juan", "author_url": "", "post_date": "2014-10-10T19:21:13.120000", "content": "(1) Actually the fact is completely contrary to what you mentioned. We one-hot
encode everything. The "integer" you mentioned is actually the index of a
feature. For example, consider the feature "C1:5a9ed9b0", and it is hashed
to an integer, say, 872342. It does not mean that a feature's value is
872342; it actually means that the value of 872342nd feature is non-zero.
(2) Consider the example in the 15th page of the slide. x is an instance of
training data, and w is the model. Because x is very sparse, we use sparse
format to represent it. That is, only the 376th, the 248th, the 571st, and
the 942nd features are non-zero. phi(w,x) can be obtained as the slide
shows. So, for your question, x_j1, x_j2, w_j1,f2, and w_j2,f1 are symbols
used to represent the corresponding w and x in each term. For example, in
the first term, x_j1 is x_376, x_j2 is x_248, w_j1,f2 is w_376,2, and
w_j2,f1 is w_248,1.
Not sure if you think the questions are answered. Please let me know if it is
not clear. :)
[quote=Inspector;55862]
\nFor those of us (me ;-)) just starting to learn about FM, hope you don't mind a couple elementary questions.
\n1) Are you only using numeric variables in the model - no one-hot encoding at all? Everything is hashed to an integer?
\n
2) On Slide #14-15 I am unsure what {x_j1, x_j2 } and {wj1,f2, wj2,f1} are. Would you explain?
[/quote]
", "votes": 1, "replies": [] }, { "id": 55830, "author_name": "Yuchin Juan", "author_url": "", "post_date": "2014-10-09T10:49:35.607000", "content": "Tianqi:
\n(1) We have not submitted one using the global latent factor version. On our validation set, the difference was around 0.001. (Both of them did not use GBDT features.)
\n(2) We compare them in the last page of our slide. "FM+GBDT" and "FM" respectively represent training FM with/without adding the GBDT features. We can see the around 0.001 improvement by adding the GBDT features.
", "votes": 1, "replies": [] }, { "id": 56807, "author_name": "Torgos", "author_url": "", "post_date": "2014-10-26T17:36:41.653000", "content": "Thank you for posting your solution, but the linked explanation pdf currently returns the error "You don't have permission to access /~r01922136/kaggle-2014-criteo.pdf on this server."
", "votes": 2, "replies": [] }, { "id": 55966, "author_name": "Tianqi Chen", "author_url": "", "post_date": "2014-10-12T16:15:07.360000", "content": "Having to do early stopping seems to be a very common thing that can not be avoided in factorization model. Sometimes using an alternative regularization(e.g max-norm on latent factor) can make the long tail effect comes slower, however it will still come, and the best solution is usually worse than what you can have in optimal iteration.
\nAmong the factorization models, bayesian version is stable to long-tail effect, however they are not practical(at least not the plain version) because you have to store all the generated samples for prediction. I have no idea how this can be avoided easily:) Maybe using other stable model to calibrate what is generated from FM models...
\nThe regularization you use appears to be the common one used by MF folks. Since lambda is applied only when nonzero feature occurs. It is not standard L2 regularizer. It is a weighed L2 regularization where weight on feature i is lambda times number of nonzero feature i
\nThanks for answering so many questions! I learned a lot from your solution
\n[quote=guestwalk;55952]
\nFor the questions about the sensitivity and the situation after the best
iteration, please see the attachment. (tr_loss and va_loss respectively
represent the logloss on the training and the validation set.) I do this on our
own validation set; also, I removed GBDT features so that the experiments can
run faster.
We can see the setting eta = 0.1 indeed achieve the best performance on the
validation set. For eta = 0.2, eta = 0.05, and eta = 0.02, a similar
performance can be achieved. The smaller eta is, the more iterations are
required to achieve a good performance; it is the classic SGD behavior. If eta
is too large (in this case, 0.5), then the result is significantly worse; for
the setting eta = 0.01, it do not converge at 100th iterations. My opinion is
choosing a good eta is still crucial to SGD. For a large eta, the accuracy can
be hurt; for a small eta, it takes a long time to converge.
On the other hand, the "long tail" iterations indeed hurts a lot. In fact, we
also used this learning rate schedule to solve the simple matrix factorization
(MF) problem, and the performance in the long tail iterations is quite stable
(i.e., close to the best performance). I think the difference may be that FM is
much complicated compared to MF, so the overfitting problem is much harder to
be avoided. (In some sense this FM problem is a combination of 39*38/2 MF
problem, given there are 39 features for each impression.) This problem is
really annoying because it makes our solution less useful in practice. Any
suggestion?
For regularization, our implementation is very simple. Every time we choose an
instance, we only update the corresponding coordinate of the non-zero features.
For example, if the non-zero coordinates of an instance x are 376, 248, 571,
and 942, then only w376, w248, w571, and w942 are updated. Take w248 as an
example, the corresponding gradient is
g248 = lambda*w248 + (the gradient of the loss term)
\nNote that lambda is a fixed parameter, we don't change it during training at
all. Not sure if it is one of the approaches you mentioned.
Finally, the gradient of regularization is also accumulated.
\n[quote=Tianqi Chen;55931]
\nInteresting! This is the first time I see AdaGrad to work on factorization models.
\nSometimes it works great for convex models and some neuralnets. But I wasn't able get it work better than constant learning rate or decayed learning rate on factorization models.
\nHow sensitive is the result to the initial learning rate, I suspect it is just for lower rate, you run for longer rounds, but still get similar results?
\nI get your point about choosing the stopping condition. Since you use AdaGrad, I guess the raising up period after optimal point will be more smoother? Sometimes Adagrad in convex models tends to less overfit, say 50 rounds gets you there, and running for 70 rounds won't hurt too much. I am wondering what is in your case.
\nFor L2 regularizer, how do you implement it? If it is standard L2 regularizer, you don;t want to regularize parameters every rounds, so either some lazy trick or divide the regularizer by #occurance can be used. In many MF codes, sometimes ppl simply take another regularizer lambda #occraunce_of_fi w_i^2 and run weight decay every time you see the parameter.
\nDoes the gradient of regularizer goes into the accumulated gradient?
\nTianqi
\n[/quote]
\n[/quote]
", "votes": 2, "replies": [] }, { "id": 55931, "author_name": "Tianqi Chen", "author_url": "", "post_date": "2014-10-11T16:02:50.270000", "content": "Interesting! This is the first time I see AdaGrad to work on factorization models.
\nSometimes it works great for convex models and some neuralnets. But I wasn't able get it work better than constant learning rate or decayed learning rate on factorization models.
\nHow sensitive is the result to the initial learning rate, I suspect it is just for lower rate, you run for longer rounds, but still get similar results?
\nI get your point about choosing the stopping condition. Since you use AdaGrad, I guess the raising up period after optimal point will be more smoother? Sometimes Adagrad in convex models tends to less overfit, say 50 rounds gets you there, and running for 70 rounds won't hurt too much. I am wondering what is in your case.
\nFor L2 regularizer, how do you implement it? If it is standard L2 regularizer, you don;t want to regularize parameters every rounds, so either some lazy trick or divide the regularizer by #occurance can be used. In many MF codes, sometimes ppl simply take another regularizer lambda #occraunce_of_fi w_i^2 and run weight decay every time you see the parameter.
\nDoes the gradient of regularizer goes into the accumulated gradient?
\nTianqi
\n[quote=guestwalk;55930]
\nYou are very welcome!
\nHere are my answers:
\n(1) For the weight decay of the learning rate (step size), we use
per-coordinate learning rate schedule. For a specific coordinate, the
update rule is:
G = G + g*g
w = w - eta*(1/sqrt(G))*g,
where g is the gradient of that coordinate, G records the accumulated
squared sum of the gradients in the past updates, eta is the initial
learning rate, and w is the model. Note that the initial value of G is set
to 1.
This learning rate schedule has been studied in, for example, this paper.
Because we know this learning rate schedule before the competition, we did
not spend any time tuning.
(2) For the value of initial learning rate (eta), we tried 0.5, 0.2, 0.1, 0.05,
0.02, and 0.01 at the early stage we used FM. Then we found that 0.1 is a
good balance between the performance and the training time, so we just
fixed it and had never changed it.
Because at that time we haven't included GBDT features, FM ran about three
times faster compared to the current setting. These experiments took us
around 6 hours.
(3) For the stopping criterion, we did not find a good one. What we did was:
\n1. Run experiments on our validation set, and record the number of
iterations to achieve the best performance.
2. Use this number of iterations to train the model on the whole
training data.
So for each of our submissions, we run experiments at least two times: one
on validation set and the other on the whole data set.
(4) There is another parameter need to be tuned: lambda (regularization
penalty). Like other machine learning models such as SVM or standard LR,
this parameter need to be tuned. We ran experiments very similar to what we
mentioned in tuning eta. The time cost of tuning this parameter was also
around 6 hours.
Please feel free to let me know if you have more questions. :)
\n[quote=Tianqi Chen;55890]
\nThanks for detailed answer!
\nI like your comment on the data normalization trick. This is also what I observed repeatly in experimenting with factorization models. I think this is mainly because the MF models applies L2 projection and L2 inner product, so keeping the features' L2 norm constant makes different product roughly in the same scale without introducing instance bias.
\nHere is another question:) I took a look over your solver and seems it is SGD based solver, so I assume you need to tune the step size and weight decay and do early stopping on validation set? How does the three factors affect your experiment? I guess usually different step size and weight decay could lead to slightly different stoping round. How much effort do you spent on tuning them?
\nTianqi
\n[/quote]
\n[/quote]
", "votes": 2, "replies": [] }, { "id": 55930, "author_name": "Yuchin Juan", "author_url": "", "post_date": "2014-10-11T15:34:23.923000", "content": "You are very welcome!
\nHere are my answers:
\n(1) For the weight decay of the learning rate (step size), we use
per-coordinate learning rate schedule. For a specific coordinate, the
update rule is:
G = G + g*g
w = w - eta*(1/sqrt(G))*g,
where g is the gradient of that coordinate, G records the accumulated
squared sum of the gradients in the past updates, eta is the initial
learning rate, and w is the model. Note that the initial value of G is set
to 1.
This learning rate schedule has been studied in, for example, this paper.
Because we know this learning rate schedule before the competition, we did
not spend any time tuning.
(2) For the value of initial learning rate (eta), we tried 0.5, 0.2, 0.1, 0.05,
0.02, and 0.01 at the early stage we used FM. Then we found that 0.1 is a
good balance between the performance and the training time, so we just
fixed it and had never changed it.
Because at that time we haven't included GBDT features, FM ran about three
times faster compared to the current setting. These experiments took us
around 6 hours.
(3) For the stopping criterion, we did not find a good one. What we did was:
\n1. Run experiments on our validation set, and record the number of
iterations to achieve the best performance.
2. Use this number of iterations to train the model on the whole
training data.
So for each of our submissions, we run experiments at least two times: one
on validation set and the other on the whole data set.
(4) There is another parameter need to be tuned: lambda (regularization
penalty). Like other machine learning models such as SVM or standard LR,
this parameter need to be tuned. We ran experiments very similar to what we
mentioned in tuning eta. The time cost of tuning this parameter was also
around 6 hours.
Please feel free to let me know if you have more questions. :)
\n[quote=Tianqi Chen;55890]
\nThanks for detailed answer!
\nI like your comment on the data normalization trick. This is also what I observed repeatly in experimenting with factorization models. I think this is mainly because the MF models applies L2 projection and L2 inner product, so keeping the features' L2 norm constant makes different product roughly in the same scale without introducing instance bias.
\nHere is another question:) I took a look over your solver and seems it is SGD based solver, so I assume you need to tune the step size and weight decay and do early stopping on validation set? How does the three factors affect your experiment? I guess usually different step size and weight decay could lead to slightly different stoping round. How much effort do you spent on tuning them?
\nTianqi
\n[/quote]
", "votes": 2, "replies": [] }, { "id": 55890, "author_name": "Tianqi Chen", "author_url": "", "post_date": "2014-10-10T16:35:31.920000", "content": "Thanks for detailed answer!
\nI like your comment on the data normalization trick. This is also what I observed repeatly in experimenting with factorization models. I think this is mainly because the MF models applies L2 projection and L2 inner product, so keeping the features' L2 norm constant makes different product roughly in the same scale without introducing instance bias.
\nHere is another question:) I took a look over your solver and seems it is SGD based solver, so I assume you need to tune the step size and weight decay and do early stopping on validation set? How does the three factors affect your experiment? I guess usually different step size and weight decay could lead to slightly different stoping round. How much effort do you spent on tuning them?
\nTianqi
\n[quote=guestwalk;55888]
\nTianqi:
\nBecause I do not know how other people used FM, sorry I have no idea about the
"difference". I guess the following three settings might be helpful:
1. Transforming infrequent features into a special tag. Conceptually, infrequent
features should only include very little or even no information, so it
should be very hard for a model to extract those information. In fact, these
features can be very noisy. We got significant improvement (more than 0.0005)
by transforming these features into a special tag.
2. Transforming numerical features (I1-I13) to categorical features.
Empirically we observe using categorical features is always better than
using numerical features. However, too many features are generated if
numerical features are directly transformed into categorical features, so we
use
v <- floor(log(v)^2)
\nto reduce the number of features generated.
\n3. Data normalization. According to our empirical experience, instance-wise
data normalization makes the optimization problem easier to be solved.
For example, for the following vector x
(0, 0, 1, 1, 0, 1, 0, 1, 0),
\nthe standard data normalization divides each element of x by by the 2-norm
of x. The normalized x becomes
(0, 0, 0.5, 0.5, 0, 0.5, 0, 0.5, 0).
\nThe hashing trick do not contribute any improvement on the leaderboard. We
apply the hashing trick only because it makes our life easier to generate
features. :)
[/quote]
", "votes": 2, "replies": [] }, { "id": 55851, "author_name": "Tianqi Chen", "author_url": "", "post_date": "2014-10-09T17:11:34.677000", "content": "Thanks for the clarification!! Since a lot of people used FM and I do not think they get to the performance as you do. Can you comment on which part of your method(apart from GBDT features) brings you the difference?
\nAs far as I can list, this could be parameter tuning, or the Prep-B step you introduced a) changing the less frequent occurred instance to a single tag; b) Hashing trick to limit all the thing to 1M categories; c) The integer remapping techniques
\n\n[quote=guestwalk;55830]
\nTianqi:
\n(1) We have not submitted one using the global latent factor version. On our validation set, the difference was around 0.001. (Both of them did not use GBDT features.)
\n(2) We compare them in the last page of our slide. "FM+GBDT" and "FM" respectively represent training FM with/without adding the GBDT features. We can see the around 0.001 improvement by adding the GBDT features.
\n[/quote]
", "votes": 2, "replies": [] }, { "id": 3194949, "author_name": "xj", "author_url": "", "post_date": "2025-05-06T13:13:28.103000", "content": "@guestwalk The link to this document is broken, please do you still have a backup?
", "votes": 0, "replies": [] }, { "id": 336386, "author_name": "Todd Neller", "author_url": "", "post_date": "2018-05-31T15:56:42.503000", "content": "Thank you all for sharing your code and documenting your experiences. What an excellent education for us all!
\n\nIn your slides, you note that version2 (v2) has \"50 trees and 8 latent factors\", i.e. the GBDT stage uses 50 trees (compared to 30 in v1), and the FFM stage uses 8 latent factors. Questions:
\n\nHi ! First of all, congrat for the solution,
\nin your solution (page 13), i do not really understand what do you mean by CTR in your case . for me you get a CTR by making a threshold on your probability predictions to make decision of click or not and after that you get the CRT which is (# really clicked / # impressions).
\nor do you mean that the CTR = mean of your probability predictions ??
\nthank you for response
", "votes": 0, "replies": [] }, { "id": 66427, "author_name": "karlos", "author_url": "", "post_date": "2015-03-16T18:01:41.773000", "content": "Please feel free to fork or send a pull request if there's anything I missed or could be done better.
", "votes": 0, "replies": [] }, { "id": 66426, "author_name": "karlos", "author_url": "", "post_date": "2015-03-16T18:01:30.637000", "content": "I made a couple of small changes so that the code would compile on Cygwin. Project is on github
\nhttps://github.com/tadkar/libFFM-1.02-cygwin
", "votes": 0, "replies": [] }, { "id": 65813, "author_name": "mandora", "author_url": "", "post_date": "2015-03-09T20:44:24.460000", "content": "The parameters we used for LibFM:
\nsolver: als
\nnumber of iteration: 100
\ndimension: 1,1,20 (bias on, linear term on, the dimension of latent space is 20)
\nregularization coefficient: 0,1,10
\n\n[quote=irsneg;57614]
\nHi again!
\nI was wondering if you could please share the parameters that you were using with libFM when you guys tried it in your experiments (-learn_rate, -regular, -init_stddev). Thanks!
\n[quote=guestwalk;56262]
\nThanks. :) Please see my opinions:
\n1. As mentioned on page 14 of our slide, this approach is firstly proposed by
Michael Jahrer et al. in KDD Cup 2012 Track 2. In our experiments, this
approach is better than the standard FM model. (0.001 - 0.002 improvement)
The standard FM shares the same latent space, and the field-aware FM has
dedicated latent space for each pair of fields. I think this is the reason
why the field-aware FM outperforms the standard FM in this competition.
2. Sorry I do not quite get what you mean by "simple features". May you explain
more about it?
3. We tried that, but not including these terms gave us better
results. I do not remember what the difference was, so I cannot report a
number.
[quote=irsneg;56198]
\nHi!
\nCongrats! Nice and creative approach :). Thanks for sharing and answering all the previous questions; that's very interesting. I have a few addition questions that might interest the community as well; please have a look:
\n- How did you come up with this "field-aware" version of FM? Is it performing much better than Rendle's original approach on this problem? Do you have any thoughts on why this is the case?
- What are you doing with simple features? (They are not in the formula of phi(w,x) on slide 14)?
- Why did not you include interactions terms xj * xj (when j1 == j2 on slide 14)?
Thanks!
\n[quote=guestwalk;55762]
\nDear all,
\nWe have prepared our codes and documents.
\nFor the codes, please see here; and for the documents, please see here.
\nYour comments are very welcome. Thanks!
\n-- 3 Idiots
\n[/quote]
\n[/quote]
\n[/quote]
\n[/quote]
", "votes": 0, "replies": [] }, { "id": 65762, "author_name": "Yuchin Juan", "author_url": "", "post_date": "2015-03-09T08:08:12.977000", "content": "[quote=rokh;58729]
\nThanks for such a great solution! Hope you don't mind another little question.
How and when did you choose threshold for transforming infrequent features?
[/quote]
\nThis threshold was selected based on experiments. We tried something like 2, 5, 10, 20, 50, 100, and chose the best one among them.
", "votes": 0, "replies": [] }, { "id": 65761, "author_name": "Yuchin Juan", "author_url": "", "post_date": "2015-03-09T08:05:14.980000", "content": "[quote=wugui;57754]
\ncongratulations，i am a newer in competition, please bear with me.
\ni have a question about click rate feature.
\nbecause i think click rate feature may be useful for predicting, and i do not find it in your ppt, so i want to know why you don't use click rate feature in your solution?
\n[/quote]
\nYes, we tried what you mentioned, but did not get better results.
", "votes": 0, "replies": [] }, { "id": 65760, "author_name": "Yuchin Juan", "author_url": "", "post_date": "2015-03-09T08:03:59.583000", "content": "[quote=clustifier;57412]
\n@guestwalk,
\nDoes special care is needed for generating the gbdt features to avoid overfitting?
\nI didn't thought it needed but it looks like I do have some kind of overfitting problem when trying to implement if naively.
\nThanks,
\nC.
\n[/quote]
\nSorry for the late reply. If you find gbdt overfits, I think reducing the number of trees and the depth of a tree may help.
", "votes": 0, "replies": [] }, { "id": 58729, "author_name": "rokh", "author_url": "", "post_date": "2014-11-23T11:55:33.983000", "content": "Thanks for such a great solution! Hope you don't mind another little question.
How and when did you choose threshold for transforming infrequent features?
[quote=guestwalk;55762]
\nDear all,
\nWe have prepared our codes and documents.
\nFor the codes, please see here; and for the documents, please see here.
\nYour comments are very welcome. Thanks!
\n-- 3 Idiots
\n[/quote]
", "votes": 0, "replies": [] }, { "id": 58115, "author_name": "onthehilluu", "author_url": "", "post_date": "2014-11-15T16:09:47.273000", "content": "Chrome did not work for me to download the paper. But IE or Firefox works.
", "votes": 0, "replies": [] }, { "id": 58002, "author_name": "noooooo", "author_url": "", "post_date": "2014-11-13T15:55:15.877000", "content": "[quote=Rafal Jozefowicz;57996]
\n[quote=noooooo;57995]
\nDoes any one has the pdf of the winner?
\nThe url has nothing in it rightnow....
\nthanks all
\n[/quote]
\nOpening pdf in chrome incognito mode works for me.
\n[/quote]
\nthanks, still not work for me, but i use a software to download it :)
", "votes": 0, "replies": [] }, { "id": 57996, "author_name": "", "author_url": "", "post_date": "2014-11-13T15:25:13.490000", "content": "", "votes": 0, "replies": [] }, { "id": 57995, "author_name": "", "author_url": "", "post_date": "2014-11-13T15:21:21.243000", "content": "", "votes": 0, "replies": [] }, { "id": 57754, "author_name": "", "author_url": "", "post_date": "2014-11-11T05:39:19.063000", "content": "", "votes": 0, "replies": [] }, { "id": 57614, "author_name": "", "author_url": "", "post_date": "2014-11-10T16:15:57.390000", "content": "", "votes": 0, "replies": [] }, { "id": 57412, "author_name": "", "author_url": "", "post_date": "2014-11-06T16:04:28.657000", "content": "", "votes": 0, "replies": [] }, { "id": 56854, "author_name": "", "author_url": "", "post_date": "2014-10-27T14:15:18.443000", "content": "", "votes": 0, "replies": [] }, { "id": 56706, "author_name": "", "author_url": "", "post_date": "2014-10-25T05:57:39.553000", "content": "", "votes": 0, "replies": [] }, { "id": 56356, "author_name": "", "author_url": "", "post_date": "2014-10-21T21:48:01.993000", "content": "", "votes": 0, "replies": [] }, { "id": 56350, "author_name": "", "author_url": "", "post_date": "2014-10-21T20:08:11.527000", "content": "", "votes": 0, "replies": [] }, { "id": 56349, "author_name": "", "author_url": "", "post_date": "2014-10-21T19:56:55.673000", "content": "", "votes": 0, "replies": [] }, { "id": 56305, "author_name": "", "author_url": "", "post_date": "2014-10-20T23:09:06.797000", "content": "", "votes": 0, "replies": [] }, { "id": 56287, "author_name": "", "author_url": "", "post_date": "2014-10-20T14:01:08.750000", "content": "", "votes": 0, "replies": [] }, { "id": 56274, "author_name": "", "author_url": "", "post_date": "2014-10-20T07:17:10.273000", "content": "", "votes": 0, "replies": [] }, { "id": 56273, "author_name": "", "author_url": "", "post_date": "2014-10-20T05:32:43.947000", "content": "", "votes": 0, "replies": [] }, { "id": 56272, "author_name": "", "author_url": "", "post_date": "2014-10-20T05:30:58.267000", "content": "", "votes": 0, "replies": [] }, { "id": 56266, "author_name": "", "author_url": "", "post_date": "2014-10-19T23:22:44.510000", "content": "", "votes": 0, "replies": [] }, { "id": 56264, "author_name": "", "author_url": "", "post_date": "2014-10-19T19:52:53.797000", "content": "", "votes": 0, "replies": [] }, { "id": 56262, "author_name": "", "author_url": "", "post_date": "2014-10-19T19:12:52.550000", "content": "", "votes": 0, "replies": [] }, { "id": 56217, "author_name": "", "author_url": "", "post_date": "2014-10-18T08:20:10.440000", "content": "", "votes": 0, "replies": [] }, { "id": 56198, "author_name": "", "author_url": "", "post_date": "2014-10-18T06:39:36.077000", "content": "", "votes": 0, "replies": [] }, { "id": 56066, "author_name": "", "author_url": "", "post_date": "2014-10-14T20:13:00.173000", "content": "", "votes": 0, "replies": [] }, { "id": 56009, "author_name": "", "author_url": "", "post_date": "2014-10-13T09:52:42.617000", "content": "", "votes": 0, "replies": [] }, { "id": 55952, "author_name": "", "author_url": "", "post_date": "2014-10-12T06:54:36.037000", "content": "", "votes": 0, "replies": [] }, { "id": 55906, "author_name": "", "author_url": "", "post_date": "2014-10-10T20:59:22.207000", "content": "", "votes": 0, "replies": [] }, { "id": 55862, "author_name": "", "author_url": "", "post_date": "2014-10-09T22:35:57.823000", "content": "", "votes": 0, "replies": [] }, { "id": 55799, "author_name": "", "author_url": "", "post_date": "2014-10-08T17:28:06.353000", "content": "", "votes": 0, "replies": [] }, { "id": 372895, "author_name": "", "author_url": "", "post_date": "2018-08-20T15:19:06.467000", "content": "", "votes": 0, "replies": [] } ], "comment_count": 57, "messages": [ { "id": 55762, "postDate": "2014-10-08T07:53:43.963Z", "content": "Dear all,
\nWe have prepared our codes and documents.
\nFor the codes, please see here; and for the documents, please see here.
\nIn this competition and Avazu's CTR competition, we find field-aware factorization machines (FFM) are powerful models. We have implemented it in the package LIBFFM.
\nYour comments are very welcome. Thanks!
\n-- 3 Idiots
", "votes": 71 }, { "id": 578934, "postDate": "2019-07-18T10:01:02.703Z", "content": "Hey I have a little doubt, are the features belonging to same field mutually exclusive or can they occur together.\n for ex. can we have libffm data like 0 1:21:1 1:3:1 1:55:1 ... I tried looking it up online but couldn't find much info. Thanks for help.
", "rawMarkdown": "Hey I have a little doubt, are the features belonging to same field mutually exclusive or can they occur together.\n for ex. can we have libffm data like 0 1:21:1 1:3:1 1:55:1 ... I tried looking it up online but couldn't find much info. Thanks for help. ", "votes": 1, "replies": [ { "id": 868746, "postDate": "2020-05-31T13:07:17.803Z", "content": "hi,\nI have the same question,\nI just want to some help to transform the CSV to FFM data format ,\ncould you pls help?\nthx
", "rawMarkdown": "hi,\nI have the same question,\nI just want to some help to transform the CSV to FFM data format ,\ncould you pls help?\nthx" } ] }, { "id": 55888, "postDate": "2014-10-10T16:17:20.997Z", "content": "Tianqi:
\nBecause I do not know how other people used FM, sorry I have no idea about the
"difference". I guess the following three settings might be helpful:
1. Transforming infrequent features into a special tag. Conceptually, infrequent
features should only include very little or even no information, so it
should be very hard for a model to extract those information. In fact, these
features can be very noisy. We got significant improvement (more than 0.0005)
by transforming these features into a special tag.
2. Transforming numerical features (I1-I13) to categorical features.
Empirically we observe using categorical features is always better than
using numerical features. However, too many features are generated if
numerical features are directly transformed into categorical features, so we
use
v <- floor(log(v)^2)
\nto reduce the number of features generated.
\n3. Data normalization. According to our empirical experience, instance-wise
data normalization makes the optimization problem easier to be solved.
For example, for the following vector x
(0, 0, 1, 1, 0, 1, 0, 1, 0),
\nthe standard data normalization divides each element of x by by the 2-norm
of x. The normalized x becomes
(0, 0, 0.5, 0.5, 0, 0.5, 0, 0.5, 0).
\nThe hashing trick do not contribute any improvement on the leaderboard. We
apply the hashing trick only because it makes our life easier to generate
features. :)
Amazing solution!! Some quick questions:
\n(1) What is the impact of using field aware FM and the global latent factor version?
\n(2) By LR and FM, do you use features generated by GBDT? Have you measure the impact of the GBDT generated feature?
", "votes": 9 }, { "id": 59345, "postDate": "2014-12-01T17:13:32.243Z", "content": "Very nice solution. I'm curious - how did you determine the average CTR's on the public/private leaderboards (slide 16)?
", "votes": 5 }, { "id": 182484, "postDate": "2017-05-14T04:16:37.237Z", "content": "我觉得ffm是个很好的主意
", "rawMarkdown": "我觉得ffm是个很好的主意" }, { "id": 65763, "postDate": "2015-03-09T08:25:50.893Z", "content": "[quote=jwjohnson314;59345]
\nVery nice solution. I'm curious - how did you determine the average CTR's on the public/private leaderboards (slide 16)?
\n[/quote]
\nSuppose we have an impression whose label is 1. If the prediction for this
impression is 0, then we should get an infinite logloss. In practice this is
not desired, so on the submission system there is a ceiling (C) for logloss.
If the logloss for an impression is greater than this ceiling, then it will be
truncated. Note that we do not know the value of this ceiling.
Using this feature, we can hack the average CTR by two submissions.
\nThe first submission contains all ones, so the logloss we get is P1 =
(nr_non_click*C)/nr_instance.
The second submission contains all zeros, so the logloss we get is P2 =
(nr_click*C)/nr_instance.
We can then get the average CTR = P2/(P1+P2).
", "votes": 4 }, { "id": 56827, "postDate": "2014-10-27T05:59:10.610Z", "content": "[quote=Torgos;56807]
\nThank you for posting your solution, but the linked explanation pdf currently returns the error "You don't have permission to access /~r01922136/kaggle-2014-criteo.pdf on this server."
\n[/quote]
\nIt should be back now. Please try again. Thanks.
", "votes": 3 }, { "id": 55932, "postDate": "2014-10-11T16:04:12.633Z", "content": "You are very welcome!
\n(2) They are similar but not the same. The FM model in the paper you provided
is field unaware. The difference between equation 1 in the paper and the
formula on page 14 of our slide is that our w is not only indexed by j1 and
j2, but also indexed by f1 and f2. Consider the example on page 15, if
Rendle's FM is applied, it becomes:
w376^Tw248x376x248 + w376^Tw571x376x571 + w376^Tw942x376x942
+ w248^Tw571x248x571 + w248^Tw942x248x942
+ w571^Tw942x571x942
BTW, we use k = 4, not k = 2.
\nPlease let me know if you have more questions. :)
\n[quote=Inspector;55906]
\n1) I think this helped a lot. I was confused what the hashing trick was doing. I was thinking perhaps the value of a feature, say 5a9ed9b0, was REPLACED by an integer. So, I understand now that one-hot-encoding is still being used, it is just the indexing of the data which is improved (memory wise) when hashed.
\n2) So this is essentially the same model as shown in the Rendle paper http://www.ismll.uni-hildesheim.de/pub/pdfs/Rendle2010FM.pdf (equation 1) , where you used k=2 (the number of factors)? Is this correct?
\nThanks very much!!
\n[/quote]
", "votes": 1 }, { "id": 55898, "postDate": "2014-10-10T19:21:13.120Z", "content": "(1) Actually the fact is completely contrary to what you mentioned. We one-hot
encode everything. The "integer" you mentioned is actually the index of a
feature. For example, consider the feature "C1:5a9ed9b0", and it is hashed
to an integer, say, 872342. It does not mean that a feature's value is
872342; it actually means that the value of 872342nd feature is non-zero.
(2) Consider the example in the 15th page of the slide. x is an instance of
training data, and w is the model. Because x is very sparse, we use sparse
format to represent it. That is, only the 376th, the 248th, the 571st, and
the 942nd features are non-zero. phi(w,x) can be obtained as the slide
shows. So, for your question, x_j1, x_j2, w_j1,f2, and w_j2,f1 are symbols
used to represent the corresponding w and x in each term. For example, in
the first term, x_j1 is x_376, x_j2 is x_248, w_j1,f2 is w_376,2, and
w_j2,f1 is w_248,1.
Not sure if you think the questions are answered. Please let me know if it is
not clear. :)
[quote=Inspector;55862]
\nFor those of us (me ;-)) just starting to learn about FM, hope you don't mind a couple elementary questions.
\n1) Are you only using numeric variables in the model - no one-hot encoding at all? Everything is hashed to an integer?
\n
2) On Slide #14-15 I am unsure what {x_j1, x_j2 } and {wj1,f2, wj2,f1} are. Would you explain?
[/quote]
", "votes": 1 }, { "id": 55830, "postDate": "2014-10-09T10:49:35.607Z", "content": "Tianqi:
\n(1) We have not submitted one using the global latent factor version. On our validation set, the difference was around 0.001. (Both of them did not use GBDT features.)
\n(2) We compare them in the last page of our slide. "FM+GBDT" and "FM" respectively represent training FM with/without adding the GBDT features. We can see the around 0.001 improvement by adding the GBDT features.
", "votes": 1 }, { "id": 56807, "postDate": "2014-10-26T17:36:41.653Z", "content": "Thank you for posting your solution, but the linked explanation pdf currently returns the error "You don't have permission to access /~r01922136/kaggle-2014-criteo.pdf on this server."
", "votes": 2 }, { "id": 55966, "postDate": "2014-10-12T16:15:07.360Z", "content": "Having to do early stopping seems to be a very common thing that can not be avoided in factorization model. Sometimes using an alternative regularization(e.g max-norm on latent factor) can make the long tail effect comes slower, however it will still come, and the best solution is usually worse than what you can have in optimal iteration.
\nAmong the factorization models, bayesian version is stable to long-tail effect, however they are not practical(at least not the plain version) because you have to store all the generated samples for prediction. I have no idea how this can be avoided easily:) Maybe using other stable model to calibrate what is generated from FM models...
\nThe regularization you use appears to be the common one used by MF folks. Since lambda is applied only when nonzero feature occurs. It is not standard L2 regularizer. It is a weighed L2 regularization where weight on feature i is lambda times number of nonzero feature i
\nThanks for answering so many questions! I learned a lot from your solution
\n[quote=guestwalk;55952]
\nFor the questions about the sensitivity and the situation after the best
iteration, please see the attachment. (tr_loss and va_loss respectively
represent the logloss on the training and the validation set.) I do this on our
own validation set; also, I removed GBDT features so that the experiments can
run faster.
We can see the setting eta = 0.1 indeed achieve the best performance on the
validation set. For eta = 0.2, eta = 0.05, and eta = 0.02, a similar
performance can be achieved. The smaller eta is, the more iterations are
required to achieve a good performance; it is the classic SGD behavior. If eta
is too large (in this case, 0.5), then the result is significantly worse; for
the setting eta = 0.01, it do not converge at 100th iterations. My opinion is
choosing a good eta is still crucial to SGD. For a large eta, the accuracy can
be hurt; for a small eta, it takes a long time to converge.
On the other hand, the "long tail" iterations indeed hurts a lot. In fact, we
also used this learning rate schedule to solve the simple matrix factorization
(MF) problem, and the performance in the long tail iterations is quite stable
(i.e., close to the best performance). I think the difference may be that FM is
much complicated compared to MF, so the overfitting problem is much harder to
be avoided. (In some sense this FM problem is a combination of 39*38/2 MF
problem, given there are 39 features for each impression.) This problem is
really annoying because it makes our solution less useful in practice. Any
suggestion?
For regularization, our implementation is very simple. Every time we choose an
instance, we only update the corresponding coordinate of the non-zero features.
For example, if the non-zero coordinates of an instance x are 376, 248, 571,
and 942, then only w376, w248, w571, and w942 are updated. Take w248 as an
example, the corresponding gradient is
g248 = lambda*w248 + (the gradient of the loss term)
\nNote that lambda is a fixed parameter, we don't change it during training at
all. Not sure if it is one of the approaches you mentioned.
Finally, the gradient of regularization is also accumulated.
\n[quote=Tianqi Chen;55931]
\nInteresting! This is the first time I see AdaGrad to work on factorization models.
\nSometimes it works great for convex models and some neuralnets. But I wasn't able get it work better than constant learning rate or decayed learning rate on factorization models.
\nHow sensitive is the result to the initial learning rate, I suspect it is just for lower rate, you run for longer rounds, but still get similar results?
\nI get your point about choosing the stopping condition. Since you use AdaGrad, I guess the raising up period after optimal point will be more smoother? Sometimes Adagrad in convex models tends to less overfit, say 50 rounds gets you there, and running for 70 rounds won't hurt too much. I am wondering what is in your case.
\nFor L2 regularizer, how do you implement it? If it is standard L2 regularizer, you don;t want to regularize parameters every rounds, so either some lazy trick or divide the regularizer by #occurance can be used. In many MF codes, sometimes ppl simply take another regularizer lambda #occraunce_of_fi w_i^2 and run weight decay every time you see the parameter.
\nDoes the gradient of regularizer goes into the accumulated gradient?
\nTianqi
\n[/quote]
\n[/quote]
", "votes": 2 }, { "id": 55931, "postDate": "2014-10-11T16:02:50.270Z", "content": "Interesting! This is the first time I see AdaGrad to work on factorization models.
\nSometimes it works great for convex models and some neuralnets. But I wasn't able get it work better than constant learning rate or decayed learning rate on factorization models.
\nHow sensitive is the result to the initial learning rate, I suspect it is just for lower rate, you run for longer rounds, but still get similar results?
\nI get your point about choosing the stopping condition. Since you use AdaGrad, I guess the raising up period after optimal point will be more smoother? Sometimes Adagrad in convex models tends to less overfit, say 50 rounds gets you there, and running for 70 rounds won't hurt too much. I am wondering what is in your case.
\nFor L2 regularizer, how do you implement it? If it is standard L2 regularizer, you don;t want to regularize parameters every rounds, so either some lazy trick or divide the regularizer by #occurance can be used. In many MF codes, sometimes ppl simply take another regularizer lambda #occraunce_of_fi w_i^2 and run weight decay every time you see the parameter.
\nDoes the gradient of regularizer goes into the accumulated gradient?
\nTianqi
\n[quote=guestwalk;55930]
\nYou are very welcome!
\nHere are my answers:
\n(1) For the weight decay of the learning rate (step size), we use
per-coordinate learning rate schedule. For a specific coordinate, the
update rule is:
G = G + g*g
w = w - eta*(1/sqrt(G))*g,
where g is the gradient of that coordinate, G records the accumulated
squared sum of the gradients in the past updates, eta is the initial
learning rate, and w is the model. Note that the initial value of G is set
to 1.
This learning rate schedule has been studied in, for example, this paper.
Because we know this learning rate schedule before the competition, we did
not spend any time tuning.
(2) For the value of initial learning rate (eta), we tried 0.5, 0.2, 0.1, 0.05,
0.02, and 0.01 at the early stage we used FM. Then we found that 0.1 is a
good balance between the performance and the training time, so we just
fixed it and had never changed it.
Because at that time we haven't included GBDT features, FM ran about three
times faster compared to the current setting. These experiments took us
around 6 hours.
(3) For the stopping criterion, we did not find a good one. What we did was:
\n1. Run experiments on our validation set, and record the number of
iterations to achieve the best performance.
2. Use this number of iterations to train the model on the whole
training data.
So for each of our submissions, we run experiments at least two times: one
on validation set and the other on the whole data set.
(4) There is another parameter need to be tuned: lambda (regularization
penalty). Like other machine learning models such as SVM or standard LR,
this parameter need to be tuned. We ran experiments very similar to what we
mentioned in tuning eta. The time cost of tuning this parameter was also
around 6 hours.
Please feel free to let me know if you have more questions. :)
\n[quote=Tianqi Chen;55890]
\nThanks for detailed answer!
\nI like your comment on the data normalization trick. This is also what I observed repeatly in experimenting with factorization models. I think this is mainly because the MF models applies L2 projection and L2 inner product, so keeping the features' L2 norm constant makes different product roughly in the same scale without introducing instance bias.
\nHere is another question:) I took a look over your solver and seems it is SGD based solver, so I assume you need to tune the step size and weight decay and do early stopping on validation set? How does the three factors affect your experiment? I guess usually different step size and weight decay could lead to slightly different stoping round. How much effort do you spent on tuning them?
\nTianqi
\n[/quote]
\n[/quote]
", "votes": 2 }, { "id": 55930, "postDate": "2014-10-11T15:34:23.923Z", "content": "You are very welcome!
\nHere are my answers:
\n(1) For the weight decay of the learning rate (step size), we use
per-coordinate learning rate schedule. For a specific coordinate, the
update rule is:
G = G + g*g
w = w - eta*(1/sqrt(G))*g,
where g is the gradient of that coordinate, G records the accumulated
squared sum of the gradients in the past updates, eta is the initial
learning rate, and w is the model. Note that the initial value of G is set
to 1.
This learning rate schedule has been studied in, for example, this paper.
Because we know this learning rate schedule before the competition, we did
not spend any time tuning.
(2) For the value of initial learning rate (eta), we tried 0.5, 0.2, 0.1, 0.05,
0.02, and 0.01 at the early stage we used FM. Then we found that 0.1 is a
good balance between the performance and the training time, so we just
fixed it and had never changed it.
Because at that time we haven't included GBDT features, FM ran about three
times faster compared to the current setting. These experiments took us
around 6 hours.
(3) For the stopping criterion, we did not find a good one. What we did was:
\n1. Run experiments on our validation set, and record the number of
iterations to achieve the best performance.
2. Use this number of iterations to train the model on the whole
training data.
So for each of our submissions, we run experiments at least two times: one
on validation set and the other on the whole data set.
(4) There is another parameter need to be tuned: lambda (regularization
penalty). Like other machine learning models such as SVM or standard LR,
this parameter need to be tuned. We ran experiments very similar to what we
mentioned in tuning eta. The time cost of tuning this parameter was also
around 6 hours.
Please feel free to let me know if you have more questions. :)
\n[quote=Tianqi Chen;55890]
\nThanks for detailed answer!
\nI like your comment on the data normalization trick. This is also what I observed repeatly in experimenting with factorization models. I think this is mainly because the MF models applies L2 projection and L2 inner product, so keeping the features' L2 norm constant makes different product roughly in the same scale without introducing instance bias.
\nHere is another question:) I took a look over your solver and seems it is SGD based solver, so I assume you need to tune the step size and weight decay and do early stopping on validation set? How does the three factors affect your experiment? I guess usually different step size and weight decay could lead to slightly different stoping round. How much effort do you spent on tuning them?
\nTianqi
\n[/quote]
", "votes": 2 }, { "id": 55890, "postDate": "2014-10-10T16:35:31.920Z", "content": "Thanks for detailed answer!
\nI like your comment on the data normalization trick. This is also what I observed repeatly in experimenting with factorization models. I think this is mainly because the MF models applies L2 projection and L2 inner product, so keeping the features' L2 norm constant makes different product roughly in the same scale without introducing instance bias.
\nHere is another question:) I took a look over your solver and seems it is SGD based solver, so I assume you need to tune the step size and weight decay and do early stopping on validation set? How does the three factors affect your experiment? I guess usually different step size and weight decay could lead to slightly different stoping round. How much effort do you spent on tuning them?
\nTianqi
\n[quote=guestwalk;55888]
\nTianqi:
\nBecause I do not know how other people used FM, sorry I have no idea about the
"difference". I guess the following three settings might be helpful:
1. Transforming infrequent features into a special tag. Conceptually, infrequent
features should only include very little or even no information, so it
should be very hard for a model to extract those information. In fact, these
features can be very noisy. We got significant improvement (more than 0.0005)
by transforming these features into a special tag.
2. Transforming numerical features (I1-I13) to categorical features.
Empirically we observe using categorical features is always better than
using numerical features. However, too many features are generated if
numerical features are directly transformed into categorical features, so we
use
v <- floor(log(v)^2)
\nto reduce the number of features generated.
\n3. Data normalization. According to our empirical experience, instance-wise
data normalization makes the optimization problem easier to be solved.
For example, for the following vector x
(0, 0, 1, 1, 0, 1, 0, 1, 0),
\nthe standard data normalization divides each element of x by by the 2-norm
of x. The normalized x becomes
(0, 0, 0.5, 0.5, 0, 0.5, 0, 0.5, 0).
\nThe hashing trick do not contribute any improvement on the leaderboard. We
apply the hashing trick only because it makes our life easier to generate
features. :)
[/quote]
", "votes": 2 }, { "id": 55851, "postDate": "2014-10-09T17:11:34.677Z", "content": "Thanks for the clarification!! Since a lot of people used FM and I do not think they get to the performance as you do. Can you comment on which part of your method(apart from GBDT features) brings you the difference?
\nAs far as I can list, this could be parameter tuning, or the Prep-B step you introduced a) changing the less frequent occurred instance to a single tag; b) Hashing trick to limit all the thing to 1M categories; c) The integer remapping techniques
\n\n[quote=guestwalk;55830]
\nTianqi:
\n(1) We have not submitted one using the global latent factor version. On our validation set, the difference was around 0.001. (Both of them did not use GBDT features.)
\n(2) We compare them in the last page of our slide. "FM+GBDT" and "FM" respectively represent training FM with/without adding the GBDT features. We can see the around 0.001 improvement by adding the GBDT features.
\n[/quote]
", "votes": 2 }, { "id": 3194949, "postDate": "2025-05-06T13:13:28.103Z", "content": "@guestwalk The link to this document is broken, please do you still have a backup?
", "rawMarkdown": "@guestwalk The link to this document is broken, please do you still have a backup?" }, { "id": 336386, "postDate": "2018-05-31T15:56:42.503Z", "content": "Thank you all for sharing your code and documenting your experiences. What an excellent education for us all!
\n\nIn your slides, you note that version2 (v2) has \"50 trees and 8 latent factors\", i.e. the GBDT stage uses 50 trees (compared to 30 in v1), and the FFM stage uses 8 latent factors. Questions:
\n\nHi ! First of all, congrat for the solution,
\nin your solution (page 13), i do not really understand what do you mean by CTR in your case . for me you get a CTR by making a threshold on your probability predictions to make decision of click or not and after that you get the CRT which is (# really clicked / # impressions).
\nor do you mean that the CTR = mean of your probability predictions ??
\nthank you for response
" }, { "id": 66427, "postDate": "2015-03-16T18:01:41.773Z", "content": "Please feel free to fork or send a pull request if there's anything I missed or could be done better.
" }, { "id": 66426, "postDate": "2015-03-16T18:01:30.637Z", "content": "I made a couple of small changes so that the code would compile on Cygwin. Project is on github
\nhttps://github.com/tadkar/libFFM-1.02-cygwin
" }, { "id": 65813, "postDate": "2015-03-09T20:44:24.460Z", "content": "The parameters we used for LibFM:
\nsolver: als
\nnumber of iteration: 100
\ndimension: 1,1,20 (bias on, linear term on, the dimension of latent space is 20)
\nregularization coefficient: 0,1,10
\n\n[quote=irsneg;57614]
\nHi again!
\nI was wondering if you could please share the parameters that you were using with libFM when you guys tried it in your experiments (-learn_rate, -regular, -init_stddev). Thanks!
\n[quote=guestwalk;56262]
\nThanks. :) Please see my opinions:
\n1. As mentioned on page 14 of our slide, this approach is firstly proposed by
Michael Jahrer et al. in KDD Cup 2012 Track 2. In our experiments, this
approach is better than the standard FM model. (0.001 - 0.002 improvement)
The standard FM shares the same latent space, and the field-aware FM has
dedicated latent space for each pair of fields. I think this is the reason
why the field-aware FM outperforms the standard FM in this competition.
2. Sorry I do not quite get what you mean by "simple features". May you explain
more about it?
3. We tried that, but not including these terms gave us better
results. I do not remember what the difference was, so I cannot report a
number.
[quote=irsneg;56198]
\nHi!
\nCongrats! Nice and creative approach :). Thanks for sharing and answering all the previous questions; that's very interesting. I have a few addition questions that might interest the community as well; please have a look:
\n- How did you come up with this "field-aware" version of FM? Is it performing much better than Rendle's original approach on this problem? Do you have any thoughts on why this is the case?
- What are you doing with simple features? (They are not in the formula of phi(w,x) on slide 14)?
- Why did not you include interactions terms xj * xj (when j1 == j2 on slide 14)?
Thanks!
\n[quote=guestwalk;55762]
\nDear all,
\nWe have prepared our codes and documents.
\nFor the codes, please see here; and for the documents, please see here.
\nYour comments are very welcome. Thanks!
\n-- 3 Idiots
\n[/quote]
\n[/quote]
\n[/quote]
\n[/quote]
" }, { "id": 65762, "postDate": "2015-03-09T08:08:12.977Z", "content": "[quote=rokh;58729]
\nThanks for such a great solution! Hope you don't mind another little question.
How and when did you choose threshold for transforming infrequent features?
[/quote]
\nThis threshold was selected based on experiments. We tried something like 2, 5, 10, 20, 50, 100, and chose the best one among them.
" }, { "id": 65761, "postDate": "2015-03-09T08:05:14.980Z", "content": "[quote=wugui;57754]
\ncongratulations，i am a newer in competition, please bear with me.
\ni have a question about click rate feature.
\nbecause i think click rate feature may be useful for predicting, and i do not find it in your ppt, so i want to know why you don't use click rate feature in your solution?
\n[/quote]
\nYes, we tried what you mentioned, but did not get better results.
" }, { "id": 65760, "postDate": "2015-03-09T08:03:59.583Z", "content": "[quote=clustifier;57412]
\n@guestwalk,
\nDoes special care is needed for generating the gbdt features to avoid overfitting?
\nI didn't thought it needed but it looks like I do have some kind of overfitting problem when trying to implement if naively.
\nThanks,
\nC.
\n[/quote]
\nSorry for the late reply. If you find gbdt overfits, I think reducing the number of trees and the depth of a tree may help.
" }, { "id": 58729, "postDate": "2014-11-23T11:55:33.983Z", "content": "Thanks for such a great solution! Hope you don't mind another little question.
How and when did you choose threshold for transforming infrequent features?
[quote=guestwalk;55762]
\nDear all,
\nWe have prepared our codes and documents.
\nFor the codes, please see here; and for the documents, please see here.
\nYour comments are very welcome. Thanks!
\n-- 3 Idiots
\n[/quote]
" }, { "id": 58115, "postDate": "2014-11-15T16:09:47.273Z", "content": "Chrome did not work for me to download the paper. But IE or Firefox works.
" }, { "id": 58002, "postDate": "2014-11-13T15:55:15.877Z", "content": "[quote=Rafal Jozefowicz;57996]
\n[quote=noooooo;57995]
\nDoes any one has the pdf of the winner?
\nThe url has nothing in it rightnow....
\nthanks all
\n[/quote]
\nOpening pdf in chrome incognito mode works for me.
\n[/quote]
\nthanks, still not work for me, but i use a software to download it :)
" }, { "id": 57996, "postDate": "2014-11-13T15:25:13.490Z", "content": "[quote=noooooo;57995]
\nDoes any one has the pdf of the winner?
\nThe url has nothing in it rightnow....
\nthanks all
\n[/quote]
\nOpening pdf in chrome incognito mode works for me.
" }, { "id": 57995, "postDate": "2014-11-13T15:21:21.243Z", "content": "Does any one has the pdf of the winner?
\nThe url has nothing in it rightnow....
\nthanks all
" }, { "id": 57754, "postDate": "2014-11-11T05:39:19.063Z", "content": "congratulations，i am a newer in competition, please bear with me.
\ni have a question about click rate feature.
\nbecause i think click rate feature may be useful for predicting, and i do not find it in your ppt, so i want to know why you don't use click rate feature in your solution?
" }, { "id": 57614, "postDate": "2014-11-10T16:15:57.390Z", "content": "Hi again!
\nI was wondering if you could please share the parameters that you were using with libFM when you guys tried it in your experiments (-learn_rate, -regular, -init_stddev). Thanks!
\n\n[quote=guestwalk;56262]
\nThanks. :) Please see my opinions:
\n1. As mentioned on page 14 of our slide, this approach is firstly proposed by
Michael Jahrer et al. in KDD Cup 2012 Track 2. In our experiments, this
approach is better than the standard FM model. (0.001 - 0.002 improvement)
The standard FM shares the same latent space, and the field-aware FM has
dedicated latent space for each pair of fields. I think this is the reason
why the field-aware FM outperforms the standard FM in this competition.
2. Sorry I do not quite get what you mean by "simple features". May you explain
more about it?
3. We tried that, but not including these terms gave us better
results. I do not remember what the difference was, so I cannot report a
number.
[quote=irsneg;56198]
\nHi!
\nCongrats! Nice and creative approach :). Thanks for sharing and answering all the previous questions; that's very interesting. I have a few addition questions that might interest the community as well; please have a look:
\n- How did you come up with this "field-aware" version of FM? Is it performing much better than Rendle's original approach on this problem? Do you have any thoughts on why this is the case?
- What are you doing with simple features? (They are not in the formula of phi(w,x) on slide 14)?
- Why did not you include interactions terms xj * xj (when j1 == j2 on slide 14)?
Thanks!
\n[quote=guestwalk;55762]
\nDear all,
\nWe have prepared our codes and documents.
\nFor the codes, please see here; and for the documents, please see here.
\nYour comments are very welcome. Thanks!
\n-- 3 Idiots
\n[/quote]
\n[/quote]
\n[/quote]
" }, { "id": 57412, "postDate": "2014-11-06T16:04:28.657Z", "content": "@guestwalk,
\nDoes special care is needed for generating the gbdt features to avoid overfitting?
\nI didn't thought it needed but it looks like I do have some kind of overfitting problem when trying to implement if naively.
\nThanks,
\nC.
" }, { "id": 56854, "postDate": "2014-10-27T14:15:18.443Z", "content": "Thanks; it's working.
" }, { "id": 56706, "postDate": "2014-10-25T05:57:39.553Z", "content": "1. Right.
\n2. Yes.
\n[quote=saurk;56356]
\nThe 4 tuple (f1,j1,f2, j2) uniquely indexes the parameter dot product, right ? Can we construct a unique 2 tuple (f1__j1, f2_j2) from the given 4 tuple which indexes to the same dot product value ?
\n[/quote]
" }, { "id": 56356, "postDate": "2014-10-21T21:48:01.993Z", "content": "The 4 tuple (f1,j1,f2, j2) uniquely indexes the parameter dot product, right ? Can we construct a unique 2 tuple (f1__j1, f2_j2) from the given 4 tuple which indexes to the same dot product value ?
" }, { "id": 56350, "postDate": "2014-10-21T20:08:11.527Z", "content": "[quote=Inspector;56305]
\nI am struggling to see the difference. Are their any tutorials or explanations online you know of that explains field aware FM versus standard FM?
\n[/quote]
\nThe only reference I know is these slides provided by Opera Solutions in KDD Cup 2012. Please check page 11. Though the symbols are quite different from ours, the concept behind is the same.
" }, { "id": 56349, "postDate": "2014-10-21T19:56:55.673Z", "content": "No, field-aware FM is nothing to do with hashing collision. For the difference between field-aware FM and standard FM, please see this post.
\n[quote=saurk;56287]
\nPlease correct me if wrong but my understanding was that the reason you introduce field-awareness is because the hashing on feature-pairs result is collision. If the dimensions were much less so there was no need for hashing, would we still use field-aware FMs or thats redundant ?
\n[/quote]
" }, { "id": 56305, "postDate": "2014-10-20T23:09:06.797Z", "content": "[quote=guestwalk;55932]
\nYou are very welcome!
\n(2) They are similar but not the same. The FM model in the paper you provided
is field unaware. The difference between equation 1 in the paper and the
formula on page 14 of our slide is that our w is not only indexed by j1 and
j2, but also indexed by f1 and f2. Consider the example on page 15, if
Rendle's FM is applied, it becomes:
w376^Tw248x376x248 + w376^Tw571x376x571 + w376^Tw942x376x942
+ w248^Tw571x248x571 + w248^Tw942x248x942
+ w571^Tw942x571x942
[/quote]
\n\nI am struggling to see the difference. Are their any tutorials or explanations online you know of that explains field aware FM versus standard FM?
" }, { "id": 56287, "postDate": "2014-10-20T14:01:08.750Z", "content": "Please correct me if wrong but my understanding was that the reason you introduce field-awareness is because the hashing on feature-pairs result is collision. If the dimensions were much less so there was no need for hashing, would we still use field-aware FMs or thats redundant ?
" }, { "id": 56274, "postDate": "2014-10-20T07:17:10.273Z", "content": "Thanks for the quick response again. You answered all my questions!
\n1. OK :).
\n2. Oh, that's surprising. So you use only interaction terms; interesting.
\n\n[quote=guestwalk;56272]
\nYou are welcome. :)
\n1. Yes. A 0.001 improvement is very significant in this competition.
\n2. I see. The same as your previous question (3). We have tried that, but it did not
\ngive us better result.
\n[quote=irsneg;56264]
\nThanks for the quick answer :).
\n1. OK, interesting. The improvement is rather marginal though. Is it even significant?
\n2. I meant w^T * x from slide 13 (without interaction terms), so that the overall model is with phi(w,x) = w^T * x + sum < w_j1,f1,w_j2,f2> * x_j1 * x_j2 and not only the sum as in the slides, right (I cannot remove the strange ="" which was added automatically...)?
\n3. OK.
\n[/quote]
\n[/quote]
" }, { "id": 56273, "postDate": "2014-10-20T05:32:43.947Z", "content": "I do not quite understand what do you mean. May you give a mathematical formulation or an example? Thanks!
\n[quote=saurk;56266]
\nIf we incorporate the field awareness in the feature value itself, say the last two digits of each feature value maps one on one with the source column from which it came, will this FM be same as Rendel's libFM implementation ??
\n[/quote]
" }, { "id": 56272, "postDate": "2014-10-20T05:30:58.267Z", "content": "You are welcome. :)
\n1. Yes. A 0.001 improvement is very significant in this competition.
\n2. I see. The same as your previous question (3). We have tried that, but it did not
\ngive us better result.
\n[quote=irsneg;56264]
\nThanks for the quick answer :).
\n1. OK, interesting. The improvement is rather marginal though. Is it even significant?
\n2. I meant w^T * x from slide 13 (without interaction terms), so that the overall model is with phi(w,x) = w^T * x + sum < w_j1,f1,w_j2,f2> * x_j1 * x_j2 and not only the sum as in the slides, right (I cannot remove the strange ="" which was added automatically...)?
\n3. OK.
\n[/quote]
" }, { "id": 56266, "postDate": "2014-10-19T23:22:44.510Z", "content": "If we incorporate the field awareness in the feature value itself, say the last two digits of each feature value maps one on one with the source column from which it came, will this FM be same as Rendel's libFM implementation ??
" }, { "id": 56264, "postDate": "2014-10-19T19:52:53.797Z", "content": "Thanks for the quick answer :).
\n1. OK, interesting. The improvement is rather marginal though. Is it even significant?
\n2. I meant w^T * x from slide 13 (without interaction terms), so that the overall model is with phi(w,x) = w^T * x + sum < w_j1,f1,w_j2,f2> * x_j1 * x_j2 and not only the sum as in the slides, right (I cannot remove the strange ="" which was added automatically...)?
\n3. OK.
\n[quote=guestwalk;56262]
\nThanks. :) Please see my opinions:
\n1. As mentioned on page 14 of our slide, this approach is firstly proposed by
Michael Jahrer et al. in KDD Cup 2012 Track 2. In our experiments, this
approach is better than the standard FM model. (0.001 - 0.002 improvement)
The standard FM shares the same latent space, and the field-aware FM has
dedicated latent space for each pair of fields. I think this is the reason
why the field-aware FM outperforms the standard FM in this competition.
2. Sorry I do not quite get what you mean by "simple features". May you explain
more about it?
3. We tried that, but not including these terms gave us better
results. I do not remember what the difference was, so I cannot report a
number.
[quote=irsneg;56198]
\nHi!
\nCongrats! Nice and creative approach :). Thanks for sharing and answering all the previous questions; that's very interesting. I have a few addition questions that might interest the community as well; please have a look:
\n- How did you come up with this "field-aware" version of FM? Is it performing much better than Rendle's original approach on this problem? Do you have any thoughts on why this is the case?
- What are you doing with simple features? (They are not in the formula of phi(w,x) on slide 14)?
- Why did not you include interactions terms xj * xj (when j1 == j2 on slide 14)?
Thanks!
\n[quote=guestwalk;55762]
\nDear all,
\nWe have prepared our codes and documents.
\nFor the codes, please see here; and for the documents, please see here.
\nYour comments are very welcome. Thanks!
\n-- 3 Idiots
\n[/quote]
\n[/quote]
\n[/quote]
" }, { "id": 56262, "postDate": "2014-10-19T19:12:52.550Z", "content": "Thanks. :) Please see my opinions:
\n1. As mentioned on page 14 of our slide, this approach is firstly proposed by
Michael Jahrer et al. in KDD Cup 2012 Track 2. In our experiments, this
approach is better than the standard FM model. (0.001 - 0.002 improvement)
The standard FM shares the same latent space, and the field-aware FM has
dedicated latent space for each pair of fields. I think this is the reason
why the field-aware FM outperforms the standard FM in this competition.
2. Sorry I do not quite get what you mean by "simple features". May you explain
more about it?
3. We tried that, but not including these terms gave us better
results. I do not remember what the difference was, so I cannot report a
number.
[quote=irsneg;56198]
\nHi!
\nCongrats! Nice and creative approach :). Thanks for sharing and answering all the previous questions; that's very interesting. I have a few addition questions that might interest the community as well; please have a look:
\n- How did you come up with this "field-aware" version of FM? Is it performing much better than Rendle's original approach on this problem? Do you have any thoughts on why this is the case?
- What are you doing with simple features? (They are not in the formula of phi(w,x) on slide 14)?
- Why did not you include interactions terms xj * xj (when j1 == j2 on slide 14)?
Thanks!
\n[quote=guestwalk;55762]
\nDear all,
\nWe have prepared our codes and documents.
\nFor the codes, please see here; and for the documents, please see here.
\nYour comments are very welcome. Thanks!
\n-- 3 Idiots
\n[/quote]
\n[/quote]
" }, { "id": 56217, "postDate": "2014-10-18T08:20:10.440Z", "content": "Sorry for the late reply!
\nFor your question, I don't think it is a binary classification problem. That
is, from some points of view, they are generic; but from some others, they are
customized.
It is generic because the input to these solvers are just a label vector and a
matrix. Therefore, for any problem, as long as it can be transformed into the
required format, our solvers work.
On the other hand, it is customized because:
\n1. So far we only support binary sparse matrices (sparse matrices whose values
are either zero or one). Actually it is not difficult to support generic
sparse matrices. Two main reasons we support binary sparse matrices only are
that it consumes less memory and that it runs faster.
2. There are some usability issues. For example, we do not save model, and
test set is forced to be specified during training.
To make these solvers more generic, I think there is still a long way to go.
Though it is not difficult to address these issues, it definitely needs some
hard works.
BTW, we just wrote README to introduce the input file formats of GBDT and
FM. We are still improving the documents and the codes.
[quote=saurk;56066]
\nCongratulations ! Wondering if the FM and GBDT source code you have here is generic or it's customized to Criteo problem ? If generic, maybe we can use this instead of R or Python's GBM packages.
\n[/quote]
" }, { "id": 56198, "postDate": "2014-10-18T06:39:36.077Z", "content": "Hi!
\nCongrats! Nice and creative approach :). Thanks for sharing and answering all the previous questions; that's very interesting. I have a few addition questions that might interest the community as well; please have a look:
\n- How did you come up with this "field-aware" version of FM? Is it performing much better than Rendle's original approach on this problem? Do you have any thoughts on why this is the case?
- What are you doing with simple features? (They are not in the formula of phi(w,x) on slide 14)?
- Why did not you include interactions terms xj * xj (when j1 == j2 on slide 14)?
Thanks!
\n[quote=guestwalk;55762]
\nDear all,
\nWe have prepared our codes and documents.
\nFor the codes, please see here; and for the documents, please see here.
\nYour comments are very welcome. Thanks!
\n-- 3 Idiots
\n[/quote]
" }, { "id": 56066, "postDate": "2014-10-14T20:13:00.173Z", "content": "Congratulations ! Wondering if the FM and GBDT source code you have here is generic or it's customized to Criteo problem ? If generic, maybe we can use this instead of R or Python's GBM packages.
" }, { "id": 56009, "postDate": "2014-10-13T09:52:42.617Z", "content": "I see. Many thanks for your comments! :)
\n[quote=Tianqi Chen;55966]
\nHaving to do early stopping seems to be a very common thing that can not be avoided in factorization model. Sometimes using an alternative regularization(e.g max-norm on latent factor) can make the long tail effect comes slower, however it will still come, and the best solution is usually worse than what you can have in optimal iteration.
\nAmong the factorization models, bayesian version is stable to long-tail effect, however they are not practical(at least not the plain version) because you have to store all the generated samples for prediction. I have no idea how this can be avoided easily:) Maybe using other stable model to calibrate what is generated from FM models...
\nThe regularization you use appears to be the common one used by MF folks. Since lambda is applied only when nonzero feature occurs. It is not standard L2 regularizer. It is a weighed L2 regularization where weight on feature i is lambda times number of nonzero feature i
\nThanks for answering so many questions! I learned a lot from your solution
\n[/quote]
" }, { "id": 55952, "postDate": "2014-10-12T06:54:36.037Z", "content": "For the questions about the sensitivity and the situation after the best
iteration, please see the attachment. (tr_loss and va_loss respectively
represent the logloss on the training and the validation set.) I do this on our
own validation set; also, I removed GBDT features so that the experiments can
run faster.
We can see the setting eta = 0.1 indeed achieve the best performance on the
validation set. For eta = 0.2, eta = 0.05, and eta = 0.02, a similar
performance can be achieved. The smaller eta is, the more iterations are
required to achieve a good performance; it is the classic SGD behavior. If eta
is too large (in this case, 0.5), then the result is significantly worse; for
the setting eta = 0.01, it do not converge at 100th iterations. My opinion is
choosing a good eta is still crucial to SGD. For a large eta, the accuracy can
be hurt; for a small eta, it takes a long time to converge.
On the other hand, the "long tail" iterations indeed hurts a lot. In fact, we
also used this learning rate schedule to solve the simple matrix factorization
(MF) problem, and the performance in the long tail iterations is quite stable
(i.e., close to the best performance). I think the difference may be that FM is
much complicated compared to MF, so the overfitting problem is much harder to
be avoided. (In some sense this FM problem is a combination of 39*38/2 MF
problem, given there are 39 features for each impression.) This problem is
really annoying because it makes our solution less useful in practice. Any
suggestion?
For regularization, our implementation is very simple. Every time we choose an
instance, we only update the corresponding coordinate of the non-zero features.
For example, if the non-zero coordinates of an instance x are 376, 248, 571,
and 942, then only w376, w248, w571, and w942 are updated. Take w248 as an
example, the corresponding gradient is
g248 = lambda*w248 + (the gradient of the loss term)
\nNote that lambda is a fixed parameter, we don't change it during training at
all. Not sure if it is one of the approaches you mentioned.
Finally, the gradient of regularization is also accumulated.
\n[quote=Tianqi Chen;55931]
\nInteresting! This is the first time I see AdaGrad to work on factorization models.
\nSometimes it works great for convex models and some neuralnets. But I wasn't able get it work better than constant learning rate or decayed learning rate on factorization models.
\nHow sensitive is the result to the initial learning rate, I suspect it is just for lower rate, you run for longer rounds, but still get similar results?
\nI get your point about choosing the stopping condition. Since you use AdaGrad, I guess the raising up period after optimal point will be more smoother? Sometimes Adagrad in convex models tends to less overfit, say 50 rounds gets you there, and running for 70 rounds won't hurt too much. I am wondering what is in your case.
\nFor L2 regularizer, how do you implement it? If it is standard L2 regularizer, you don;t want to regularize parameters every rounds, so either some lazy trick or divide the regularizer by #occurance can be used. In many MF codes, sometimes ppl simply take another regularizer lambda #occraunce_of_fi w_i^2 and run weight decay every time you see the parameter.
\nDoes the gradient of regularizer goes into the accumulated gradient?
\nTianqi
\n[/quote]
" }, { "id": 55906, "postDate": "2014-10-10T20:59:22.207Z", "content": "[quote=guestwalk;55898]
\nNot sure if you think the questions are answered. Please let me know if it is
not clear. :)
[/quote]
\n\n1) I think this helped a lot. I was confused what the hashing trick was doing. I was thinking perhaps the value of a feature, say 5a9ed9b0, was REPLACED by an integer. So, I understand now that one-hot-encoding is still being used, it is just the indexing of the data which is improved (memory wise) when hashed.
\n\n2) So this is essentially the same model as shown in the Rendle paper http://www.ismll.uni-hildesheim.de/pub/pdfs/Rendle2010FM.pdf (equation 1) , where you used k=2 (the number of factors)? Is this correct?
\n\nThanks very much!!
" }, { "id": 55862, "postDate": "2014-10-09T22:35:57.823Z", "content": "[quote=guestwalk;55762]
\nDear all,
\nWe have prepared our codes and documents.
\nFor the codes, please see here; and for the documents, please see here.
\nYour comments are very welcome. Thanks!
\n-- 3 Idiots
\n[/quote]
\n\n\nFor those of us (me ;-)) just starting to learn about FM, hope you don't mind a couple elementary questions.
\n1) Are you only using numeric variables in the model - no one-hot encoding at all? Everything is hashed to an integer?
\n
2) On Slide #14-15 I am unsure what {x_j1, x_j2 } and {wj1,f2, wj2,f1} are. Would you explain?
Excellent job! This is about innovation. Deep learners claim that GBDT having 3-level of depth. (linear models have 1 level. others with nonlinear feature generation have depth of 2 such as polynomial kernel, decision tree). I thought how about using GBDT for feature generation before, but didn't get workable ideas. BTW, C11 looks nicer.
" }, { "id": 372895, "postDate": "2018-08-20T15:19:06.467Z", "rawMarkdown": "", "isDeleted": true } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T14:44:23Z", "competition_id": 3934, "competition_slug": "criteo-display-ad-challenge", "team_id": 111677, "solution_url": "/competitions/criteo-display-ad-challenge/writeups/3-idiots-3-idiots-solution-libffm" }