Dear all,
\nWe have prepared our code and document.
\nIn this competition and Criteo's CTR competition, we find field-aware factorization machine (FFM) is a powerful model. We hope this model can be more widely used, so we also release an implementation LIBFFM. We are very interested to see if it is effective in other kinds of datasets.
\nYour questions or comments are welcome! :)
\n-- 4 Idiots
", "messages": [ { "id": "64863", "postDate": "02/25/2015 14:20:42", "content": "Dear all,
\nWe have prepared our code and document.
\nIn this competition and Criteo's CTR competition, we find field-aware factorization machine (FFM) is a powerful model. We hope this model can be more widely used, so we also release an implementation LIBFFM. We are very interested to see if it is effective in other kinds of datasets.
\nYour questions or comments are welcome! :)
\n-- 4 Idiots
", "rawMarkdown": "", "votes": null }, { "id": "64865", "postDate": "02/25/2015 14:36:59", "content": "Thank you very much! I have a question :D Why is the FFM particularly good for this problem? What data structure/characteristic in general is FFM suitable for ? Thank you and congratulations again!
", "rawMarkdown": "", "votes": null }, { "id": "64869", "postDate": "02/25/2015 15:11:11", "content": "Awesome - thank you for sharing!
\n\nOne basic question up front : Is a field aware FM the same as a FM (e.g. LIBFM) trained on data set that has a distinct name spaces for every predictor (using 'name space' to mean the role played by this concept in Vowpal Wabbit)? So for example if there were two predictors in a data set:
\nA which takes values 'RED' and 'BLUE'
\nB which also takes values 'RED' and 'BLUE'
\n\nInstead of having data that hashed RED and BLUE to the same value for both A and B, the data would be A_RED or B_RED thus properly treating them as two separate things? Or is there more to it? :)
", "rawMarkdown": "", "votes": null }, { "id": "64871", "postDate": "02/25/2015 15:59:02", "content": "[quote=rcarson;64865]
\nWhat data structure/characteristic in general is FFM suitable for?
\n[/quote]
\nFrom Rendle's paper: FM work well on datasets with a huge sparsity, like this clicklog dataset, recommender datasets, and also text (bag-of-words).
\nThanks to 3 4 Idiots for releasing LIBFFM. Will play around with it and report back.
[quote=Triskelion;64871]
\nFrom Rendle's paper: FM work well on datasets with a huge sparsity, like this clicklog dataset, recommender datasets, and also text (bag-of-words).
\nThanks to 3 Idiots for releasing LIBFFM. Wil play around with it and report back.
\n[/quote]
\nThank you very much! That's an excellent paper :D
", "rawMarkdown": "", "votes": null }, { "id": "64903", "postDate": "02/26/2015 01:41:04", "content": "Excellent work indeed!
\nAnd Thanks for the FFM library
", "rawMarkdown": "", "votes": null }, { "id": "64910", "postDate": "02/26/2015 03:41:17", "content": "[quote=rcarson;64872]
\n[quote=Triskelion;64871]
\nFrom Rendle's paper: FM work well on datasets with a huge sparsity, like this clicklog dataset, recommender datasets, and also text (bag-of-words).
\nThanks to 3 Idiots for releasing LIBFFM. Wil play around with it and report back.
\n[/quote]
\nThank you very much! That's an excellent paper :D
\n[/quote]
\nThanks! Here are my opinions for your questions:
\n1. As Triskelion has mentioned, I agree with section IV in Rendle's paper is a
good explanation for the reason why FM-based models can be better than
SVM-based models on sparse datasets.
2. For this question, I have no idea. So far I have only used FFM on CTR
datasets. I am also very interested to see if it works on other types of
data. That's why we released LIBFFM. :D
[quote=guestwalk;64863]
\nIn this competition and Criteo's CTR competition, we find field-aware factorization machine (FFM) is a powerful model. We hope this model can be more widely used, so we also release an implementation LIBFFM. We are very interested to see if it is effective in other kinds of datasets.
\n[/quote]
\nHi, guestwalk,
\nThank you very much for sharing and congratulation! I would like to ask the question about implementation: in stochastic gradient descent (file ffm.cpp, function wTx) you used the update w = w - lr * \\sqrt(1+sum of squares of previous gradients) * gradient. I am wondering why you used this factor \\sqrt(1+sum of squares of previous gradients)? Maybe you could give some links for the description of this idea?
\nDmitry.
", "rawMarkdown": "", "votes": null }, { "id": "64938", "postDate": "02/26/2015 12:08:53", "content": "[quote=Dmitry Efimov;64936]
\n[quote=guestwalk;64863]
\nIn this competition and Criteo's CTR competition, we find field-aware factorization machine (FFM) is a powerful model. We hope this model can be more widely used, so we also release an implementation LIBFFM. We are very interested to see if it is effective in other kinds of datasets.
\n[/quote]
\nHi, guestwalk,
\nThank you very much for sharing and congratulation! I would like to ask the question about implementation: in stochastic gradient descent (file ffm.cpp, function wTx) you used the update w = w - lr * \\sqrt(1+sum of squares of previous gradients) * gradient. I am wondering why you used this factor \\sqrt(1+sum of squares of previous gradients)? Maybe you could give some links for the description of this idea?
\nDmitry.
\n[/quote]
\nHi Dmitry, this learning-rate schedule is described in our recent paper. Hope it is useful for you.
", "rawMarkdown": "", "votes": null }, { "id": "64943", "postDate": "02/26/2015 13:01:00", "content": "Great! Thanks!
\nOne more question, could you explain a little bit more about parameter "v" you are using for prediction? If I understood correctly, you need this parameter to adjust weights only (to avoid them to be very small). Is it correct? Or you need this parameter for something else?
", "rawMarkdown": "", "votes": null }, { "id": "64999", "postDate": "02/26/2015 22:47:32", "content": "[quote=Dmitry Efimov;64943]
\nGreat! Thanks!
\nOne more question, could you explain a little bit more about parameter "v" you are using for prediction? If I understood correctly, you need this parameter to adjust weights only (to avoid them to be very small). Is it correct? Or you need this parameter for something else?
\n[/quote]
\nSorry I don't quite understand. Do you mean "ffm_float v" in "struct ffm_node"??
", "rawMarkdown": "", "votes": null }, { "id": "65009", "postDate": "02/27/2015 05:15:39", "content": "Yes, right! In your previous version it was 2/nr_field. In new version you defined it as 2.0f*v1*v2*r in function wTx, file ffm.cpp.
", "rawMarkdown": "", "votes": null }, { "id": "65016", "postDate": "02/27/2015 10:06:07", "content": "Okay I see!
\n1. "v1" and "v2" respectively are x_j1 and x_j2 in page 6 in this slide.
\n2. "r" is the normalization constant, when auto-normalization is not enabled, it is always 1; and if it is enabled, we automatically normalize each instance. For example, if we have x = "1:3:1 2:7:2 3:9:1.5", and normalization is enabled, then r = 1/sqrt(1*1+2*2+1.5*1.5).
\n3. For the constant "2.0f." Sorry I don't have a good explanation now. Actually FFM still works if this constant is removed, but empirically I find adding this constant makes the convergence faster. One possible reason is that FFM is used to model interactions between features. If we consider polynomial models, for instance, (x1+x2+x3)^2 = x1^2+x2^2+x3^2+2*x1*x2+2*x1*x3+2*x2*x3. The interaction terms have "2" here, so I also have this "2" in FFM.
\nPlease feel free to let me know if the explanations above are not clear.
", "rawMarkdown": "", "votes": null }, { "id": "65021", "postDate": "02/27/2015 11:37:32", "content": "Yes, it is clear now, thank you very much! This is really great work!
", "rawMarkdown": "", "votes": null }, { "id": "65052", "postDate": "02/27/2015 17:26:21", "content": "[quote=Inspector;64869]
\nAwesome - thank you for sharing!
\nOne basic question up front : Is a field aware FM the same as a FM (e.g. LIBFM) trained on data set that has a distinct name spaces for every predictor (using 'name space' to mean the role played by this concept in Vowpal Wabbit)? So for example if there were two predictors in a data set:
\nA which takes values 'RED' and 'BLUE'
\nB which also takes values 'RED' and 'BLUE'
\nInstead of having data that hashed RED and BLUE to the same value for both A and B, the data would be A_RED or B_RED thus properly treating them as two separate things? Or is there more to it? :)
\n[/quote]
\nSorry I still don't quite understand what "name space" is. In your example, my understanding is: "A" and "B" are predictors, and "RED" and "BLUE" are values. Assuming my understanding is correct, in your first question you have the term "name spaces for every predictors." So in your example, what are namespaces for "A" and "B," respectively? Thanks! :)
", "rawMarkdown": "", "votes": null }, { "id": "65099", "postDate": "02/28/2015 03:33:11", "content": "Congratulations, thank you for sharing your solution. But I have some concerns about the FFM model. In practice, especially when use it in prediction, this model need too many parameters for sparse features. In standard FM model, each sparse feature only has one dense vector of latent factors. But in FFM model, each sparse feature have N(number of feature field) - 1 dense vectors to be stored. If both of the dimension of sparse features and number of feature field are big, the storage of the FFM model in single machine will be challenge in model training and online prediction. Do you have any advice for this problem?
\nThanks again!
", "rawMarkdown": "", "votes": null }, { "id": "65105", "postDate": "02/28/2015 05:14:11", "content": "[quote=guestwalk;64863]
\nDear all,
\nWe have prepared our code and document.
\nIn this competition and Criteo's CTR competition, we find field-aware factorization machine (FFM) is a powerful model. We hope this model can be more widely used, so we also release an implementation LIBFFM. We are very interested to see if it is effective in other kinds of datasets.
\nYour questions or comments are welcome! :)
\n-- 4 Idiots
\n[/quote]
\n\nwhat purpose of bag features?
", "rawMarkdown": "", "votes": null }, { "id": "65143", "postDate": "02/28/2015 18:22:48", "content": "[quote=lynn;65099]
\nCongratulations, thank you for sharing your solution. But I have some concerns about the FFM model. In practice, especially when use it in prediction, this model need too many parameters for sparse features. In standard FM model, each sparse feature only has one dense vector of latent factors. But in FFM model, each sparse feature have N(number of feature field) - 1 dense vectors to be stored. If both of the dimension of sparse features and number of feature field are big, the storage of the FFM model in single machine will be challenge in model training and online prediction. Do you have any advice for this problem?
\nThanks again!
\n[/quote]
\nYes, you are right. The training / prediction time of FFM is usually greater than FM, especially when the number of fields is large.
\nHowever, the difference may not be as large as you think. In my experience, the best number of latent factors (k) for FFM is much smaller than that for FM. For example, if I remember correctly, in Criteo's competition, the best k for FFM is 8, and the best k for FM is 100.
", "rawMarkdown": "", "votes": null }, { "id": "65144", "postDate": "02/28/2015 18:27:19", "content": "[quote=earthMovingEquipment;65105]
\n[quote=guestwalk;64863]
\nDear all,
\nWe have prepared our code and document.
\nIn this competition and Criteo's CTR competition, we find field-aware factorization machine (FFM) is a powerful model. We hope this model can be more widely used, so we also release an implementation LIBFFM. We are very interested to see if it is effective in other kinds of datasets.
\nYour questions or comments are welcome! :)
\n-- 4 Idiots
\n[/quote]
\nwhat purpose of bag features?
\n[/quote]
\nThe concept behind this feature is that if some users have similar app_id (or something else), then these users may have some similarity.
", "rawMarkdown": "", "votes": null }, { "id": "65151", "postDate": "02/28/2015 21:44:02", "content": "I tried to build LibFFM on Windows, but had to give up and switch to Ubuntu. Install and usage was easy. I could not get it to work on text tokens from longer documents. I guess text tokens do not really have a field (or it should be word order).
\nI tried with static fields
\n1 0:token_id:1 0:token_id:1 0:token_id:1 0:token_id:1 (seems to take only first token_id for learning)
\nand incrementing fields
\n1 0:token_id:1 1:token_id:1 2:token_id:1 3:token_id:1 ... n:token_id:1 (first 38 words of document)
\nHowever this field characteristic is still very interesting, I' ll have to try it on other highly dimensional sparse datasets with multiple columns instead of raw text columns (maybe PCA first?).
", "rawMarkdown": "", "votes": null }, { "id": "65169", "postDate": "03/01/2015 07:31:18", "content": "[quote=Triskelion;65151]
\nI tried to build LibFFM on Windows, but had to give up and switch to Ubuntu. Install and usage was easy. I could not get it to work on text tokens from longer documents. I guess text tokens do not really have a field (or it should be word order).
\nI tried with static fields
\n1 0:token_id:1 0:token_id:1 0:token_id:1 0:token_id:1 (seems to take only first token_id for learning)
\nand incrementing fields
\n1 0:token_id:1 1:token_id:1 2:token_id:1 3:token_id:1 ... n:token_id:1 (first 38 words of document)
\nHowever this field characteristic is still very interesting, I' ll have to try it on other highly dimensional sparse datasets with multiple columns instead of raw text columns (maybe PCA first?).
\n[/quote]
\nSorry we do not support Windows at this stage. Thanks very much for giving me
feedback! I have some ideas:
1. I don't think your first setting (setting all fields to 0) will work. The
reason is that we do not allow interactions within the same field. So
basically you are not training anything. If you want to use this setting,
you can change
if(j2 >= model.n || f2 >= model.m || f1 == f2)
\nto
\nif(j2 >= model.n || f2 >= model.m)
\nYou can find this code in ffm.cpp. I think it will work. However, because
all features are within the same field, FFM reduces to FM in this case.
2. There is no strong field concepts in document datasets. FFM may not work
very well in this kind of dataset..
3. Did you use ``--norm'' to enable normalization? If not, I suggest you to
try it. In my experience with normalization, the result can be much better.
how to convert seamlessly any .csv file into the format of Libffm?
", "rawMarkdown": "", "votes": null }, { "id": "65188", "postDate": "03/01/2015 18:00:50", "content": "Thanks for sharing!
\nCan you explain, what is the difference between FFM and PITF?
", "rawMarkdown": "", "votes": null }, { "id": "65259", "postDate": "03/02/2015 16:50:32", "content": "[quote=Herimanitra;65170]
\nhow to convert seamlessly any .csv file into the format of Libffm?
\n[/quote]
\nWe do not have such a tool. Users have to convert from raw data to LIBFFM format by by themselves.
", "rawMarkdown": "", "votes": null }, { "id": "65260", "postDate": "03/02/2015 16:51:56", "content": "[quote=Mikhail Trofimov;65188]
\nThanks for sharing!
\nCan you explain, what is the difference between FFM and PITF?
\n[/quote]
\nI don't know what PITF is. May you provide a reference? Thanks!
", "rawMarkdown": "", "votes": null }, { "id": "65267", "postDate": "03/02/2015 17:07:17", "content": "[quote=guestwalk;65260]
\n[quote=Mikhail Trofimov;65188]
\nThanks for sharing!
\nCan you explain, what is the difference between FFM and PITF?
\n[/quote]
\nI don't know what PITF is. May you provide a reference? Thanks!
\n[/quote]
\nOh, you are talking about this paper. These two models can be equivalent. This paper is for recommender systems, so there are three fields "user," "item," and "tag." (see equation 12) In FFM we don't have such restriction.
", "rawMarkdown": "", "votes": null }, { "id": "65283", "postDate": "03/02/2015 19:35:26", "content": "Congratulations..
\nDid you change the code a lot since the Criteo version ?
I compiled that one to win64 for this competition.
Perhaps I could build a Windows version again.
Also, as a completely non-scientific experiment I also added the term where feature values were multiplied against themselves. (something like (w[jn,fm],w[jn,fm]) xjn*xjn if xn was present for fm.)
I thought it would be an approximation for the normal SUM(xn * wn)first order terms in the original FM. Halfway the competition I tried it and it gave slightly better results so I kept it in.
Perhaps you can say something sensible about it. Did you consider adding this term or is it complete non-sense. :)
Congratulations and thanks for releasing LIBFFM.
\nI looked at the slides, but I'm still not clear what's the difference between FM and FFM. Is it that FFM for each variable have more than one vector of factors (separate vector for each field)?
", "rawMarkdown": "", "votes": null }, { "id": "65324", "postDate": "03/03/2015 11:27:31", "content": "[quote=Julian de Wit;65283]
\nCongratulations..
\nDid you change the code a lot since the Criteo version ?
I compiled that one to win64 for this competition.
Perhaps I could build a Windows version again.
Also, as a completely non-scientific experiment I also added the term where feature values were multiplied against themselves. (something like (w[jn,fm],w[jn,fm]) xjn*xjn if xn was present for fm.)
I thought it would be an approximation for the normal SUM(xn * wn)first order terms in the original FM. Halfway the competition I tried it and it gave slightly better results so I kept it in.
Perhaps you can say something sensible about it. Did you consider adding this term or is it complete non-sense. :)
[/quote]
\n1. For the Windows part: We will make our code compilable on Windows soon. However, we are not planning to release a Windows binary yet.
\n2. For the term you added: Yes, I did exactly the same thing before. :) My experiment results was with/without this term make almost no difference, so I do not include this term in LIBFFM.
", "rawMarkdown": "", "votes": null }, { "id": "65334", "postDate": "03/03/2015 14:01:21", "content": "I'm feeling like libFFM is just another implementation of libFM by just using second order interaction and having a second index for each 'field' and avoid to have an index shifting for different 'fields'.
But then you have cross-validation build-in and multi-thread more.
HI!
Congratulations!
guestwalk, I have few questions:
Can you explain why you use exactly this categorical variables
,'C14','C17','C20','C21', but not another anonymized categorical variables?
And also how you find which variables you should use?
And how did you choose condition for subsetting?
And also, sorry but I did not find what did you do after hashing? Did you after that use hot-one-encoding of hashed values for get variables or it is already variable for FFM?
@ThierryS,
\nFFM is not FM in formulation, algorithm, as well as implementation. You can try to use FM to achieve the same score.
\n\n1. Formulation
\nIn FFM, one feature has distinct vectors to interact with features from different fields while FM assigns one feature only one vector. Conceptually, FFM does not mess interactions between device_ip---app_id and device_ip--app_domain up, but FM does.
\n\n2. Algorithm
\nOur algorithm is based on our recent research. See the attachement.
", "rawMarkdown": "", "votes": null }, { "id": "65610", "postDate": "03/06/2015 16:10:10", "content": "Hi, I'm still trying to understand field and index of a feature. My understanding is that given a categorical feature, say device_id, has N unique values, id-1, id-2, .. id-N. So, device_id, is the field and id-i is index. and the value of the corresponding encoded or hashed feature is 0 or 1, indicating the value id-i appears or not in device_id. Am I right? If so, that means all features in the end are binary but how could the example in this slide has numerical features in the end?
\nlike 1:121:0.3, field:index:value in page 6? Thank you very much.
", "rawMarkdown": "", "votes": null }, { "id": "65715", "postDate": "03/08/2015 18:52:43", "content": "Hi, there. I am running the bigdata example coming with libffm. What do the features in bigdata.tr.txt mean originally? like the first line,1 0:0:0.3651 2:1163:0.3651 3:8672:0.3651
\nWhat are the field and index standing for in this case? Thank you very much.
", "rawMarkdown": "", "votes": null }, { "id": "65728", "postDate": "03/08/2015 21:19:19", "content": "[quote=rcarson;65610]
\nHi, I'm still trying to understand field and index of a feature. My understanding is that given a categorical feature, say device_id, has N unique values, id-1, id-2, .. id-N. So, device_id, is the field and id-i is index. and the value of the corresponding encoded or hashed feature is 0 or 1, indicating the value id-i appears or not in device_id. Am I right? If so, that means all features in the end are binary but how could the example in this slide has numerical features in the end?
\nlike 1:121:0.3, field:index:value in page 6? Thank you very much.
\n[/quote]
\nI think you can do things like movies_streamed:movie_id:vote dvds_rented:movie_id:vote.
\nI use it for when I do not want to onehot encode certain columns with floats. (meta_pred_35,0.4355432) becomes something like 35:0:0.4355432. The index for field 35 is always 0 but the value can differ (and I hope FFM can learn from this).
", "rawMarkdown": "", "votes": null }, { "id": "65753", "postDate": "03/09/2015 05:09:47", "content": "@Foxtrot, ThierryS, rcarson
\nWe understand that the concept of fields is not clear in the slides you checked. We have updated it. There is a concrete example in the updated version. Please check it again. If you still have questions, please re-post your questions and I will reply. Thanks!
", "rawMarkdown": "", "votes": null }, { "id": "65764", "postDate": "03/09/2015 09:34:23", "content": "Very clear thank you for the update.
", "rawMarkdown": "", "votes": null }, { "id": "65784", "postDate": "03/09/2015 15:13:23", "content": "[quote=KamilBek;65405]
\nHI!
Congratulations!
guestwalk, I have few questions:
Can you explain why you use exactly this categorical variables
,'C14','C17','C20','C21', but not another anonymized categorical variables?
And also how you find which variables you should use?
And how did you choose condition for subsetting?
And also, sorry but I did not find what did you do after hashing? Did you after that use hot-one-encoding of hashed values for get variables or it is already variable for FFM?
[/quote]
\nThanks. Here are my answers to your questions.
\n1. In all following experiments, to measure performance, we use the last day of training data as validation set and the other days as training set.
\n2. For variable selection: We select variables by running experiments on a subsampled dataset. The size is 10% of the whole training data. We use all features as a baseline, and if removing a certain feature gives us the same or even better result, then we remove this feature.
\n3. For subset:
\na. We first select top 50 or so frequent features.
b. We do not consider those anonymized features because we completely have no idea what they are.
c. For each feature, we try to run experiments on the subset that contains these feature. If after ensemble there is an improvement, then we add this subset into our model pool.
4. For the question what we do after hashing: After hashing, we feed the hashed data to FFM. You can find the flowchart on page 5 of these slides.
", "rawMarkdown": "", "votes": null }, { "id": "66007", "postDate": "03/12/2015 06:12:57", "content": "Congratulations and thanks for your sharing.
\nI have one question: if I get it right, FFM only models the interaction between variables. Will the performance be improved if you include the bias term for each variable in the model?
", "rawMarkdown": "", "votes": null }, { "id": "66025", "postDate": "03/12/2015 09:05:52", "content": "[quote=Jianfeng Zhu;66007]
\nCongratulations and thanks for your sharing.
\nI have one question: if I get it right, FFM only models the interaction between variables. Will the performance be improved if you include the bias term for each variable in the model?
\n[/quote]
\nYes, I have tried this. In my experience adding these terms or not do not
affect the loss on the test set a lot. In Random Walker's solution they have
the linear terms and the result is slight better according to their
observation.
Congratulations to Yu-Chin, Wei-Sheng, Yong and Michael!
\nThere have been several questions about the relationship between FM and FFM. Here are my thoughts about the differences and similarities.
\nNotation
\nModels (slightly simplified)
\n y(x) = sum_i sum_j>i 〈v_i,v_j〉 x_i x_j
y(x) = sum_i sum_j>i 〈v^J_i,v^I_j〉 x_i x_j
The difference between both models is that FFM assumes that the factors between interactions (e.g. v_i of (I,J) and v_i of (I,L)) are independent whereas FM uses a shared parameter space.
\nNumber of parameters and costs
\nThat means from a cost point of view, an FFM with dimensionality k' should be compared to an FM with an m times larger dimension, i.e. k=k'*m. With this choice both FFM and FM have the same number of parameters (memory costs) and the same runtime complexity.
\nExpressiveness
\nFFM and FM have different assumptions on the interaction matrix V. But given a large enough k and k', both can represent any possible second order polynomial model.
\nThe motivation of FM and FFM is to approximate the (unobserved) pairwise interaction matrix W of polynomial regression by a low rank solution V*V^t = W. FM and FFM have different assumptions how V looks like. FFM assumes that V has a block structure:
\n
| v^2_1 v^1_2 0 0 0 |
| v^3_1 0 v^1_3 0 0 |
| v^4_1 0 0 v^1_4 0 |
V(FFM) = | v^5_1 0 0 0 v^1_5 |
| 0 v^3_2 v^2_3 0 0 |
| 0 v^4_2 0 v^2_4 0 |
| ... |
FM does not assume such a structure:
\nV(FM) = | v_1 v_2 v_3 v_4 v_5 |
(Note that v are not scalars but vectors of length k' (for FFM) or of length k (for FM). Also to shorten notation, one entry v in the matrices above represents all the v vectors of a "field"/ categorical variable.)
\nIf the assumption of FFM holds, then FFM needs less parameters than FM to describe V because FM would need parameters to capture the 0s.
\nIf the assumption of FM holds, then FM needs less parameters than FFM to describe V because FFM would need to repeat values of vectors as it requires separate parameters.
", "rawMarkdown": "", "votes": null }, { "id": "69699", "postDate": "04/05/2015 08:05:29", "content": "In case if somebody is interested I have posted the R slidify presentation about my solution:
\nhttp://mech.math.msu.su/~efimov/2015_04_ICMS_Click/html/index.html#/slide-1
", "rawMarkdown": "", "votes": null }, { "id": "69714", "postDate": "04/05/2015 15:05:20", "content": "@Steffen: Great comments! In my experience even with k = k'*m, FM is usually faster probably because of better memory locality.
\n@Dmitry: Very interesting to see the improvement by sorting data with different features.
", "rawMarkdown": "", "votes": null }, { "id": "71672", "postDate": "04/15/2015 09:11:05", "content": "Congratulations! Thanks for your sharing the solution. It is a great work!
\nI have two questions:
\n1) The 3 Idiots' solution in criteo includes GBDT to generate additional features which is feed to FFM. But it is omitted in this solution. I guess it is FFM that make most contribution and GBDT help a little to improve the score. Right?
\n2) Although both of solutions adapted FFM, the feature engineering methods differ (one use GBDT, the other use bag of words...). What is your basic idea or inspiration to choose different approach?
\nThanks again,
\nSteven
", "rawMarkdown": "", "votes": null }, { "id": "72040", "postDate": "04/17/2015 02:39:01", "content": "[quote=LifeMatrix;71672]
\nCongratulations! Thanks for your sharing the solution. It is a great work!
\nI have two questions:
\n1) The 3 Idiots' solution in criteo includes GBDT to generate additional features which is feed to FFM. But it is omitted in this solution. I guess it is FFM that make most contribution and GBDT help a little to improve the score. Right?
\n2) Although both of solutions adapted FFM, the feature engineering methods differ (one use GBDT, the other use bag of words...). What is your basic idea or inspiration to choose different approach?
\nThanks again,
\nSteven
\n[/quote]
\nHi Steven,
\nFor you questions:
\n1. The main reason we don't use GBDT in this competition is that there is no numeric features. (Tree based models are conceptually more suitable for numerical features.)
\n2. Sorry I do not have a good answer to this question. In my opinion, sometimes feature engineering is just intuition and trial-and-error.
\nYu-Chin
", "rawMarkdown": "", "votes": null }, { "id": "74765", "postDate": "04/25/2015 22:29:25", "content": "You referenced a paper on stochastic gradient descent (mf_adaptive_pakdd.pdf), but it doesn't seem like the libFM uses SGD. Have you thought about implement libFFM using SGD?
", "rawMarkdown": "", "votes": null }, { "id": "74769", "postDate": "04/25/2015 22:54:25", "content": "Yeah. We tried it. Our algorithm is exactly a sort of SG but the convergence rate of vanilla SG is slow.
", "rawMarkdown": "", "votes": null }, { "id": "74777", "postDate": "04/26/2015 00:12:31", "content": "I haven't gone thru your libFFM code, but I see the double for loop in the train() method, which make me think you are doing direct matrix manipulations. I tried doing it using SGD, but there is fundamental difficulty in the math. If you recall the derivation in Rendle's original paper (see attached), he was able to factor out vi,f from vj,f because i and j are independent. However, in ffm, vi,f is actually vi,fieldof(j),f. So the coefficients can not be separated into distinct summations. In this case, I don't see how you can derive a gradient descent method which is O(N). Would you mind give me an hint?
", "rawMarkdown": "", "votes": null }, { "id": "74787", "postDate": "04/26/2015 01:05:01", "content": "The computational complexity per iteration in LIBFFM is not O(N), we do need to go through feature pairs once as their coefficients are independent.
", "rawMarkdown": "", "votes": null }, { "id": "74788", "postDate": "04/26/2015 01:08:06", "content": "So, we are in agreement. In fact the complexity is O(N square). That makes it a difficult algorithm to use in production. Except, or course, for features sets of about 10 vectors like in this project, this would not be a factor.
", "rawMarkdown": "", "votes": null }, { "id": "75819", "postDate": "05/01/2015 00:31:05", "content": "I am looking over how you selected your features. I already read thru the steps you listed on page 2. Some of the conditions you have in the code are truly bizarre looking. Of course, I have no doubt you had good reasons for choosing them. Here,
\nif is_app(row):
new_row['pub_id'] = row['app_id']
new_row['pub_domain'] = row['app_domain']
new_row['pub_category'] = row['app_category']
writer_app.writerow(new_row)
else:
new_row['pub_id'] = row['site_id']
new_row['pub_domain'] = row['site_domain']
new_row['pub_category'] = row['site_category']
writer_site.writerow(new_row)
You are not subsetting, but rather, you chose app_* over site_* whenever site_id == '85f751fd'. The chosen app_* and site_* values get lumped into the pub_* feature. I really can't imagine what is the reason behind that choice, and why it can lead to better results. Was there a reason for picking 85f751fd?
\nIn another example,
\nif int(row['device_ip_count']) > 1000:
feats.append(hashstr('device_ip-'+row['device_ip']))
else:
feats.append(hashstr('device_ip-less-'+row['device_ip_count']))
You change the hash input value based on the count of some original feature. How were you able to deduce such a threshold even experimentally? Could you give any theoretical basis such a distinction can lead to better prediction?
", "rawMarkdown": "", "votes": null }, { "id": "75983", "postDate": "05/01/2015 22:04:31", "content": "[quote=Bruce Ho;74788]
\nSo, we are in agreement. In fact the complexity is O(N square). That makes it a difficult algorithm to use in production. Except, or course, for features sets of about 10 vectors like in this project, this would not be a factor.
\n[/quote]
\nIt depends. We are very happy to learn the scenario of your production.
", "rawMarkdown": "", "votes": null }, { "id": "76033", "postDate": "05/02/2015 05:22:16", "content": "We are still experimenting with our feature set. The number of features should not be huge. However, the data set could easily run into millions, therefore, performance is still a consideration. We are definitely interested in how you picked your features, which we are having a hard time to understand.
", "rawMarkdown": "", "votes": null }, { "id": "76638", "postDate": "05/04/2015 16:52:39", "content": "[quote=Bruce Ho;75819]
\nI am looking over how you selected your features. I already read thru the steps you listed on page 2. Some of the conditions you have in the code are truly bizarre looking. Of course, I have no doubt you had good reasons for choosing them. Here,
\nif is_app(row):
new_row['pub_id'] = row['app_id']
new_row['pub_domain'] = row['app_domain']
new_row['pub_category'] = row['app_category']
writer_app.writerow(new_row)
else:
new_row['pub_id'] = row['site_id']
new_row['pub_domain'] = row['site_domain']
new_row['pub_category'] = row['site_category']
writer_site.writerow(new_row)
You are not subsetting, but rather, you chose app_* over site_* whenever site_id == '85f751fd'. The chosen app_* and site_* values get lumped into the pub_* feature. I really can't imagine what is the reason behind that choice, and why it can lead to better results. Was there a reason for picking 85f751fd?
\nWe do split the data set (see the different writers). '85f751fd' is considered to be a missing value since it's the most frequent value in this field. Basically, we assume that user behaviors should be different on different platform.
\nIn another example,
\nif int(row['device_ip_count']) > 1000:
feats.append(hashstr('device_ip-'+row['device_ip']))
else:
feats.append(hashstr('device_ip-less-'+row['device_ip_count']))
You change the hash input value based on the count of some original feature. How were you able to deduce such a threshold even experimentally? Could you give any theoretical basis such a distinction can lead to better prediction?
\nWe have it by experiments; for example, you can select the best one in {10, 100, 1000}. We don't have theoretical explanation. Sorry.
\n[/quote]
", "rawMarkdown": "", "votes": null }, { "id": "76922", "postDate": "05/05/2015 04:59:29", "content": "I am having some serious questions concerning the way you merge the results of app and site predictions.
\nIt seems that from the logic in GenData.py, app and site data files are two distinct sets with no overlap due to the filter condition if is_app(row):
\nso app files only has rows with site_id = '85f751fd', and the opposite is true for site files. After that two separate models are created for app and site files, generating two sets of predictions. Upon merging of the predictions, the id value from both file types is used to do the merge.
\nmprd[key] = logistic_func(sum(map(inv_logistic_func, mprd[key]))/len(mprd[key]))
\n\nHowever, based on the analysis given above there will be no intersection from the two sets at all. Therefor, no averaging is ever done. The only effect is combining the two distinct prediction sets into one.
\nI want to make sure I have not missed anything in reaching this conclusion.
", "rawMarkdown": "", "votes": null }, { "id": "80326", "postDate": "05/29/2015 01:23:45", "content": "Congrads to the Winner!
\n\nI have a question about the variable "history".
\nYou showed in the document that you used history like 0,01,011 and so on.
\nDid you use the features 0,01,011 as categorical variables? If now, how did you use them?
", "rawMarkdown": "", "votes": null }, { "id": "115948", "postDate": "04/20/2016 23:31:02", "content": "Sure this is a basic question here, but can someone explain the expected data format.
\n\n<label> <field1>:<index1>:<value1> <field2>:<index2>:<value2>\nGiven a dataset:
\n\nClick | Feature1 | Feature2 | Feature3 | \n1 | value | value | value |\nI understand <index1> would = the column index.
Does <field1> correspond to the row number?
Thanks,
", "rawMarkdown": "Sure this is a basic question here, but can someone explain the expected data format. \r\n\r\n