Manually setting my lowest predictions to -33.21928095 decreases LB error by 0.006... This was just an arbitrary random trial on my very lowest predictions to see what would happen. I guess finding the right threshold for this \"minimum loyalty score\" will probably have a much greater effect.
\n\nI could be wrong about this, and I'm obviously not the first person to have concerns, but I'm dubious about the competition in general. The final score is basically going to be based on how many outliers there are in the private set.
\n\nThis makes me worry that the top model might simply be the best classifier for extremely \"disloyal\" customers... Rather than an accurate regressor for true customer loyalty (whatever that is ¯\\_(ツ)_/¯).
\n\nI guess I'll just leave my best \"random number generator\" here and see how I do.
", "messages": [ { "id": 443824, "postDate": "2018-12-22T14:10:08.797Z", "content": "Hi DStjhb,\nI think, I can explain this phenomena. Disclaimer: it's only my thoughts and they are based only on my own experience in eCommerce.\nThere are 3 major metrics in eCommerce: Revenue Per Session(RPS=Revenue/Sessions), Average Order Value(AOV=Revenue/Orders) and Conversion(CONV=Orders/Sessions). As you can see from formulas, RPS=AOV*CONV.\nAOV has exponential distribution, CONV is Binomial. RPS distribution is a combination of Binomial distribution and exponential.\n![\"Image\"](\"https://i.ibb.co/f2Zjs8y/2018-12-21-12-37-02.png\")
\nI try to make an analogy with this competition:\nIn this competition we are trying to predict loyalty score. In my opinion(!!), it's something like \"sales from recs\" and it depends on how much money user spent(exponential distribution) and did he make transaction after getting recommendation(Binomial dist). So we have same combination of Binomial-Exponental Distribution.
My thoughts. I think true loyalty Elo Formula:\n1) Takes sales from recs;\n2) Divides each value by mean(mean --> 1);\n2) Adds some small random number and logs each value(mean --> 0).
\n\nAt least it can explain why we have \"extremely good\" normal distribution of target and large bin of guys with the same value(~-33.21). A lot of people considering them to be outliers but I think, they just spent 0(did not make any transaction) for all their history(or maybe last n month) for merchant, recommendation they got referred to.\nMy idea. First, make binary classifier([-33.21]-->0, [>-33.21]-->1) and try to classify users. Classes are imbalanced, so use AUC metric. After that, for class 1 make regression.
\n\nFood for more thought. Taking log from money can explain negative values of purchase_amount, but even in this way I'm still confused about extremely large values like 60k. np.exp(60k) gives me inf.
", "rawMarkdown": "Hi DStjhb,\nI think, I can explain this phenomena. Disclaimer: it's only my thoughts and they are based only on my own experience in eCommerce.\nThere are 3 major metrics in eCommerce: Revenue Per Session(RPS=Revenue/Sessions), Average Order Value(AOV=Revenue/Orders) and Conversion(CONV=Orders/Sessions). As you can see from formulas, RPS=AOV*CONV.\nAOV has exponential distribution, CONV is Binomial. RPS distribution is a combination of Binomial distribution and exponential.\n![Image][1]\nI try to make an analogy with this competition:\nIn this competition we are trying to predict loyalty score. In my opinion(!!), it's something like \"sales from recs\" and it depends on how much money user spent(exponential distribution) and did he make transaction after getting recommendation(Binomial dist). So we have same combination of Binomial-Exponental Distribution.\n\nMy thoughts. I think true loyalty Elo Formula:\n1) Takes sales from recs;\n2) Divides each value by mean(mean --> 1);\n2) Adds some small random number and logs each value(mean --> 0).\n\nAt least it can explain why we have \"extremely good\" normal distribution of target and large bin of guys with the same value(~-33.21). A lot of people considering them to be outliers but I think, they just spent 0(did not make any transaction) for all their history(or maybe last n month) for merchant, recommendation they got referred to.\nMy idea. First, make binary classifier([-33.21]-->0, [>-33.21]-->1) and try to classify users. Classes are imbalanced, so use AUC metric. After that, for class 1 make regression.\n\nFood for more thought. Taking log from money can explain negative values of purchase_amount, but even in this way I'm still confused about extremely large values like 60k. np.exp(60k) gives me inf.\n\n[1]: https://i.ibb.co/f2Zjs8y/2018-12-21-12-37-02.png", "votes": 44, "replies": [ { "id": 443836, "postDate": "2018-12-22T14:25:12.353Z", "content": "+1 more important note!\nCompetition named \"Elo Merchant Category Recommendation\". So I think, loyalty score relates not to particular merchant, but to merchant category. So they may define \"sales from recs\" like aggregating sales by merchant categories and matching them to categories that were recommended.
", "rawMarkdown": "+1 more important note!\nCompetition named \"Elo Merchant Category Recommendation\". So I think, loyalty score relates not to particular merchant, but to merchant category. So they may define \"sales from recs\" like aggregating sales by merchant categories and matching them to categories that were recommended.", "votes": 4 }, { "id": 449800, "postDate": "2019-01-03T19:13:40.750Z", "content": "Thanks for the information and nice illustration! Although I think in our data set, it's not log transformed, if you take the mean and std of the target excluding the -33s, you get
\n\nmean -0.030879\nstd 1.717793\nwhich is pretty close to the result of a normalization process, and the mean is not exactly 0 is probably because its splitted with the test set. So I guess the original data excluding the 'outliers' is normalized, and -33 looks like the mean, but even if its true i wonder how we can use this information anyway.
", "rawMarkdown": "Thanks for the information and nice illustration! Although I think in our data set, it's not log transformed, if you take the mean and std of the target excluding the -33s, you get \n \n mean -0.030879\n std 1.717793\n\nwhich is pretty close to the result of a normalization process, and the mean is not exactly 0 is probably because its splitted with the test set. So I guess the original data excluding the 'outliers' is normalized, and -33 looks like the mean, but even if its true i wonder how we can use this information anyway.", "votes": 4 }, { "id": 453665, "postDate": "2019-01-10T15:10:40.223Z", "content": "Hi samshipengs,
\n\nYes, I looked deeper in data and now I’m close to the idea of normalization. I performed competition modeling among shoppers using ELO ranking and get a rank with long tail and normal distribution. I will share my thoughts soon.
", "rawMarkdown": "Hi samshipengs,\n\nYes, I looked deeper in data and now I’m close to the idea of normalization. I performed competition modeling among shoppers using ELO ranking and get a rank with long tail and normal distribution. I will share my thoughts soon.", "votes": 3 }, { "id": 455075, "postDate": "2019-01-12T22:31:57.593Z", "content": "How does adding one get a normal distribution? I don't believe it does, and the conversion is not necessary what we want, or binomially distributed, but I do believe the formula uses the exponential distribution/pareto distribution of purchase amount with negative binomial for conversion.\nIf you take out the outliers the distribution of scores has a fatter left tail, indicating, that a superposition of steep pareto/exponential and normal/negative binomial is correct.
\n\nThe purchase amount has been standard scaled. (X -µ)/σ , assuming that there are 0 transactions, that would make μ/σ = 0.7469078
", "rawMarkdown": "How does adding one get a normal distribution? I don't believe it does, and the conversion is not necessary what we want, or binomially distributed, but I do believe the formula uses the exponential distribution/pareto distribution of purchase amount with negative binomial for conversion.\nIf you take out the outliers the distribution of scores has a fatter left tail, indicating, that a superposition of steep pareto/exponential and normal/negative binomial is correct.\n\nThe purchase amount has been standard scaled. (X -µ)/σ , assuming that there are 0 transactions, that would make μ/σ = 0.7469078\n\n", "votes": 1 }, { "id": 467225, "postDate": "2019-02-06T16:56:30.087Z", "content": "Hi, what do you mean by -33 looks like the mean? Somebody in the discussion has already figured out that it is -10log2(10).
", "rawMarkdown": "Hi, what do you mean by -33 looks like the mean? Somebody in the discussion has already figured out that it is -10log2(10). ", "votes": 1 }, { "id": 467296, "postDate": "2019-02-06T20:11:13.610Z", "content": "@JordiClive, good stuff. Can you explain more about \"exponential distribution/pareto distribution of purchase amount with negative binomial for conversion\" Is it usual to model purchase amount as such?
\n\nAlso why would 0 transactions make μ/σ = 0.7469078? Thanks a lot for the insight!
", "rawMarkdown": "@JordiClive, good stuff. Can you explain more about \"exponential distribution/pareto distribution of purchase amount with negative binomial for conversion\" Is it usual to model purchase amount as such?\n\nAlso why would 0 transactions make μ/σ = 0.7469078? Thanks a lot for the insight!\n" } ] }, { "id": 434638, "postDate": "2018-12-06T18:01:52.103Z", "content": "Manually setting my lowest predictions to -33.21928095 decreases LB error by 0.006... This was just an arbitrary random trial on my very lowest predictions to see what would happen. I guess finding the right threshold for this \"minimum loyalty score\" will probably have a much greater effect.
\n\nI could be wrong about this, and I'm obviously not the first person to have concerns, but I'm dubious about the competition in general. The final score is basically going to be based on how many outliers there are in the private set.
\n\nThis makes me worry that the top model might simply be the best classifier for extremely \"disloyal\" customers... Rather than an accurate regressor for true customer loyalty (whatever that is ¯\\_(ツ)_/¯).
\n\nI guess I'll just leave my best \"random number generator\" here and see how I do.
", "rawMarkdown": "Manually setting my lowest predictions to -33.21928095 decreases LB error by 0.006... This was just an arbitrary random trial on my very lowest predictions to see what would happen. I guess finding the right *threshold* for this \"minimum loyalty score\" will probably have a much greater effect.\n\nI could be wrong about this, and I'm obviously not the first person to have concerns, but I'm dubious about the competition in general. The final score is basically going to be based on how many outliers there are in the private set. \n\nThis makes me worry that the top model might simply be **the best classifier for extremely \"disloyal\" customers**... Rather than an accurate regressor for true customer loyalty (whatever that is ¯\\\\_(ツ)_/¯).\n\nI guess I'll just leave my best \"random number generator\" here and see how I do.", "votes": 38 }, { "id": 443280, "postDate": "2018-12-21T10:46:43.813Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 443824, "author_name": "Egor", "author_url": "", "post_date": "2018-12-22T14:10:08.797000", "content": "Hi DStjhb,\nI think, I can explain this phenomena. Disclaimer: it's only my thoughts and they are based only on my own experience in eCommerce.\nThere are 3 major metrics in eCommerce: Revenue Per Session(RPS=Revenue/Sessions), Average Order Value(AOV=Revenue/Orders) and Conversion(CONV=Orders/Sessions). As you can see from formulas, RPS=AOV*CONV.\nAOV has exponential distribution, CONV is Binomial. RPS distribution is a combination of Binomial distribution and exponential.\n\nI try to make an analogy with this competition:\nIn this competition we are trying to predict loyalty score. In my opinion(!!), it's something like \"sales from recs\" and it depends on how much money user spent(exponential distribution) and did he make transaction after getting recommendation(Binomial dist). So we have same combination of Binomial-Exponental Distribution.
My thoughts. I think true loyalty Elo Formula:\n1) Takes sales from recs;\n2) Divides each value by mean(mean --> 1);\n2) Adds some small random number and logs each value(mean --> 0).
\n\nAt least it can explain why we have \"extremely good\" normal distribution of target and large bin of guys with the same value(~-33.21). A lot of people considering them to be outliers but I think, they just spent 0(did not make any transaction) for all their history(or maybe last n month) for merchant, recommendation they got referred to.\nMy idea. First, make binary classifier([-33.21]-->0, [>-33.21]-->1) and try to classify users. Classes are imbalanced, so use AUC metric. After that, for class 1 make regression.
\n\nFood for more thought. Taking log from money can explain negative values of purchase_amount, but even in this way I'm still confused about extremely large values like 60k. np.exp(60k) gives me inf.
", "votes": 44, "replies": [ { "id": 443836, "author_name": "Egor", "author_url": "", "post_date": "2018-12-22T14:25:12.353000", "content": "+1 more important note!\nCompetition named \"Elo Merchant Category Recommendation\". So I think, loyalty score relates not to particular merchant, but to merchant category. So they may define \"sales from recs\" like aggregating sales by merchant categories and matching them to categories that were recommended.
", "votes": 4, "replies": [] }, { "id": 449800, "author_name": "samshipengs", "author_url": "", "post_date": "2019-01-03T19:13:40.750000", "content": "Thanks for the information and nice illustration! Although I think in our data set, it's not log transformed, if you take the mean and std of the target excluding the -33s, you get
\n\nmean -0.030879\nstd 1.717793\nwhich is pretty close to the result of a normalization process, and the mean is not exactly 0 is probably because its splitted with the test set. So I guess the original data excluding the 'outliers' is normalized, and -33 looks like the mean, but even if its true i wonder how we can use this information anyway.
", "votes": 4, "replies": [] }, { "id": 453665, "author_name": "Egor", "author_url": "", "post_date": "2019-01-10T15:10:40.223000", "content": "Hi samshipengs,
\n\nYes, I looked deeper in data and now I’m close to the idea of normalization. I performed competition modeling among shoppers using ELO ranking and get a rank with long tail and normal distribution. I will share my thoughts soon.
", "votes": 3, "replies": [] }, { "id": 455075, "author_name": "JordiClive", "author_url": "", "post_date": "2019-01-12T22:31:57.593000", "content": "How does adding one get a normal distribution? I don't believe it does, and the conversion is not necessary what we want, or binomially distributed, but I do believe the formula uses the exponential distribution/pareto distribution of purchase amount with negative binomial for conversion.\nIf you take out the outliers the distribution of scores has a fatter left tail, indicating, that a superposition of steep pareto/exponential and normal/negative binomial is correct.
\n\nThe purchase amount has been standard scaled. (X -µ)/σ , assuming that there are 0 transactions, that would make μ/σ = 0.7469078
", "votes": 1, "replies": [] }, { "id": 467225, "author_name": "LiwenHuang", "author_url": "", "post_date": "2019-02-06T16:56:30.087000", "content": "Hi, what do you mean by -33 looks like the mean? Somebody in the discussion has already figured out that it is -10log2(10).
", "votes": 1, "replies": [] }, { "id": 467296, "author_name": "LiwenHuang", "author_url": "", "post_date": "2019-02-06T20:11:13.610000", "content": "@JordiClive, good stuff. Can you explain more about \"exponential distribution/pareto distribution of purchase amount with negative binomial for conversion\" Is it usual to model purchase amount as such?
\n\nAlso why would 0 transactions make μ/σ = 0.7469078? Thanks a lot for the insight!
", "votes": 0, "replies": [] } ] }, { "id": 443280, "author_name": "", "author_url": "", "post_date": "2018-12-21T10:46:43.813000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "443824": "Hi DStjhb,\nI think, I can explain this phenomena. Disclaimer: it's only my thoughts and they are based only on my own experience in eCommerce.\nThere are 3 major metrics in eCommerce: Revenue Per Session(RPS=Revenue/Sessions), Average Order Value(AOV=Revenue/Orders) and Conversion(CONV=Orders/Sessions). As you can see from formulas, RPS=AOV*CONV.\nAOV has exponential distribution, CONV is Binomial. RPS distribution is a combination of Binomial distribution and exponential.\n![Image][1]\nI try to make an analogy with this competition:\nIn this competition we are trying to predict loyalty score. In my opinion(!!), it's something like \"sales from recs\" and it depends on how much money user spent(exponential distribution) and did he make transaction after getting recommendation(Binomial dist). So we have same combination of Binomial-Exponental Distribution.\n\nMy thoughts. I think true loyalty Elo Formula:\n1) Takes sales from recs;\n2) Divides each value by mean(mean --> 1);\n2) Adds some small random number and logs each value(mean --> 0).\n\nAt least it can explain why we have \"extremely good\" normal distribution of target and large bin of guys with the same value(~-33.21). A lot of people considering them to be outliers but I think, they just spent 0(did not make any transaction) for all their history(or maybe last n month) for merchant, recommendation they got referred to.\nMy idea. First, make binary classifier([-33.21]-->0, [>-33.21]-->1) and try to classify users. Classes are imbalanced, so use AUC metric. After that, for class 1 make regression.\n\nFood for more thought. Taking log from money can explain negative values of purchase_amount, but even in this way I'm still confused about extremely large values like 60k. np.exp(60k) gives me inf.\n\n[1]: https://i.ibb.co/f2Zjs8y/2018-12-21-12-37-02.png", "434638": "Manually setting my lowest predictions to -33.21928095 decreases LB error by 0.006... This was just an arbitrary random trial on my very lowest predictions to see what would happen. I guess finding the right *threshold* for this \"minimum loyalty score\" will probably have a much greater effect.\n\nI could be wrong about this, and I'm obviously not the first person to have concerns, but I'm dubious about the competition in general. The final score is basically going to be based on how many outliers there are in the private set. \n\nThis makes me worry that the top model might simply be **the best classifier for extremely \"disloyal\" customers**... Rather than an accurate regressor for true customer loyalty (whatever that is ¯\\\\_(ツ)_/¯).\n\nI guess I'll just leave my best \"random number generator\" here and see how I do.", "443280": "" } }