My solution is some ad-hoc feature engineering, which is rather boring and trivial compared to sjv's clean and cool neural network solution. Here are some tips:
\n\nUncertainty and Correlation of Estimated Probabilities: Faron's O(n^2) F1-optimization script is sub-optimal because it does not address correlation. Uncertainty is also an issue: when all of the estimated probabilities have high uncertainty, it is better to choose all the products to optimize F1. I have two ways to address this issue:
\n\nMultiply the probability by a factor that is negatively correlated with the conditional basket size. I tried some parametric function, and I chose 1.6^(1/n^2), where n is the average conditional basket size: conditional on user purchasing a certain product, how many products he purchase in this order. For \"None\", n=1.
Exact F1 Measure, based on the paper An Exact Algorithm for F-Measure Maximization. The algorithm is O(n^3) if we supply the joint probability matrix(n*n) of a product being purchased and the corresponding basket size of the order. When the purchase probability are independent, it is easy to compute such theoretical matrix using O(n^2) and the result would be the same to Faron's F1 optimization. I constructed an empirical distribution of conditional basket size and take a weighted average between the probability matrix with independent assumption and this empirical one. \nI gained about 0.001-0.0015 from the above optimization.
catboost: catboost is my best single model, with a 0.4086 public LB and 0.4070 private LB. It is about 0.0003-0.0005 better than lightgbm and xgboost. I turned on the \"accurate\" option of xgboost, and it is 0.0015 better than the approximate one.
\n\nLDA and NMF: I also tried to use LDA and NMF features, which gives me 0.0005-0.001 increase.
\n\nWhat did not worked for me:
\n\nCongrats for your golden rank, and thanks for sharing. Engineering isn't boring, hope you can say more.
\n\nInteresting that LDA worked for you. I tried LSI and it idn't yield anything useful.
", "votes": 1, "replies": [ { "id": 213600, "author_name": "GeorgeGui", "author_url": "", "post_date": "2017-08-15T01:34:37.797000", "content": "Thanks. Engineering is fun when it improves the score and gives insights. But @sjv's method is so cool that mine becomes boring in comparison.
\n\nI only used the user-product repurchase rate imputed by LDA and NMF.
", "votes": 0, "replies": [] }, { "id": 213622, "author_name": "Yihong Chen", "author_url": "", "post_date": "2017-08-15T02:03:35.373000", "content": "LDA works for me too ! I use representations of users and items to calculate distance in topic space.
", "votes": 0, "replies": [] } ] }, { "id": 213753, "author_name": "Jing Lu", "author_url": "", "post_date": "2017-08-15T07:17:56.087000", "content": "Congratulations, GeorgeGui! Very insightful! How do you come up with the factor 1.6^(1/n^2)? For n = 2, the factor will be 1.124683. Will you then cap the probability prediction to be 1? The \"Correlation of Estimated Probabilities\", do you mean the predicted probability is highly correlated with the average products in orders (conditional on user purchasing a this product)? So, it's necessary to remove that get a fair estimation.
", "votes": 0, "replies": [ { "id": 213852, "author_name": "GeorgeGui", "author_url": "", "post_date": "2017-08-15T11:48:18.410000", "content": "I tried several functional forms and did a grid search to come up with the factor. The bottom line is that when n is large, you would want the factor to be close to 1, and when n is small, you would like it to be a little bit greater than 1. I capped the probability at 0.99.
\n\nFor correlation, suppose whenever product A is purchased, product B is also purchased( i.e. A and B are perfectly correlated). Then if we have P(A) = P(B) > P(C), there are situations in which it is better to choose (A,C) or (B,C) than (A,B), if C is not correlated with A and B.
", "votes": 0, "replies": [] }, { "id": 213892, "author_name": "", "author_url": "", "post_date": "2017-08-15T13:40:30.973000", "content": "", "votes": 0, "replies": [] }, { "id": 213900, "author_name": "GeorgeGui", "author_url": "", "post_date": "2017-08-15T14:04:30.383000", "content": "@<^..^>: For the first optimization method, it is just multiplying a factor before putting them into Faron's Script, or its Rcpp equivalent. It is quite fast and is even slightly better than the second method.
\n\nMy current code for the second method is a bit messy and extremely slow because it involves estimating the empirical count distribution for each product, which requires more tuning than the first method. The O(n^3) exact F1-optimization method is well written on page 5 of the paper and is quite easy to implement (it only took me less than 15 minutes), especially if you understand Faron's script well enough. However, it is not useful unless we give a decent count distribution. I do not like to use this method because it is kind of slow compared to the O(n^2).
", "votes": 1, "replies": [] }, { "id": 213905, "author_name": "", "author_url": "", "post_date": "2017-08-15T14:14:12.353000", "content": "", "votes": 0, "replies": [] }, { "id": 213906, "author_name": "GeorgeGui", "author_url": "", "post_date": "2017-08-15T14:15:13.513000", "content": "Just try to multiply your probability by a factor. It is the easiest thing to add ^.^
", "votes": 1, "replies": [] }, { "id": 213911, "author_name": "Ravi Teja Gutta", "author_url": "", "post_date": "2017-08-15T14:22:57.120000", "content": "Great insight !!, I knew that if a user buys a certain product, then it will affect the probabilities of other products too. Faron's script assumes that all products are independent. I didn't know how to deal with conditional probabilites and it seems that using correlation of probs is the way to go.Congrats on on 4th place
", "votes": 0, "replies": [] } ] }, { "id": 213596, "author_name": "AlphaMeow", "author_url": "", "post_date": "2017-08-15T01:30:34.970000", "content": "Congratulations GGui and thanks for sharing these thoughts! Learnt a lot from your posts in this comp. :)
", "votes": 0, "replies": [] }, { "id": 213714, "author_name": "YaGana Sheriff-Hussaini", "author_url": "", "post_date": "2017-08-15T05:17:41.983000", "content": "Congratulations and thanks for sharing.
", "votes": 0, "replies": [] } ], "comment_count": 12, "messages": [ { "id": 213591, "postDate": "2017-08-15T01:24:59.820Z", "content": "My solution is some ad-hoc feature engineering, which is rather boring and trivial compared to sjv's clean and cool neural network solution. Here are some tips:
\n\nUncertainty and Correlation of Estimated Probabilities: Faron's O(n^2) F1-optimization script is sub-optimal because it does not address correlation. Uncertainty is also an issue: when all of the estimated probabilities have high uncertainty, it is better to choose all the products to optimize F1. I have two ways to address this issue:
\n\nMultiply the probability by a factor that is negatively correlated with the conditional basket size. I tried some parametric function, and I chose 1.6^(1/n^2), where n is the average conditional basket size: conditional on user purchasing a certain product, how many products he purchase in this order. For \"None\", n=1.
Exact F1 Measure, based on the paper An Exact Algorithm for F-Measure Maximization. The algorithm is O(n^3) if we supply the joint probability matrix(n*n) of a product being purchased and the corresponding basket size of the order. When the purchase probability are independent, it is easy to compute such theoretical matrix using O(n^2) and the result would be the same to Faron's F1 optimization. I constructed an empirical distribution of conditional basket size and take a weighted average between the probability matrix with independent assumption and this empirical one. \nI gained about 0.001-0.0015 from the above optimization.
catboost: catboost is my best single model, with a 0.4086 public LB and 0.4070 private LB. It is about 0.0003-0.0005 better than lightgbm and xgboost. I turned on the \"accurate\" option of xgboost, and it is 0.0015 better than the approximate one.
\n\nLDA and NMF: I also tried to use LDA and NMF features, which gives me 0.0005-0.001 increase.
\n\nWhat did not worked for me:
\n\nCongrats for your golden rank, and thanks for sharing. Engineering isn't boring, hope you can say more.
\n\nInteresting that LDA worked for you. I tried LSI and it idn't yield anything useful.
", "rawMarkdown": "Congrats for your golden rank, and thanks for sharing. Engineering isn't boring, hope you can say more.\n\nInteresting that LDA worked for you. I tried LSI and it idn't yield anything useful.", "votes": 1, "replies": [ { "id": 213600, "postDate": "2017-08-15T01:34:37.797Z", "content": "Thanks. Engineering is fun when it improves the score and gives insights. But @sjv's method is so cool that mine becomes boring in comparison.
\n\nI only used the user-product repurchase rate imputed by LDA and NMF.
", "rawMarkdown": "Thanks. Engineering is fun when it improves the score and gives insights. But @sjv's method is so cool that mine becomes boring in comparison. \n\nI only used the user-product repurchase rate imputed by LDA and NMF. " }, { "id": 213622, "postDate": "2017-08-15T02:03:35.373Z", "content": "LDA works for me too ! I use representations of users and items to calculate distance in topic space.
", "rawMarkdown": "LDA works for me too ! I use representations of users and items to calculate distance in topic space." } ] }, { "id": 213753, "postDate": "2017-08-15T07:17:56.087Z", "content": "Congratulations, GeorgeGui! Very insightful! How do you come up with the factor 1.6^(1/n^2)? For n = 2, the factor will be 1.124683. Will you then cap the probability prediction to be 1? The \"Correlation of Estimated Probabilities\", do you mean the predicted probability is highly correlated with the average products in orders (conditional on user purchasing a this product)? So, it's necessary to remove that get a fair estimation.
", "rawMarkdown": "Congratulations, GeorgeGui! Very insightful! How do you come up with the factor 1.6^(1/n^2)? For n = 2, the factor will be 1.124683. Will you then cap the probability prediction to be 1? The \"Correlation of Estimated Probabilities\", do you mean the predicted probability is highly correlated with the average products in orders (conditional on user purchasing a this product)? So, it's necessary to remove that get a fair estimation.", "replies": [ { "id": 213852, "postDate": "2017-08-15T11:48:18.410Z", "content": "I tried several functional forms and did a grid search to come up with the factor. The bottom line is that when n is large, you would want the factor to be close to 1, and when n is small, you would like it to be a little bit greater than 1. I capped the probability at 0.99.
\n\nFor correlation, suppose whenever product A is purchased, product B is also purchased( i.e. A and B are perfectly correlated). Then if we have P(A) = P(B) > P(C), there are situations in which it is better to choose (A,C) or (B,C) than (A,B), if C is not correlated with A and B.
", "rawMarkdown": "I tried several functional forms and did a grid search to come up with the factor. The bottom line is that when n is large, you would want the factor to be close to 1, and when n is small, you would like it to be a little bit greater than 1. I capped the probability at 0.99. \n\nFor correlation, suppose whenever product A is purchased, product B is also purchased( i.e. A and B are perfectly correlated). Then if we have P(A) = P(B) > P(C), there are situations in which it is better to choose (A,C) or (B,C) than (A,B), if C is not correlated with A and B. \n\n\n\n " }, { "id": 213892, "postDate": "2017-08-15T13:40:30.973Z", "rawMarkdown": "", "isDeleted": true }, { "id": 213900, "postDate": "2017-08-15T14:04:30.383Z", "content": "@<^..^>: For the first optimization method, it is just multiplying a factor before putting them into Faron's Script, or its Rcpp equivalent. It is quite fast and is even slightly better than the second method.
\n\nMy current code for the second method is a bit messy and extremely slow because it involves estimating the empirical count distribution for each product, which requires more tuning than the first method. The O(n^3) exact F1-optimization method is well written on page 5 of the paper and is quite easy to implement (it only took me less than 15 minutes), especially if you understand Faron's script well enough. However, it is not useful unless we give a decent count distribution. I do not like to use this method because it is kind of slow compared to the O(n^2).
", "rawMarkdown": "@<^..^>: For the first optimization method, it is just multiplying a factor before putting them into Faron's Script, or its Rcpp equivalent. It is quite fast and is even slightly better than the second method. \n\nMy current code for the second method is a bit messy and extremely slow because it involves estimating the empirical count distribution for each product, which requires more tuning than the first method. The O(n^3) exact F1-optimization method is well written on page 5 of the paper and is quite easy to implement (it only took me less than 15 minutes), especially if you understand Faron's script well enough. However, it is not useful unless we give a decent count distribution. I do not like to use this method because it is kind of slow compared to the O(n^2). \n", "votes": 1 }, { "id": 213905, "postDate": "2017-08-15T14:14:12.353Z", "rawMarkdown": "", "isDeleted": true }, { "id": 213906, "postDate": "2017-08-15T14:15:13.513Z", "content": "Just try to multiply your probability by a factor. It is the easiest thing to add ^.^
", "rawMarkdown": "Just try to multiply your probability by a factor. It is the easiest thing to add ^.^", "votes": 1 }, { "id": 213911, "postDate": "2017-08-15T14:22:57.120Z", "content": "Great insight !!, I knew that if a user buys a certain product, then it will affect the probabilities of other products too. Faron's script assumes that all products are independent. I didn't know how to deal with conditional probabilites and it seems that using correlation of probs is the way to go.Congrats on on 4th place
", "rawMarkdown": "Great insight !!, I knew that if a user buys a certain product, then it will affect the probabilities of other products too. Faron's script assumes that all products are independent. I didn't know how to deal with conditional probabilites and it seems that using correlation of probs is the way to go.Congrats on on 4th place" } ] }, { "id": 213596, "postDate": "2017-08-15T01:30:34.970Z", "content": "Congratulations GGui and thanks for sharing these thoughts! Learnt a lot from your posts in this comp. :)
", "rawMarkdown": "Congratulations GGui and thanks for sharing these thoughts! Learnt a lot from your posts in this comp. :)" }, { "id": 213714, "postDate": "2017-08-15T05:17:41.983Z", "content": "Congratulations and thanks for sharing.
", "rawMarkdown": "Congratulations and thanks for sharing." } ], "complete": true, "source": "live", "fetched_at": "2026-10-20T02:03:31Z", "competition_id": 6644, "competition_slug": "basket-analysis", "team_id": 709978, "solution_url": "/competitions/basket-analysis/writeups/georgegui-4-th-place-tips" }