Below is the screenshot of distributions of pageviews/hits ratios for the segments with revenue >0 and revenue = 0 (logarithmic y scales). Although huge majority of observations have the score close to 1, the several outliers are guaranteed to yield no profit (candidates for post processing ;) )
", "messages": [ { "id": 393379, "postDate": "2018-09-25T09:23:09.483Z", "content": "Below is the screenshot of distributions of pageviews/hits ratios for the segments with revenue >0 and revenue = 0 (logarithmic y scales). Although huge majority of observations have the score close to 1, the several outliers are guaranteed to yield no profit (candidates for post processing ;) )
", "rawMarkdown": "Below is the screenshot of distributions of pageviews/hits ratios for the segments with revenue >0 and revenue = 0 (logarithmic y scales). Although huge majority of observations have the score close to 1, the several outliers are guaranteed to yield no profit (candidates for post processing ;) )", "votes": 66 }, { "id": 397540, "postDate": "2018-10-02T16:34:54.653Z", "content": "I observe similar behaviour as descibred by @fatihozturk or @AhmetErdem: my CV improves, but my LB decreases.
\n\nI was trying to go one level deeper in the thinking. We have the following situation:
\n\nEven if we don't include the ratio A/B as an explicit feature, LGBM will still try to 'find it' via tree splits. So, my point is we cannot simply not include data['totals.hits'] / data['totals.pageviews'] as a feature and forget about it.
\n\nI don't have a ready prescription of what we should do, but until I understand why this happens, I feel I don't understand this competition.
", "rawMarkdown": "I observe similar behaviour as descibred by @fatihozturk or @AhmetErdem: my CV improves, but my LB decreases.\n\nI was trying to go one level deeper in the thinking. We have the following situation:\n\n - you have features A and B, they are both strong features and their ratio is also a strong feature in your model\n - the ratio A/B makes a lot of sense as a feature from business perspective\n\nEven if we don't include the ratio A/B as an explicit feature, LGBM will still try to 'find it' via tree splits. So, my point is we cannot simply not include data['totals.hits'] / data['totals.pageviews'] as a feature and forget about it. \n\nI don't have a ready prescription of what we should do, but until I understand why this happens, I feel I don't understand this competition.", "votes": 8, "replies": [ { "id": 398844, "postDate": "2018-10-04T16:52:29.273Z", "content": "I think tree based models find diffcult to learn the ratio of two features.
\n\nReference:\nhttps://arxiv.org/pdf/1701.07852.pdf
", "rawMarkdown": "I think tree based models find diffcult to learn the ratio of two features.\n\nReference:\nhttps://arxiv.org/pdf/1701.07852.pdf", "votes": 3 }, { "id": 398927, "postDate": "2018-10-04T20:16:05.060Z", "content": "Very interesting @Suchith !
", "rawMarkdown": "Very interesting @Suchith !" } ] }, { "id": 395176, "postDate": "2018-09-28T04:59:55.027Z", "content": "Totally agree that customers that purchase do more hits per pageview (or equivalently smaller pageview/hit) by clicking the product, adding to cart, checking out, etc. But Adam's screenshot shows the other way: you see non-purchaser instead of purchaser at the lower end of pageviews/hit (between 0 and 0.4). Why?
", "rawMarkdown": "Totally agree that customers that purchase do more hits per pageview (or equivalently smaller pageview/hit) by clicking the product, adding to cart, checking out, etc. But Adam's screenshot shows the other way: you see non-purchaser instead of purchaser at the lower end of pageviews/hit (between 0 and 0.4). Why?", "votes": 3, "replies": [ { "id": 398142, "postDate": "2018-10-03T15:07:21.557Z", "content": "You should focus at the purchaser graph. Purchasers start from 0.4 onwards. If you look at non-purchaser graph, it just shows everyone else that is browsing without purchasing anything.
", "rawMarkdown": "You should focus at the purchaser graph. Purchasers start from 0.4 onwards. If you look at non-purchaser graph, it just shows everyone else that is browsing without purchasing anything.", "votes": 1 } ] }, { "id": 397417, "postDate": "2018-10-02T13:08:44.783Z", "content": "I've been using this featuredata['totals.hits'] / data['totals.pageviews']since my very first submission. I was not aware of its impact to LB. Today I excluded that feature from my best single model and it worsened kfold CV by 0.017 and improved LB by 0.0003. Also worsened my time-split validation by 0.011. This is so weird and I feel like I'm lost in this competition's LB-CV movement. We need to do deeper analysis for train-test differences or just wait for a significant shake-up.
Had the same issue... and not only with the ratio but with a lot of my time based variables too.
", "rawMarkdown": "Had the same issue... and not only with the ratio but with a lot of my time based variables too.", "votes": 2 }, { "id": 397442, "postDate": "2018-10-02T13:39:36.963Z", "content": "Hey Fatih, interesting! people including you have different experiences using this single feature in their models. BTW, I am also waiting for a considerable shake-up because of issues like data imbalance, temporal nature of this problem and the evaluation metric (RMSE). Specially RMSE could lure many people into putting more faith in their LB (and hill climbing on LB) score rather than their CV. I hope this is not another Porto Seguro ;-).
", "rawMarkdown": "Hey Fatih, interesting! people including you have different experiences using this single feature in their models. BTW, I am also waiting for a considerable shake-up because of issues like data imbalance, temporal nature of this problem and the evaluation metric (RMSE). Specially RMSE could lure many people into putting more faith in their LB (and hill climbing on LB) score rather than their CV. I hope this is not another Porto Seguro ;-). \n" }, { "id": 397451, "postDate": "2018-10-02T13:53:53.653Z", "content": "@areveillon Actually most of my features improving CV also improve LB, but I couldn't measure the effect of this one since my first submission already had that. However, there are also many people saying what you said. Don't know what causes this difference.
\n\nHi @kainsama , the factors you counted are already contributers for a natural shake-up. That's right we should go with our local validations as you said, but this type of a weirdness make me question 'Go with your CV' motto as well. If features don't show same pattern related to target in train and test what can we do.. Btw, did you test the impact of this feature in your current best model or it was an old test?
", "rawMarkdown": "@areveillon Actually most of my features improving CV also improve LB, but I couldn't measure the effect of this one since my first submission already had that. However, there are also many people saying what you said. Don't know what causes this difference.\n\nHi @kainsama , the factors you counted are already contributers for a natural shake-up. That's right we should go with our local validations as you said, but this type of a weirdness make me question 'Go with your CV' motto as well. If features don't show same pattern related to target in train and test what can we do.. Btw, did you test the impact of this feature in your current best model or it was an old test?", "votes": 1 }, { "id": 397461, "postDate": "2018-10-02T14:06:50.490Z", "content": "Regarding your question, it was one the best features in my old and new models...and this feature also makes sense logically but it seems it works differently for different people and this baffles me.
", "rawMarkdown": "Regarding your question, it was one the best features in my old and new models...and this feature also makes sense logically but it seems it works differently for different people and this baffles me. ", "votes": 2 }, { "id": 397673, "postDate": "2018-10-02T22:29:19.300Z", "content": "Hey Kain, could you elaborate on how RSME would lure people into putting more faith in the LB?\nAm i wrong in assuming I should measure my model in it's rmse (local not LB)?
\n\nBTW you guys are amazing, since starting competitions on this website I learned so much by reading the discussions and kernels you provide so just a quick thank you!
", "rawMarkdown": "Hey Kain, could you elaborate on how RSME would lure people into putting more faith in the LB?\nAm i wrong in assuming I should measure my model in it's rmse (local not LB)?\n\nBTW you guys are amazing, since starting competitions on this website I learned so much by reading the discussions and kernels you provide so just a quick thank you!", "votes": 1 }, { "id": 397875, "postDate": "2018-10-03T09:02:36.607Z", "content": "Hey Christian, If we refer to the formula for RMSE we see that it is a function of d (i.e. f(d)) where d is a vector of squared differences between our predictions and the true values. As we know mean is biased towards extreme values even if there are a few of them in our vector. if we could somehow could adjust our model to have relatively smaller difference values for a couple of those extreme values we end up with a model with a relatively good score. The problem arises when CV and LB do not follow the same path so it lures many people to trust LB that is not a robust estimator of our model' performance; it is usually like a trap because the final medals and points are given based on the PB. This is troublesome when a lot of teams are clustered around a certain error value and we are fighting over 0.001+.
", "rawMarkdown": "Hey Christian, If we refer to the formula for RMSE we see that it is a function of d (i.e. f(d)) where d is a vector of squared differences between our predictions and the true values. As we know mean is biased towards extreme values even if there are a few of them in our vector. if we could somehow could adjust our model to have relatively smaller difference values for a couple of those extreme values we end up with a model with a relatively good score. The problem arises when CV and LB do not follow the same path so it lures many people to trust LB that is not a robust estimator of our model' performance; it is usually like a trap because the final medals and points are given based on the PB. This is troublesome when a lot of teams are clustered around a certain error value and we are fighting over 0.001+. ", "votes": 1 }, { "id": 398311, "postDate": "2018-10-03T21:53:16.260Z", "content": "@fatihozturk no idea how you choose your feature, but you are pretty good at it ! I tried a lot of features where my local CV for session or user based decrease but my LB increase, which is very frustrating because I don't know if I can fully trust my CV
", "rawMarkdown": "@fatihozturk no idea how you choose your feature, but you are pretty good at it ! I tried a lot of features where my local CV for session or user based decrease but my LB increase, which is very frustrating because I don't know if I can fully trust my CV" }, { "id": 398692, "postDate": "2018-10-04T13:01:51.477Z", "content": "Hi Kain,
\n\ncould you elaborate more on\" if we could somehow could adjust our model to have relatively smaller difference values for a couple of those extreme values we end up with a model with a relatively good score. \"?
\n\nI did some error analysis on my model, seems prediction for nonzero value is right-skewed, apparently it is affected by many zero target, and I arrive similar conclusion as yours, I tried to increase a bit on the zero target value, say 1, but the prediction becomes worse.
", "rawMarkdown": "Hi Kain,\n\ncould you elaborate more on\" if we could somehow could adjust our model to have relatively smaller difference values for a couple of those extreme values we end up with a model with a relatively good score. \"?\n\nI did some error analysis on my model, seems prediction for nonzero value is right-skewed, apparently it is affected by many zero target, and I arrive similar conclusion as yours, I tried to increase a bit on the zero target value, say 1, but the prediction becomes worse." }, { "id": 398797, "postDate": "2018-10-04T15:38:28.037Z", "content": "Hey Joe, For example we have 60 values like x in our test that log(x + 1) > 21 and our predictions of them predicting those lie in range [16...18], if we could somehow decrease the gap between the real values and our predictions then we will have good boost; but how can we just effect those predictions without adding a constant number to all or our predictions? or what do those big spenders have in common that can help our predictions? I wish i could elaborate more but i can't.... ;-)
", "rawMarkdown": "Hey Joe, For example we have 60 values like x in our test that log(x + 1) > 21 and our predictions of them predicting those lie in range [16...18], if we could **somehow** decrease the gap between the real values and our predictions then we will have good boost; but how can we just effect those predictions without adding a constant number to all or our predictions? or what do those big spenders have in common that can help our predictions? I wish i could elaborate more but i can't.... ;-)", "votes": 1 } ] }, { "id": 394960, "postDate": "2018-09-27T18:13:21.487Z", "content": "I did find rows where ration <0.4 had transactions. but <0.35 are all without transaction.\nSo any of you that tried the 0.4 and got worse results - try 0.35 instead.
", "rawMarkdown": "I did find rows where ration <0.4 had transactions. but <0.35 are all without transaction.\nSo any of you that tried the 0.4 and got worse results - try 0.35 instead.", "votes": 4 }, { "id": 394364, "postDate": "2018-09-26T18:16:54.477Z", "content": "Whats the difference between a hit and a page view? Number of clicks vs number of refreshes/new pages?
", "rawMarkdown": "Whats the difference between a hit and a page view? Number of clicks vs number of refreshes/new pages?", "votes": 4, "replies": [ { "id": 394374, "postDate": "2018-09-26T18:41:24.557Z", "content": "Hit: Pages are comprised of files. For example, if you have a page with 10 pictures, then a request to a server to view that page generates 11 hits (10 for the pictures, and one for the html file).
Please have a look at my EDA. There's more stuff like that in there ;-)\nhttps://www.kaggle.com/erikbruin/google-analytics-eda-with-screenshots-of-the-app
", "rawMarkdown": "`Hit: Pages are comprised of files. For example, if you have a page with 10 pictures, then a request to a server to view that page generates 11 hits (10 for the pictures, and one for the html file).`\n\nPlease have a look at my EDA. There's more stuff like that in there ;-)\nhttps://www.kaggle.com/erikbruin/google-analytics-eda-with-screenshots-of-the-app", "votes": 6 }, { "id": 394380, "postDate": "2018-09-26T18:55:32.650Z", "content": "Erik:
\n\nWhat you describe are hits from the perspective of the server hosting the website, which is different from google analytics hits.
\n\nHere, hits include page views, but also other events such as clicking on a product to view more details, or adding a product to cart.
\n\nhttps://support.google.com/analytics/answer/6086082?hl=en
", "rawMarkdown": "Erik:\n\nWhat you describe are hits from the perspective of the server hosting the website, which is different from google analytics hits.\n\nHere, hits include page views, but also other events such as clicking on a product to view more details, or adding a product to cart.\n\n\n\nhttps://support.google.com/analytics/answer/6086082?hl=en", "votes": 10 }, { "id": 394381, "postDate": "2018-09-26T18:58:15.073Z", "content": "Ok, thanks! Will have another look.....
", "rawMarkdown": "Ok, thanks! Will have another look....." }, { "id": 398144, "postDate": "2018-10-03T15:10:32.107Z", "content": "So does hits basically mean user actions?
", "rawMarkdown": "So does hits basically mean user actions?" } ] }, { "id": 393401, "postDate": "2018-09-25T09:54:16.703Z", "content": "Thanks for sharing! Super useful insight
", "rawMarkdown": "Thanks for sharing! Super useful insight", "votes": 1 }, { "id": 393810, "postDate": "2018-09-25T23:02:48.913Z", "content": "I've had similar findings. Not surprising given that hits include pageviews but also other user actions such as adding items to cart, viewing a product etc.
", "rawMarkdown": "I've had similar findings. Not surprising given that hits include pageviews but also other user actions such as adding items to cart, viewing a product etc.", "replies": [ { "id": 394287, "postDate": "2018-09-26T16:04:03.380Z", "content": "Yep. Also the inverse, hits per page, is a good indication that the user went to the shopping cart page and/or checkout. Those pages typically have more elements (hits) that load into the page as compared to a home page or product view.
", "rawMarkdown": "Yep. Also the inverse, hits per page, is a good indication that the user went to the shopping cart page and/or checkout. Those pages typically have more elements (hits) that load into the page as compared to a home page or product view.", "votes": 5 } ] }, { "id": 397628, "postDate": "2018-10-02T20:28:48.787Z", "content": "Oddly, after you shared this one thing led to another and this got me thinking about a new way to do feature splitting in my algorithms without significantly increasing the runtime, so yay for that.
", "rawMarkdown": "Oddly, after you shared this one thing led to another and this got me thinking about a new way to do feature splitting in my algorithms without significantly increasing the runtime, so yay for that." }, { "id": 395754, "postDate": "2018-09-29T09:27:47.840Z", "content": "I tried both _df[''totals.pageviews] / (data_df['totals.hits']+1) and data_df['totals.hits'] / (data_df['totals.pageviews']+1) , and it lead to a considerable CV boost but decrease on LB, sad:( .
", "rawMarkdown": "I tried both _df[''totals.pageviews] / (data_df['totals.hits']+1) and data_df['totals.hits'] / (data_df['totals.pageviews']+1) , and it lead to a considerable CV boost but decrease on LB, sad:( . " }, { "id": 394708, "postDate": "2018-09-27T09:42:58.747Z", "content": "You should keep in mind that patterns in the CV <> patterns in the test set...
", "rawMarkdown": "You should keep in mind that patterns in the CV <> patterns in the test set..." }, { "id": 394668, "postDate": "2018-09-27T08:16:22.457Z", "content": "This feature improves my CV but worsens my LB. Does anyone else experience the same?
", "rawMarkdown": "This feature improves my CV but worsens my LB. Does anyone else experience the same?", "replies": [ { "id": 394682, "postDate": "2018-09-27T08:49:28.573Z", "content": "The same for me. Important CV improvment but a slightly worse LB score. finally i don't know if I should keep it or not...
", "rawMarkdown": "The same for me. Important CV improvment but a slightly worse LB score. finally i don't know if I should keep it or not...", "votes": 1 }, { "id": 394686, "postDate": "2018-09-27T08:53:52.423Z", "content": "same here, improve 0.001 CV but worsen my LB about 0.001 .
", "rawMarkdown": "same here, improve 0.001 CV but worsen my LB about 0.001 ." }, { "id": 396973, "postDate": "2018-10-01T17:46:41.213Z", "content": "Yes - I started with this helpful user-level Light GBM-based notebook and then added pageviews/hits as an additional feature without making any other changes at all. The local CV improved by 0.013 (1.58154 to 1.56890) but the LB score got worse by 0.01 (1.4473 to 1.4568). Does anybody know why supplying an additional feature to a Light GBM model might make it get worse?
", "rawMarkdown": "Yes - I started with this helpful [user-level Light GBM-based notebook](https://www.kaggle.com/ogrellier/user-level-lightgbm-lb-1-4480) and then added pageviews/hits as an additional feature without making any other changes at all. The local CV improved by 0.013 (1.58154 to 1.56890) but the LB score got worse by 0.01 (1.4473 to 1.4568). Does anybody know why supplying an additional feature to a Light GBM model might make it get worse?", "votes": 1 }, { "id": 397052, "postDate": "2018-10-01T20:12:35.637Z", "content": "@Andrew Most probably, it is due to the difference between train and test set. This variable has different effect on the test it seems. If you use something else instead of lightgbm, I guess you will observe the same effect.
", "rawMarkdown": "@Andrew Most probably, it is due to the difference between train and test set. This variable has different effect on the test it seems. If you use something else instead of lightgbm, I guess you will observe the same effect.", "votes": 1 } ] }, { "id": 393956, "postDate": "2018-09-26T05:43:42.780Z", "content": "Thanks for sharing this helpful finding!
", "rawMarkdown": "Thanks for sharing this helpful finding!" }, { "id": 393747, "postDate": "2018-09-25T20:43:41.063Z", "content": "I tried this as a feature and it lead to a considerable CV boost. Thanx, now i know why ;-).
", "rawMarkdown": "I tried this as a feature and it lead to a considerable CV boost. Thanx, now i know why ;-).", "replies": [ { "id": 394240, "postDate": "2018-09-26T14:48:28.813Z", "content": "Did u get a boost in LB as well ? Because I fed this variable (as a numerical one) and the LB seemed to be the same
", "rawMarkdown": "Did u get a boost in LB as well ? Because I fed this variable (as a numerical one) and the LB seemed to be the same ", "votes": 1 }, { "id": 394252, "postDate": "2018-09-26T14:57:43.107Z", "content": "Yes
", "rawMarkdown": "Yes" }, { "id": 394382, "postDate": "2018-09-26T19:02:39.840Z", "content": "I am getting a marginal boost in CVs but significant at the LB. I am yet to try this new feature. Thanks!
", "rawMarkdown": "I am getting a marginal boost in CVs but significant at the LB. I am yet to try this new feature. Thanks! " }, { "id": 394750, "postDate": "2018-09-27T11:36:49.083Z", "content": "Hi Kain, how did you transform you new feature ? I assigned 0 for pageview/hits ratio < 0.4 and 1 for other , didn't improve any score.
", "rawMarkdown": "Hi Kain, how did you transform you new feature ? I assigned 0 for pageview/hits ratio < 0.4 and 1 for other , didn't improve any score. " }, { "id": 394754, "postDate": "2018-09-27T11:46:49.923Z", "content": "Hi Manish, I am using, totals.hits / (totals.pageViews + 1) as a feature and it is giving a good boost to my CV and also improves my LB but the boost in CV is more tangible. BTW, it is usually better to let your model use these kinds of features instead of using them explicitly in your predictions (e.g. hand-labeling).
", "rawMarkdown": "Hi Manish, I am using, totals.hits / (totals.pageViews + 1) as a feature and it is giving a good boost to my CV and also improves my LB but the boost in CV is more tangible. BTW, it is usually better to let your model use these kinds of features instead of using them explicitly in your predictions (e.g. hand-labeling). ", "votes": 6 }, { "id": 394929, "postDate": "2018-09-27T16:48:03.127Z", "rawMarkdown": "", "votes": 1, "isDeleted": true }, { "id": 394933, "postDate": "2018-09-27T16:55:58.100Z", "content": "Because it's an implication - you reduce amount of information available for a model instead of leting it decide what exactly about a feature is important.
", "rawMarkdown": "Because it's an implication - you reduce amount of information available for a model instead of leting it decide what exactly about a feature is important.", "votes": 1 }, { "id": 394935, "postDate": "2018-09-27T17:13:54.100Z", "rawMarkdown": "", "isDeleted": true }, { "id": 394943, "postDate": "2018-09-27T17:31:46.047Z", "content": "Hi @wenshihao1993, IMHO, For the following reasons:
\n\nI used the word usually for describing my opinion because there some Kagglers (many top Kagglers) that spend days or even weeks on finding insights (possibly leaks ;-)) to boost their scores based on their analysis.
", "rawMarkdown": "Hi @wenshihao1993, IMHO, For the following reasons: \n\n 1. In the process of extracting insights from data we humans, as a proud member of class Mammalia ( ;-) ) usually isolate the phenomena we see and try to reduce them based on our own subjective biases (and our heuristics) to some simple deductive and inductive statements. For example in this competition most of revenue comes from north America (i.e. US) so someone may conclude that we just need to use the feature about continent or subcontinent is to separate the zeros but how can we test our assumptions objectively?\n 2. Machine learning (ML) models can handle multiple features at once in a sense that they can have more realistic views of the relationship between different sets of features when it comes to predict a dependent variable (i.e. the output).\n 3. ML models are used along with an objective metric of performance (cross validation error/score) for determining the contribution of a feature to the prediction performance; statistically speaking, it is more reliable to test a feature in CV rather than proceeding with our gut feelings!\n\nI used the word *usually* for describing my opinion because there some Kagglers (many top Kagglers) that spend days or even weeks on finding insights (possibly leaks ;-)) to boost their scores based on their analysis.\n\n \n\n ", "votes": 7 }, { "id": 394947, "postDate": "2018-09-27T17:46:44.233Z", "rawMarkdown": "", "votes": 1, "isDeleted": true }, { "id": 394958, "postDate": "2018-09-27T18:10:30.063Z", "content": "@Kain I disagree somewhat with your suggestion. Using both hand-labeling and feeding as a feature seems best. For example, we have always been feeding totals.bounces as a feature, but sometimes our model predicts positive revenue for when totals.bounces==1, even though we know that it must be zero.
Thanks for sharing. I've used totals.hits / totals.pageViews as a new feature and it also gave me a boost in my local CV while my LB decreased about 0.04. I think I need to give totals.hits / (totals.pageViews + 1) a try.
", "rawMarkdown": "Thanks for sharing. I've used totals.hits / totals.pageViews as a new feature and it also gave me a boost in my local CV while my LB decreased about 0.04. I think I need to give totals.hits / (totals.pageViews + 1) a try.", "votes": 1 }, { "id": 395184, "postDate": "2018-09-28T05:18:50.043Z", "content": "Hi Kain , shouldn't we be using train_df[''totals.pageviews] / (train_df['totals.hits']+1) instead of train_df['totals.hits'] / (train_df['totals.pageviews']+1) , because the visualization posted in this thread is about pageviews/ hits ratio rather that hits/pageviews ratio.
", "rawMarkdown": "Hi Kain , shouldn't we be using **train_df[''totals.pageviews] / (train_df['totals.hits']+1)** instead of **train_df['totals.hits'] / (train_df['totals.pageviews']+1)** , because the visualization posted in this thread is about pageviews/ hits ratio rather that hits/pageviews ratio. " }, { "id": 395223, "postDate": "2018-09-28T06:48:39.377Z", "content": "I am not sure but some algorithms are indifferent to the choice between a / (b + 1) and b / (a + 1); the usual practice is to give them a try in a validation scheme to see what works best.
", "rawMarkdown": "I am not sure but some algorithms are indifferent to the choice between a / (b + 1) and b / (a + 1); the usual practice is to give them a try in a validation scheme to see what works best." }, { "id": 395224, "postDate": "2018-09-28T06:51:00.810Z", "content": "Used both , there is no effect on LB score . What CV strategy are you using ? Timebased ? Can you share your kernel once ?
", "rawMarkdown": "Used both , there is no effect on LB score . What CV strategy are you using ? Timebased ? Can you share your kernel once ?" }, { "id": 395227, "postDate": "2018-09-28T06:56:58.127Z", "content": "@returnofsputnik I agree with you in a sense that there are exceptions where things like hand-labeling would get you the 1st place; What I learned by competing at Kaggle is that we should try as many ideas as we could. At the end of the day, whatever works is the winner.
", "rawMarkdown": "@returnofsputnik I agree with you in a sense that there are exceptions where things like hand-labeling would get you [the 1st place][1]; What I learned by competing at Kaggle is that we should **try** as many ideas as we could. At the end of the day, whatever works is the winner.\n\n\n [1]: https://www.kaggle.com/c/predicting-red-hat-business-value/discussion/23786#latest-136487", "votes": 1 }, { "id": 395241, "postDate": "2018-09-28T07:14:45.097Z", "content": "@mmanishh, I have a simple 5 fold-CV at session level and like you I have tried many things that boosted my CV without changing my LB much in the past few days. As for kernel, my code is very similar to the public kernel but with different features. BTW there is a good discussion about using public kernels here.
", "rawMarkdown": "@mmanishh, I have a simple 5 fold-CV at session level and like you I have tried many things that boosted my CV without changing my LB much in the past few days. As for kernel, my code is very similar to the public kernel but with different features. BTW there is a good discussion about using public kernels [here][1].\n\n\n [1]: https://www.kaggle.com/c/porto-seguro-safe-driver-prediction/discussion/44853#252748", "votes": 1 }, { "id": 397946, "postDate": "2018-10-03T11:35:18.933Z", "content": "Where +1 came from? Every session has at least one pageview and one hit so in both cases the denominator isn't zero.
", "rawMarkdown": "Where +1 came from? Every session has at least one pageview and one hit so in both cases the denominator isn't zero." }, { "id": 397952, "postDate": "2018-10-03T11:44:30.970Z", "content": "In my code i tried different transformations of hits and pageviews, I added a +1 to all denominators in my code to cover cases like log(hits) / log(pagevews)...but as you noticed it is not necessary for hits / pageviews...guess I am too lazy to change it ;).
", "rawMarkdown": "In my code i tried different transformations of hits and pageviews, I added a +1 to all denominators in my code to cover cases like log(hits) / log(pagevews)...but as you noticed it is not necessary for hits / pageviews...guess I am too lazy to change it ;)." }, { "id": 397953, "postDate": "2018-10-03T11:46:21.827Z", "content": ":) It's ok. I was just afraid that I'm doing something wrong.
", "rawMarkdown": ":) It's ok. I was just afraid that I'm doing something wrong." }, { "id": 399060, "postDate": "2018-10-05T04:57:27.557Z", "content": "Might be a dumb question but how did you determine the usefulness of the transformations you performed? Do you look at their correlation with the response or do you have different criteria? Be gentle, I'm new haha
", "rawMarkdown": "Might be a dumb question but how did you determine the usefulness of the transformations you performed? Do you look at their correlation with the response or do you have different criteria? Be gentle, I'm new haha " } ] }, { "id": 394478, "postDate": "2018-09-26T23:24:37.540Z", "rawMarkdown": "", "isDeleted": true }, { "id": 393663, "postDate": "2018-09-25T18:26:47.753Z", "content": "Thanks for sharing! Improved my result.
", "rawMarkdown": "Thanks for sharing! Improved my result." } ], "comments": [ { "id": 397540, "author_name": "narsil (jobs-in-data.com)", "author_url": "", "post_date": "2018-10-02T16:34:54.653000", "content": "I observe similar behaviour as descibred by @fatihozturk or @AhmetErdem: my CV improves, but my LB decreases.
\n\nI was trying to go one level deeper in the thinking. We have the following situation:
\n\nEven if we don't include the ratio A/B as an explicit feature, LGBM will still try to 'find it' via tree splits. So, my point is we cannot simply not include data['totals.hits'] / data['totals.pageviews'] as a feature and forget about it.
\n\nI don't have a ready prescription of what we should do, but until I understand why this happens, I feel I don't understand this competition.
", "votes": 8, "replies": [ { "id": 398844, "author_name": "redshellspy", "author_url": "", "post_date": "2018-10-04T16:52:29.273000", "content": "I think tree based models find diffcult to learn the ratio of two features.
\n\nReference:\nhttps://arxiv.org/pdf/1701.07852.pdf
", "votes": 3, "replies": [] }, { "id": 398927, "author_name": "Jerome Vallet", "author_url": "", "post_date": "2018-10-04T20:16:05.060000", "content": "Very interesting @Suchith !
", "votes": 0, "replies": [] } ] }, { "id": 395176, "author_name": "mightyoz", "author_url": "", "post_date": "2018-09-28T04:59:55.027000", "content": "Totally agree that customers that purchase do more hits per pageview (or equivalently smaller pageview/hit) by clicking the product, adding to cart, checking out, etc. But Adam's screenshot shows the other way: you see non-purchaser instead of purchaser at the lower end of pageviews/hit (between 0 and 0.4). Why?
", "votes": 3, "replies": [ { "id": 398142, "author_name": "SamNKH", "author_url": "", "post_date": "2018-10-03T15:07:21.557000", "content": "You should focus at the purchaser graph. Purchasers start from 0.4 onwards. If you look at non-purchaser graph, it just shows everyone else that is browsing without purchasing anything.
", "votes": 1, "replies": [] } ] }, { "id": 397417, "author_name": "Fatih Öztürk", "author_url": "", "post_date": "2018-10-02T13:08:44.783000", "content": "I've been using this featuredata['totals.hits'] / data['totals.pageviews']since my very first submission. I was not aware of its impact to LB. Today I excluded that feature from my best single model and it worsened kfold CV by 0.017 and improved LB by 0.0003. Also worsened my time-split validation by 0.011. This is so weird and I feel like I'm lost in this competition's LB-CV movement. We need to do deeper analysis for train-test differences or just wait for a significant shake-up.
Had the same issue... and not only with the ratio but with a lot of my time based variables too.
", "votes": 2, "replies": [] }, { "id": 397442, "author_name": "Kain", "author_url": "", "post_date": "2018-10-02T13:39:36.963000", "content": "Hey Fatih, interesting! people including you have different experiences using this single feature in their models. BTW, I am also waiting for a considerable shake-up because of issues like data imbalance, temporal nature of this problem and the evaluation metric (RMSE). Specially RMSE could lure many people into putting more faith in their LB (and hill climbing on LB) score rather than their CV. I hope this is not another Porto Seguro ;-).
", "votes": 0, "replies": [] }, { "id": 397451, "author_name": "Fatih Öztürk", "author_url": "", "post_date": "2018-10-02T13:53:53.653000", "content": "@areveillon Actually most of my features improving CV also improve LB, but I couldn't measure the effect of this one since my first submission already had that. However, there are also many people saying what you said. Don't know what causes this difference.
\n\nHi @kainsama , the factors you counted are already contributers for a natural shake-up. That's right we should go with our local validations as you said, but this type of a weirdness make me question 'Go with your CV' motto as well. If features don't show same pattern related to target in train and test what can we do.. Btw, did you test the impact of this feature in your current best model or it was an old test?
", "votes": 1, "replies": [] }, { "id": 397461, "author_name": "Kain", "author_url": "", "post_date": "2018-10-02T14:06:50.490000", "content": "Regarding your question, it was one the best features in my old and new models...and this feature also makes sense logically but it seems it works differently for different people and this baffles me.
", "votes": 2, "replies": [] }, { "id": 397673, "author_name": "Cozy Doomer", "author_url": "", "post_date": "2018-10-02T22:29:19.300000", "content": "Hey Kain, could you elaborate on how RSME would lure people into putting more faith in the LB?\nAm i wrong in assuming I should measure my model in it's rmse (local not LB)?
\n\nBTW you guys are amazing, since starting competitions on this website I learned so much by reading the discussions and kernels you provide so just a quick thank you!
", "votes": 1, "replies": [] }, { "id": 397875, "author_name": "Kain", "author_url": "", "post_date": "2018-10-03T09:02:36.607000", "content": "Hey Christian, If we refer to the formula for RMSE we see that it is a function of d (i.e. f(d)) where d is a vector of squared differences between our predictions and the true values. As we know mean is biased towards extreme values even if there are a few of them in our vector. if we could somehow could adjust our model to have relatively smaller difference values for a couple of those extreme values we end up with a model with a relatively good score. The problem arises when CV and LB do not follow the same path so it lures many people to trust LB that is not a robust estimator of our model' performance; it is usually like a trap because the final medals and points are given based on the PB. This is troublesome when a lot of teams are clustered around a certain error value and we are fighting over 0.001+.
", "votes": 1, "replies": [] }, { "id": 398311, "author_name": "Antoine", "author_url": "", "post_date": "2018-10-03T21:53:16.260000", "content": "@fatihozturk no idea how you choose your feature, but you are pretty good at it ! I tried a lot of features where my local CV for session or user based decrease but my LB increase, which is very frustrating because I don't know if I can fully trust my CV
", "votes": 0, "replies": [] }, { "id": 398692, "author_name": "Joe Ho", "author_url": "", "post_date": "2018-10-04T13:01:51.477000", "content": "Hi Kain,
\n\ncould you elaborate more on\" if we could somehow could adjust our model to have relatively smaller difference values for a couple of those extreme values we end up with a model with a relatively good score. \"?
\n\nI did some error analysis on my model, seems prediction for nonzero value is right-skewed, apparently it is affected by many zero target, and I arrive similar conclusion as yours, I tried to increase a bit on the zero target value, say 1, but the prediction becomes worse.
", "votes": 0, "replies": [] }, { "id": 398797, "author_name": "Kain", "author_url": "", "post_date": "2018-10-04T15:38:28.037000", "content": "Hey Joe, For example we have 60 values like x in our test that log(x + 1) > 21 and our predictions of them predicting those lie in range [16...18], if we could somehow decrease the gap between the real values and our predictions then we will have good boost; but how can we just effect those predictions without adding a constant number to all or our predictions? or what do those big spenders have in common that can help our predictions? I wish i could elaborate more but i can't.... ;-)
", "votes": 1, "replies": [] } ] }, { "id": 394960, "author_name": "AmirH", "author_url": "", "post_date": "2018-09-27T18:13:21.487000", "content": "I did find rows where ration <0.4 had transactions. but <0.35 are all without transaction.\nSo any of you that tried the 0.4 and got worse results - try 0.35 instead.
", "votes": 4, "replies": [] }, { "id": 394364, "author_name": "Sam Shleifer", "author_url": "", "post_date": "2018-09-26T18:16:54.477000", "content": "Whats the difference between a hit and a page view? Number of clicks vs number of refreshes/new pages?
", "votes": 4, "replies": [ { "id": 394374, "author_name": "Erik Bruin", "author_url": "", "post_date": "2018-09-26T18:41:24.557000", "content": "Hit: Pages are comprised of files. For example, if you have a page with 10 pictures, then a request to a server to view that page generates 11 hits (10 for the pictures, and one for the html file).
Please have a look at my EDA. There's more stuff like that in there ;-)\nhttps://www.kaggle.com/erikbruin/google-analytics-eda-with-screenshots-of-the-app
", "votes": 6, "replies": [] }, { "id": 394380, "author_name": "S D", "author_url": "", "post_date": "2018-09-26T18:55:32.650000", "content": "Erik:
\n\nWhat you describe are hits from the perspective of the server hosting the website, which is different from google analytics hits.
\n\nHere, hits include page views, but also other events such as clicking on a product to view more details, or adding a product to cart.
\n\nhttps://support.google.com/analytics/answer/6086082?hl=en
", "votes": 10, "replies": [] }, { "id": 394381, "author_name": "Erik Bruin", "author_url": "", "post_date": "2018-09-26T18:58:15.073000", "content": "Ok, thanks! Will have another look.....
", "votes": 0, "replies": [] }, { "id": 398144, "author_name": "SamNKH", "author_url": "", "post_date": "2018-10-03T15:10:32.107000", "content": "So does hits basically mean user actions?
", "votes": 0, "replies": [] } ] }, { "id": 393401, "author_name": "Prashanth Thangavel", "author_url": "", "post_date": "2018-09-25T09:54:16.703000", "content": "Thanks for sharing! Super useful insight
", "votes": 1, "replies": [] }, { "id": 393810, "author_name": "S D", "author_url": "", "post_date": "2018-09-25T23:02:48.913000", "content": "I've had similar findings. Not surprising given that hits include pageviews but also other user actions such as adding items to cart, viewing a product etc.
", "votes": 0, "replies": [ { "id": 394287, "author_name": "JohnM", "author_url": "", "post_date": "2018-09-26T16:04:03.380000", "content": "Yep. Also the inverse, hits per page, is a good indication that the user went to the shopping cart page and/or checkout. Those pages typically have more elements (hits) that load into the page as compared to a home page or product view.
", "votes": 5, "replies": [] } ] }, { "id": 397628, "author_name": "j_scheibel", "author_url": "", "post_date": "2018-10-02T20:28:48.787000", "content": "Oddly, after you shared this one thing led to another and this got me thinking about a new way to do feature splitting in my algorithms without significantly increasing the runtime, so yay for that.
", "votes": 0, "replies": [] }, { "id": 395754, "author_name": "upup", "author_url": "", "post_date": "2018-09-29T09:27:47.840000", "content": "I tried both _df[''totals.pageviews] / (data_df['totals.hits']+1) and data_df['totals.hits'] / (data_df['totals.pageviews']+1) , and it lead to a considerable CV boost but decrease on LB, sad:( .
", "votes": 0, "replies": [] }, { "id": 394708, "author_name": "Arthurim", "author_url": "", "post_date": "2018-09-27T09:42:58.747000", "content": "You should keep in mind that patterns in the CV <> patterns in the test set...
", "votes": 0, "replies": [] }, { "id": 394668, "author_name": "Ahmet Erdem", "author_url": "", "post_date": "2018-09-27T08:16:22.457000", "content": "This feature improves my CV but worsens my LB. Does anyone else experience the same?
", "votes": 0, "replies": [ { "id": 394682, "author_name": "Jerome Vallet", "author_url": "", "post_date": "2018-09-27T08:49:28.573000", "content": "The same for me. Important CV improvment but a slightly worse LB score. finally i don't know if I should keep it or not...
", "votes": 1, "replies": [] }, { "id": 394686, "author_name": "Angus Chang", "author_url": "", "post_date": "2018-09-27T08:53:52.423000", "content": "same here, improve 0.001 CV but worsen my LB about 0.001 .
", "votes": 0, "replies": [] }, { "id": 396973, "author_name": "Andrew", "author_url": "", "post_date": "2018-10-01T17:46:41.213000", "content": "Yes - I started with this helpful user-level Light GBM-based notebook and then added pageviews/hits as an additional feature without making any other changes at all. The local CV improved by 0.013 (1.58154 to 1.56890) but the LB score got worse by 0.01 (1.4473 to 1.4568). Does anybody know why supplying an additional feature to a Light GBM model might make it get worse?
", "votes": 1, "replies": [] }, { "id": 397052, "author_name": "Ahmet Erdem", "author_url": "", "post_date": "2018-10-01T20:12:35.637000", "content": "@Andrew Most probably, it is due to the difference between train and test set. This variable has different effect on the test it seems. If you use something else instead of lightgbm, I guess you will observe the same effect.
", "votes": 1, "replies": [] } ] }, { "id": 393956, "author_name": "trphng", "author_url": "", "post_date": "2018-09-26T05:43:42.780000", "content": "Thanks for sharing this helpful finding!
", "votes": 0, "replies": [] }, { "id": 393747, "author_name": "Kain", "author_url": "", "post_date": "2018-09-25T20:43:41.063000", "content": "I tried this as a feature and it lead to a considerable CV boost. Thanx, now i know why ;-).
", "votes": 0, "replies": [ { "id": 394240, "author_name": "Prashanth Thangavel", "author_url": "", "post_date": "2018-09-26T14:48:28.813000", "content": "Did u get a boost in LB as well ? Because I fed this variable (as a numerical one) and the LB seemed to be the same
", "votes": 1, "replies": [] }, { "id": 394252, "author_name": "Kain", "author_url": "", "post_date": "2018-09-26T14:57:43.107000", "content": "Yes
", "votes": 0, "replies": [] }, { "id": 394382, "author_name": "Shahebaz Mohammad", "author_url": "", "post_date": "2018-09-26T19:02:39.840000", "content": "I am getting a marginal boost in CVs but significant at the LB. I am yet to try this new feature. Thanks!
", "votes": 0, "replies": [] }, { "id": 394750, "author_name": "Manish Maharjan", "author_url": "", "post_date": "2018-09-27T11:36:49.083000", "content": "Hi Kain, how did you transform you new feature ? I assigned 0 for pageview/hits ratio < 0.4 and 1 for other , didn't improve any score.
", "votes": 0, "replies": [] }, { "id": 394754, "author_name": "Kain", "author_url": "", "post_date": "2018-09-27T11:46:49.923000", "content": "Hi Manish, I am using, totals.hits / (totals.pageViews + 1) as a feature and it is giving a good boost to my CV and also improves my LB but the boost in CV is more tangible. BTW, it is usually better to let your model use these kinds of features instead of using them explicitly in your predictions (e.g. hand-labeling).
", "votes": 6, "replies": [] }, { "id": 394929, "author_name": "", "author_url": "", "post_date": "2018-09-27T16:48:03.127000", "content": "", "votes": 1, "replies": [] }, { "id": 394933, "author_name": "Adams", "author_url": "", "post_date": "2018-09-27T16:55:58.100000", "content": "Because it's an implication - you reduce amount of information available for a model instead of leting it decide what exactly about a feature is important.
", "votes": 1, "replies": [] }, { "id": 394935, "author_name": "", "author_url": "", "post_date": "2018-09-27T17:13:54.100000", "content": "", "votes": 0, "replies": [] }, { "id": 394943, "author_name": "Kain", "author_url": "", "post_date": "2018-09-27T17:31:46.047000", "content": "Hi @wenshihao1993, IMHO, For the following reasons:
\n\nI used the word usually for describing my opinion because there some Kagglers (many top Kagglers) that spend days or even weeks on finding insights (possibly leaks ;-)) to boost their scores based on their analysis.
", "votes": 7, "replies": [] }, { "id": 394947, "author_name": "", "author_url": "", "post_date": "2018-09-27T17:46:44.233000", "content": "", "votes": 1, "replies": [] }, { "id": 394958, "author_name": "CoreyJamesLevinson", "author_url": "", "post_date": "2018-09-27T18:10:30.063000", "content": "@Kain I disagree somewhat with your suggestion. Using both hand-labeling and feeding as a feature seems best. For example, we have always been feeding totals.bounces as a feature, but sometimes our model predicts positive revenue for when totals.bounces==1, even though we know that it must be zero.
Thanks for sharing. I've used totals.hits / totals.pageViews as a new feature and it also gave me a boost in my local CV while my LB decreased about 0.04. I think I need to give totals.hits / (totals.pageViews + 1) a try.
", "votes": 1, "replies": [] }, { "id": 395184, "author_name": "Manish Maharjan", "author_url": "", "post_date": "2018-09-28T05:18:50.043000", "content": "Hi Kain , shouldn't we be using train_df[''totals.pageviews] / (train_df['totals.hits']+1) instead of train_df['totals.hits'] / (train_df['totals.pageviews']+1) , because the visualization posted in this thread is about pageviews/ hits ratio rather that hits/pageviews ratio.
", "votes": 0, "replies": [] }, { "id": 395223, "author_name": "Kain", "author_url": "", "post_date": "2018-09-28T06:48:39.377000", "content": "I am not sure but some algorithms are indifferent to the choice between a / (b + 1) and b / (a + 1); the usual practice is to give them a try in a validation scheme to see what works best.
", "votes": 0, "replies": [] }, { "id": 395224, "author_name": "Manish Maharjan", "author_url": "", "post_date": "2018-09-28T06:51:00.810000", "content": "Used both , there is no effect on LB score . What CV strategy are you using ? Timebased ? Can you share your kernel once ?
", "votes": 0, "replies": [] }, { "id": 395227, "author_name": "Kain", "author_url": "", "post_date": "2018-09-28T06:56:58.127000", "content": "@returnofsputnik I agree with you in a sense that there are exceptions where things like hand-labeling would get you the 1st place; What I learned by competing at Kaggle is that we should try as many ideas as we could. At the end of the day, whatever works is the winner.
", "votes": 1, "replies": [] }, { "id": 395241, "author_name": "Kain", "author_url": "", "post_date": "2018-09-28T07:14:45.097000", "content": "@mmanishh, I have a simple 5 fold-CV at session level and like you I have tried many things that boosted my CV without changing my LB much in the past few days. As for kernel, my code is very similar to the public kernel but with different features. BTW there is a good discussion about using public kernels here.
", "votes": 1, "replies": [] }, { "id": 397946, "author_name": "Yannis Pappas", "author_url": "", "post_date": "2018-10-03T11:35:18.933000", "content": "Where +1 came from? Every session has at least one pageview and one hit so in both cases the denominator isn't zero.
", "votes": 0, "replies": [] }, { "id": 397952, "author_name": "Kain", "author_url": "", "post_date": "2018-10-03T11:44:30.970000", "content": "In my code i tried different transformations of hits and pageviews, I added a +1 to all denominators in my code to cover cases like log(hits) / log(pagevews)...but as you noticed it is not necessary for hits / pageviews...guess I am too lazy to change it ;).
", "votes": 0, "replies": [] }, { "id": 397953, "author_name": "Yannis Pappas", "author_url": "", "post_date": "2018-10-03T11:46:21.827000", "content": ":) It's ok. I was just afraid that I'm doing something wrong.
", "votes": 0, "replies": [] }, { "id": 399060, "author_name": "🐳", "author_url": "", "post_date": "2018-10-05T04:57:27.557000", "content": "Might be a dumb question but how did you determine the usefulness of the transformations you performed? Do you look at their correlation with the response or do you have different criteria? Be gentle, I'm new haha
", "votes": 0, "replies": [] } ] }, { "id": 394478, "author_name": "", "author_url": "", "post_date": "2018-09-26T23:24:37.540000", "content": "", "votes": 0, "replies": [] }, { "id": 393663, "author_name": "evers", "author_url": "", "post_date": "2018-09-25T18:26:47.753000", "content": "Thanks for sharing! Improved my result.
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "393379": "Below is the screenshot of distributions of pageviews/hits ratios for the segments with revenue >0 and revenue = 0 (logarithmic y scales). Although huge majority of observations have the score close to 1, the several outliers are guaranteed to yield no profit (candidates for post processing ;) )", "397540": "I observe similar behaviour as descibred by @fatihozturk or @AhmetErdem: my CV improves, but my LB decreases.\n\nI was trying to go one level deeper in the thinking. We have the following situation:\n\n - you have features A and B, they are both strong features and their ratio is also a strong feature in your model\n - the ratio A/B makes a lot of sense as a feature from business perspective\n\nEven if we don't include the ratio A/B as an explicit feature, LGBM will still try to 'find it' via tree splits. So, my point is we cannot simply not include data['totals.hits'] / data['totals.pageviews'] as a feature and forget about it. \n\nI don't have a ready prescription of what we should do, but until I understand why this happens, I feel I don't understand this competition.", "395176": "Totally agree that customers that purchase do more hits per pageview (or equivalently smaller pageview/hit) by clicking the product, adding to cart, checking out, etc. But Adam's screenshot shows the other way: you see non-purchaser instead of purchaser at the lower end of pageviews/hit (between 0 and 0.4). Why?", "397417": "I've been using this feature`data['totals.hits'] / data['totals.pageviews']`since my very first submission. I was not aware of its impact to LB. Today I excluded that feature from my best single model and it worsened kfold CV by 0.017 and improved LB by 0.0003. Also worsened my time-split validation by 0.011. This is so weird and I feel like I'm lost in this competition's LB-CV movement. We need to do deeper analysis for train-test differences or just wait for a significant shake-up.", "394960": "I did find rows where ration <0.4 had transactions. but <0.35 are all without transaction.\nSo any of you that tried the 0.4 and got worse results - try 0.35 instead.", "394364": "Whats the difference between a hit and a page view? Number of clicks vs number of refreshes/new pages?", "393401": "Thanks for sharing! Super useful insight", "393810": "I've had similar findings. Not surprising given that hits include pageviews but also other user actions such as adding items to cart, viewing a product etc.", "397628": "Oddly, after you shared this one thing led to another and this got me thinking about a new way to do feature splitting in my algorithms without significantly increasing the runtime, so yay for that.", "395754": "I tried both _df[''totals.pageviews] / (data_df['totals.hits']+1) and data_df['totals.hits'] / (data_df['totals.pageviews']+1) , and it lead to a considerable CV boost but decrease on LB, sad:( . ", "394708": "You should keep in mind that patterns in the CV <> patterns in the test set...", "394668": "This feature improves my CV but worsens my LB. Does anyone else experience the same?", "393956": "Thanks for sharing this helpful finding!", "393747": "I tried this as a feature and it lead to a considerable CV boost. Thanx, now i know why ;-).", "394478": "", "393663": "Thanks for sharing! Improved my result." } }