{
  "id": 65995,
  "title": "Sum of Logs mystery - higher spending on Public LB ?",
  "url": "/competitions/ga-customer-revenue-prediction/discussion/65995",
  "author_name": "olivier",
  "post_date": "2018-09-17T07:04:39.910000",
  "votes": 50,
  "comment_count": 66,
  "views": 0,
  "content": "<p>There has been some discussion on the fact the competition metric/scoring was unclear...</p>\n\n<p>Craig Armstrong and yilisg requested I do an experiment with my <a href=\"https://www.kaggle.com/ogrellier/user-level-lightgbm\">public kernel</a> and to my surprise the LB score improved substantially to around 1.70 when CV increased to above 2 !</p>\n\n<p>The kernel trains a model at user level and not session level. The experiment was to aggregate <code>np.log1p(transcationRevenue)</code> instead of <code>transactionRevenue</code>.</p>\n\n<p>Most of the visitors (620675 in train) have only one session and  summing up before of after going to log does not change anything for them.</p>\n\n<p>For the others, if they only have 1 session with a non zero revenue it does not change anything either. </p>\n\n<p>There are 93492 Visitors with more than 1 session in train. On these, 86616 have 0 revenue and 5909 only have 1 session with a non zero revenue !</p>\n\n<p>This means you only have 968 visitors with at least 2 non zero transactionRevenue. That's not a lot but these guys have very high spendings : <code>min=10000, max=77113430000, median=57195000</code></p>\n\n<p>The question then becomes : does public LB have higher spending ? </p>\n\n<p><code>\nnp.log1p(1 + 20000) = 9.90..  and  2 * np.log1p(1 + 10000) = 18.42..\n</code></p>",
  "messages": [
    {
      "id": 388546,
      "postDate": "2018-09-17T07:04:39.910Z",
      "content": "<p>There has been some discussion on the fact the competition metric/scoring was unclear...</p>\n\n<p>Craig Armstrong and yilisg requested I do an experiment with my <a href=\"https://www.kaggle.com/ogrellier/user-level-lightgbm\">public kernel</a> and to my surprise the LB score improved substantially to around 1.70 when CV increased to above 2 !</p>\n\n<p>The kernel trains a model at user level and not session level. The experiment was to aggregate <code>np.log1p(transcationRevenue)</code> instead of <code>transactionRevenue</code>.</p>\n\n<p>Most of the visitors (620675 in train) have only one session and  summing up before of after going to log does not change anything for them.</p>\n\n<p>For the others, if they only have 1 session with a non zero revenue it does not change anything either. </p>\n\n<p>There are 93492 Visitors with more than 1 session in train. On these, 86616 have 0 revenue and 5909 only have 1 session with a non zero revenue !</p>\n\n<p>This means you only have 968 visitors with at least 2 non zero transactionRevenue. That's not a lot but these guys have very high spendings : <code>min=10000, max=77113430000, median=57195000</code></p>\n\n<p>The question then becomes : does public LB have higher spending ? </p>\n\n<p><code>\nnp.log1p(1 + 20000) = 9.90..  and  2 * np.log1p(1 + 10000) = 18.42..\n</code></p>",
      "rawMarkdown": "There has been some discussion on the fact the competition metric/scoring was unclear...\n\nCraig Armstrong and yilisg requested I do an experiment with my [public kernel](https://www.kaggle.com/ogrellier/user-level-lightgbm) and to my surprise the LB score improved substantially to around 1.70 when CV increased to above 2 !\n\nThe kernel trains a model at user level and not session level. The experiment was to aggregate `np.log1p(transcationRevenue)` instead of `transactionRevenue`.\n\nMost of the visitors (620675 in train) have only one session and  summing up before of after going to log does not change anything for them.\n\nFor the others, if they only have 1 session with a non zero revenue it does not change anything either. \n\nThere are 93492 Visitors with more than 1 session in train. On these, 86616 have 0 revenue and 5909 only have 1 session with a non zero revenue !\n\nThis means you only have 968 visitors with at least 2 non zero transactionRevenue. That's not a lot but these guys have very high spendings : `min=10000, max=77113430000, median=57195000`\n\nThe question then becomes : does public LB have higher spending ? \n\n```\nnp.log1p(1 + 20000) = 9.90..  and  2 * np.log1p(1 + 10000) = 18.42..\n```\n\n",
      "votes": 50
    },
    {
      "id": 389172,
      "postDate": "2018-09-18T09:37:54.230Z",
      "content": "<p>My two cents on the original topic. One of my first test was with the the logp1 s of the sum customer revenue as estimates per customer. That scored 1.74. The sum of logp1's scored 1.71. Observing that the mean estimate was hogher than the mean actual on train, I multiplied the latter with mean(train)/mean(test) and that scored 1.66. The conclusio is obvious: this has nothing to do with the sum of logs versus the logs of sums; it is caused by the extremely unbalanced target variable.</p>\n\n<p>Almost all scripts here convert the target variable to logp1(y) almost without considering other options, But you have to decouple this from the conversion in the objective function.  Why not log1p(log1p(y))? Why not y** 1/8? Other competitions have shown me that (for xgboost) it can make a real difference and it is worth experimenting with.</p>\n\n<p>Any thoughts on the CV? Should we stratify on fullVisionId?</p>",
      "rawMarkdown": "My two cents on the original topic. One of my first test was with the the logp1 s of the sum customer revenue as estimates per customer. That scored 1.74. The sum of logp1's scored 1.71. Observing that the mean estimate was hogher than the mean actual on train, I multiplied the latter with mean(train)/mean(test) and that scored 1.66. The conclusio is obvious: this has nothing to do with the sum of logs versus the logs of sums; it is caused by the extremely unbalanced target variable.\n\nAlmost all scripts here convert the target variable to logp1(y) almost without considering other options, But you have to decouple this from the conversion in the objective function.  Why not log1p(log1p(y))? Why not y** 1/8? Other competitions have shown me that (for xgboost) it can make a real difference and it is worth experimenting with.\n\nAny thoughts on the CV? Should we stratify on fullVisionId?",
      "votes": 27,
      "replies": [
        {
          "id": 389179,
          "postDate": "2018-09-18T09:51:38.150Z",
          "content": "<p>Good to see you around <a href=\"/mrooijer\">@mrooijer</a> ! Thanks for sharing your findings with us.</p>\n\n<p>You're absolutely right on the metrics used for modeling. And I guess <code>GroupKFold</code> on <code>fullVisitorId</code> would avoid having same visitors spanning different folds.</p>",
          "rawMarkdown": "Good to see you around @mrooijer ! Thanks for sharing your findings with us.\n\nYou're absolutely right on the metrics used for modeling. And I guess `GroupKFold` on `fullVisitorId` would avoid having same visitors spanning different folds.\n",
          "votes": 4
        },
        {
          "id": 389527,
          "postDate": "2018-09-18T21:23:02.503Z",
          "content": "<p>Hey Olivier,</p>\n\n<p>The thing is that some of the users in the training set are also in the test set. If you do a time-based cross-validation this property holds. In my opinion going with a time-based cross-validation is closer to reality. What's more it seems to work slightly better for me, at least locally.</p>",
          "rawMarkdown": "Hey Olivier,\n\nThe thing is that some of the users in the training set are also in the test set. If you do a time-based cross-validation this property holds. In my opinion going with a time-based cross-validation is closer to reality. What's more it seems to work slightly better for me, at least locally.",
          "votes": 6
        },
        {
          "id": 389728,
          "postDate": "2018-09-19T06:15:28.090Z",
          "rawMarkdown": "",
          "isDeleted": true
        },
        {
          "id": 389735,
          "postDate": "2018-09-19T06:29:18.090Z",
          "content": "<p>Hi Zijun, no I am not.</p>",
          "rawMarkdown": "Hi Zijun, no I am not."
        },
        {
          "id": 389747,
          "postDate": "2018-09-19T06:38:56.520Z",
          "rawMarkdown": "",
          "isDeleted": true
        },
        {
          "id": 389924,
          "postDate": "2018-09-19T13:19:06.283Z",
          "content": "<p>Hello, could anyone explain what does * multiply by mean(train)/mean(test)* mean ? :) Thank you !</p>",
          "rawMarkdown": "Hello, could anyone explain what does * multiply by mean(train)/mean(test)* mean ? :) Thank you !",
          "votes": 2
        },
        {
          "id": 390108,
          "postDate": "2018-09-19T18:14:48.717Z",
          "content": "<p>mean of all log1p target values / mean of all predicted log1p observation values. It was something like 0.23/0.27</p>",
          "rawMarkdown": "mean of all log1p target values / mean of all predicted log1p observation values. It was something like 0.23/0.27",
          "votes": 3
        },
        {
          "id": 390509,
          "postDate": "2018-09-20T10:09:31.680Z",
          "content": "<p><a href=\"/mrooijer\">@mrooijer</a>\nCould you please elaborate on what do you mean by </p>\n\n<blockquote>\n  <p>But you have to decouple this from the conversion in the objective function.</p>\n</blockquote>\n\n<p>To a beginner like me it is natural to consider only log1p(sum(revenue)) or sum(log1p(revenue)), as the first is what the organisers claim to be used in eveluation and the second is a guesstimate of what can be used in practice due to some unclear feature. What is the motivation to try something else?</p>\n\n<p>Do you make an implied assumption that the distribution of target in the submission sample is very different and some other (i.e. non-adverticed) target can empirically lead to a better submission metric?</p>\n\n<p>Or is motivation something different?</p>",
          "rawMarkdown": "@mrooijer\nCould you please elaborate on what do you mean by \n\n&gt; But you have to decouple this from the conversion in the objective function.\n\nTo a beginner like me it is natural to consider only log1p(sum(revenue)) or sum(log1p(revenue)), as the first is what the organisers claim to be used in eveluation and the second is a guesstimate of what can be used in practice due to some unclear feature. What is the motivation to try something else?\n\nDo you make an implied assumption that the distribution of target in the submission sample is very different and some other (i.e. non-adverticed) target can empirically lead to a better submission metric?\n\nOr is motivation something different?"
        },
        {
          "id": 390615,
          "postDate": "2018-09-20T14:15:33.723Z",
          "content": "<p>Hi <a href=\"/maxhalford\">@maxhalford</a>, could you elaborate on what did you mean by time-based cross validation (I guess this not the same as sklearn's TimeSeriesSplit)</p>",
          "rawMarkdown": "Hi @maxhalford, could you elaborate on what did you mean by time-based cross validation (I guess this not the same as sklearn's TimeSeriesSplit)",
          "votes": 1
        },
        {
          "id": 390631,
          "postDate": "2018-09-20T14:54:00.150Z",
          "content": "<p>No it's not the same as <code>TimeSeriesSplit</code> which, if I am not mistaken, uses only one row in the validation set. The idea of time-based cross-validation is simply to use the last 20% of your data as the validation set. Then you can apply this recursively so that effectively you're walking back in time.</p>\n\n<p>Talk is cheap, <a href=\"https://github.com/MaxHalford/xam/blob/master/docs/model-selection.md#ordered-cross-validation\">show me the code</a>.</p>",
          "rawMarkdown": "No it's not the same as `TimeSeriesSplit` which, if I am not mistaken, uses only one row in the validation set. The idea of time-based cross-validation is simply to use the last 20% of your data as the validation set. Then you can apply this recursively so that effectively you're walking back in time.\n\nTalk is cheap, [show me the code](https://github.com/MaxHalford/xam/blob/master/docs/model-selection.md#ordered-cross-validation).",
          "votes": 4
        },
        {
          "id": 391043,
          "postDate": "2018-09-21T07:12:44.037Z",
          "content": "<p><a href=\"/mrooijer\">@mrooijer</a> actually, is i scale predictions by mean(train_target)/mean(train_oof_prediction), then my score gets worse by 0.02. The other two observations are similar</p>",
          "rawMarkdown": "@mrooijer actually, is i scale predictions by mean(train_target)/mean(train_oof_prediction), then my score gets worse by 0.02. The other two observations are similar"
        }
      ]
    },
    {
      "id": 391232,
      "postDate": "2018-09-21T12:45:27.380Z",
      "content": "<p>My guess:</p>\n\n<p><img src=\"https://i.imgur.com/CVcd1mJ.png\" alt=\"enter image description here\"></p>\n\n<p>I plotted the train_targets (blue line), the train_predictions (green line) in the above image. What I can see is my predictions are always lower than the actual target. Also, my model failed completely in predicting the revenue spike. </p>\n\n<p>I assume most of our predictions are similar (lower than the actual target &amp; failed to predict spike). </p>\n\n<p>Since sum of logs is larger than log of sums: log(a) + log(b) = log(ab) is always &gt;= log(a+b), where a &amp; b &gt;= 1, it is like scaling our \"under-shotting\" predictions up to fit those revenue spike. And I guess that's why most people found that sum of logs gave them better lb result. </p>",
      "rawMarkdown": "My guess:\n\n![enter image description here][1]\n\nI plotted the train_targets (blue line), the train_predictions (green line) in the above image. What I can see is my predictions are always lower than the actual target. Also, my model failed completely in predicting the revenue spike. \n\nI assume most of our predictions are similar (lower than the actual target &amp; failed to predict spike). \n\nSince sum of logs is larger than log of sums: log(a) + log(b) = log(ab) is always &gt;= log(a+b), where a &amp; b &gt;= 1, it is like scaling our \"under-shotting\" predictions up to fit those revenue spike. And I guess that's why most people found that sum of logs gave them better lb result. \n\n  [1]: https://i.imgur.com/CVcd1mJ.png",
      "votes": 19
    },
    {
      "id": 388723,
      "postDate": "2018-09-17T13:45:40.563Z",
      "content": "<p>At least, transaction frequency in test looks higher than that in train. <br>\nBut it does not directly mean higher spending in test data.  </p>\n\n<p><img src=\"https://cdn-ak.f.st-hatena.com/images/fotolife/g/greenwind120170/20180917/20180917230237.png\" alt=\"transaction comparison\"></p>",
      "rawMarkdown": "At least, transaction frequency in test looks higher than that in train.  \nBut it does not directly mean higher spending in test data.  \n  \n![transaction comparison][1]\n\n\n  [1]: https://cdn-ak.f.st-hatena.com/images/fotolife/g/greenwind120170/20180917/20180917230237.png",
      "votes": 16,
      "replies": [
        {
          "id": 389085,
          "postDate": "2018-09-18T05:59:38.703Z",
          "content": "<p>Indeed. What also puzzles me i that in 2017 there was only a narrow spike next to Christmas, whereas in 2016 it was a 1-2-month-wide bump</p>",
          "rawMarkdown": "Indeed. What also puzzles me i that in 2017 there was only a narrow spike next to Christmas, whereas in 2016 it was a 1-2-month-wide bump",
          "votes": 3
        },
        {
          "id": 389092,
          "postDate": "2018-09-18T06:24:05.543Z",
          "content": "<p>Yes. <br>\nI think this spike came from special event related to Christmas, IT tech, GMS itself or others. <br>\nFollowing picture is the comparison between train and test from <code>US</code> only. <br>\n( Some other countries would have the same tendency, I think. ) <br>\nSo exploring events in US around the spike time may unwrap the reason. <br>\n<img src=\"https://cdn-ak.f.st-hatena.com/images/fotolife/g/greenwind120170/20180918/20180918151936.png\" alt=\"US transactions\"></p>",
          "rawMarkdown": "Yes.  \nI think this spike came from special event related to Christmas, IT tech, GMS itself or others.  \nFollowing picture is the comparison between train and test from `US` only.  \n( Some other countries would have the same tendency, I think. )  \nSo exploring events in US around the spike time may unwrap the reason.  \n![US transactions][1]\n\n\n  [1]: https://cdn-ak.f.st-hatena.com/images/fotolife/g/greenwind120170/20180918/20180918151936.png",
          "votes": 3
        },
        {
          "id": 389120,
          "postDate": "2018-09-18T07:28:04.160Z",
          "content": "<p><a href=\"https://trends.google.com/trends/explore?q=Google%20home&amp;geo=US\">https://trends.google.com/trends/explore?q=Google%20home&amp;geo=US</a></p>\n\n<p>Google Home? I guess a lot of search terms spike around that time of year but it seems a likely candidate.</p>",
          "rawMarkdown": "https://trends.google.com/trends/explore?q=Google%20home&amp;geo=US\n\nGoogle Home? I guess a lot of search terms spike around that time of year but it seems a likely candidate.",
          "votes": 2
        },
        {
          "id": 389129,
          "postDate": "2018-09-18T07:53:30.463Z",
          "content": "<p>Hi <a href=\"/ca3295\">@ca3295</a></p>\n\n<p>Thank you for your insight. <br>\nWe can see the same sign in other countries. <br>\nIf Google Home was released in those countries where we can see the spikes, <br>\nyou may be right.  </p>\n\n<p><img src=\"https://cdn-ak.f.st-hatena.com/images/fotolife/g/greenwind120170/20180918/20180918164807.png\" alt=\"5 countries\"></p>",
          "rawMarkdown": "Hi @ca3295\n\nThank you for your insight.  \nWe can see the same sign in other countries.  \nIf Google Home was released in those countries where we can see the spikes,  \nyou may be right.  \n\n![5 countries][1]\n\n\n  [1]: https://cdn-ak.f.st-hatena.com/images/fotolife/g/greenwind120170/20180918/20180918164807.png",
          "votes": 4
        },
        {
          "id": 389133,
          "postDate": "2018-09-18T07:59:17.120Z",
          "content": "<p>I'm starting to doubt it now, as I'm not sure whether we are looking only at stores that sell t-shirts and mugs etc (the Google \"swag\" mentioned in the competition description i.e. <a href=\"https://shop.googlemerchandisestore.com/\">https://shop.googlemerchandisestore.com/</a> ) or whether it includes things like Google Home (<a href=\"https://store.google.com/\">https://store.google.com/</a> )</p>\n\n<p>Also the search interest in India seems to spike in April 2018, not late December as per your charts.</p>",
          "rawMarkdown": "I'm starting to doubt it now, as I'm not sure whether we are looking only at stores that sell t-shirts and mugs etc (the Google \"swag\" mentioned in the competition description i.e. https://shop.googlemerchandisestore.com/ ) or whether it includes things like Google Home (https://store.google.com/ )\n\nAlso the search interest in India seems to spike in April 2018, not late December as per your charts.",
          "votes": 3
        },
        {
          "id": 389528,
          "postDate": "2018-09-18T21:30:33.230Z",
          "content": "<p>Nice graph. Also notice the bias (trend)!\n@Maxwell @\nLooks like spending increases over time, so its not surprising that the test set has more spenders.</p>",
          "rawMarkdown": "Nice graph. Also notice the bias (trend)!\n@Maxwell @\nLooks like spending increases over time, so its not surprising that the test set has more spenders."
        },
        {
          "id": 390663,
          "postDate": "2018-09-20T15:47:59.407Z",
          "content": "<p>I noticed that <code>trafficSource_adContent</code> has  <code>Ad from 12/13/16</code> category.  </p>\n\n<pre><code>train.trafficSource_adContent.value_counts(dropna=False)\n\nNaN                                            892707\nGoogle Merchandise Collection                    5122\nGoogle Online Store                              1245\nDisplay Ad created 3/11/14                        967\nFull auto ad IMAGE ONLY                           822\nAd from 12/13/16                                  610\nAd from 11/3/16                                   489\nDisplay Ad created 3/11/15                        392\n{KeyWord:Google Brand Items}                      251\n{KeyWord:Google Merchandise}                      155\nAd from 11/7/16                                   123\nFirst Full Auto Template Test Ad                   87\nGoogle Merchandise                                 87  \n</code></pre>\n\n<p>Unfortunately this can not explain the boost next to Christmas day. <br>\n(The number of records is not enough. )  </p>\n\n<pre><code>train[(train.trafficSource_adContent == 'Ad from 12/13/16') &amp; (train.date &lt; dt.date(2016, 12, 30))].shape[0]\n\n185\n</code></pre>\n\n<p>But at least there might be some campaign, <br>\nand I doubt a lot of records were allocated to <code>NaN</code> category.  </p>",
          "rawMarkdown": "I noticed that `trafficSource_adContent` has  `Ad from 12/13/16` category.  \n\n\n    train.trafficSource_adContent.value_counts(dropna=False)\n    \n    NaN                                            892707\n    Google Merchandise Collection                    5122\n    Google Online Store                              1245\n    Display Ad created 3/11/14                        967\n    Full auto ad IMAGE ONLY                           822\n    Ad from 12/13/16                                  610\n    Ad from 11/3/16                                   489\n    Display Ad created 3/11/15                        392\n    {KeyWord:Google Brand Items}                      251\n    {KeyWord:Google Merchandise}                      155\n    Ad from 11/7/16                                   123\n    First Full Auto Template Test Ad                   87\n    Google Merchandise                                 87  \n\nUnfortunately this can not explain the boost next to Christmas day.  \n(The number of records is not enough. )  \n\n    train[(train.trafficSource_adContent == 'Ad from 12/13/16') &amp; (train.date &lt; dt.date(2016, 12, 30))].shape[0]\n     \n    185\n\nBut at least there might be some campaign,   \nand I doubt a lot of records were allocated to `NaN` category.  "
        },
        {
          "id": 392005,
          "postDate": "2018-09-22T20:40:26.673Z",
          "content": "<p>Nice work!</p>",
          "rawMarkdown": "Nice work!"
        }
      ]
    },
    {
      "id": 388711,
      "postDate": "2018-09-17T13:29:19.813Z",
      "content": "<p>When will people learn to just always use rank metrics like the Gini index?  It amazes me how so many data scientists do not appreciate how important the choice of metric is in the modeling process, and how most metrics reflect an implicit assumption about the distribution of data.  It seems like the thought process usually goes like:</p>\n\n<ol>\n<li>Let's try RMSE, that's simple.  Oh, our target values are highly skewed? </li>\n<li>Let's try logging the target values.  Oh, we have zero targets? </li>\n<li>Let's add one before logging.  Ok, that works, done!</li>\n</ol>",
      "rawMarkdown": "When will people learn to just always use rank metrics like the Gini index?  It amazes me how so many data scientists do not appreciate how important the choice of metric is in the modeling process, and how most metrics reflect an implicit assumption about the distribution of data.  It seems like the thought process usually goes like:\n\n 1.  Let's try RMSE, that's simple.  Oh, our target values are highly skewed? \n 2.  Let's try logging the target values.  Oh, we have zero targets? \n 3. Let's add one before logging.  Ok, that works, done!",
      "votes": 10,
      "replies": [
        {
          "id": 389145,
          "postDate": "2018-09-18T08:18:21.110Z",
          "content": "<p>Dmitriy, could you please explain how rank metrics can be used in this competition?</p>",
          "rawMarkdown": "Dmitriy, could you please explain how rank metrics can be used in this competition?",
          "votes": 2
        },
        {
          "id": 389415,
          "postDate": "2018-09-18T17:32:19.387Z",
          "content": "<blockquote>\n  <p><strong>Alexander Firsov wrote</strong></p>\n  \n  <blockquote>\n    <p>Dmitriy, could you please explain how rank metrics can be used in this competition?</p>\n  </blockquote>\n</blockquote>\n\n<p>You use a Gini index.  You sort your clients by their predicted total spend, and you accumulate their actual spend.  Only the ordering of prediction matters, not the distance between actual and predicted.  By taking this distance out of evaluation metric, you're taking subjectivity and implicit distributional assumptions out.</p>",
          "rawMarkdown": "\n&gt; **Alexander Firsov wrote**\n&gt; \n&gt; &gt; Dmitriy, could you please explain how rank metrics can be used in this competition?\n\nYou use a Gini index.  You sort your clients by their predicted total spend, and you accumulate their actual spend.  Only the ordering of prediction matters, not the distance between actual and predicted.  By taking this distance out of evaluation metric, you're taking subjectivity and implicit distributional assumptions out.",
          "votes": 2
        },
        {
          "id": 389461,
          "postDate": "2018-09-18T18:55:26.853Z",
          "content": "<p>Dmitriy, thanks for reply, what you described sounds interesting. I used Gini and ROC AUC in binary classification only, but have no idea of how this can be done in regression. \nDo you happen to have links to explanation or samples where Gini is used for regression, as I cannot google anything relevant.</p>",
          "rawMarkdown": "Dmitriy, thanks for reply, what you described sounds interesting. I used Gini and ROC AUC in binary classification only, but have no idea of how this can be done in regression. \nDo you happen to have links to explanation or samples where Gini is used for regression, as I cannot google anything relevant."
        },
        {
          "id": 389465,
          "postDate": "2018-09-18T19:06:29.237Z",
          "content": "<p>Using the Gini index in this way seems to be common in the insurance industry, though there is no reason why it has to be specific to insurance.  Here are a couple of competitions where Gini was used:</p>\n\n<p><a href=\"https://www.kaggle.com/c/liberty-mutual-fire-peril\">https://www.kaggle.com/c/liberty-mutual-fire-peril</a></p>\n\n<p><a href=\"https://www.kaggle.com/c/liberty-mutual-group-property-inspection-prediction\">https://www.kaggle.com/c/liberty-mutual-group-property-inspection-prediction</a></p>\n\n<p>There is really no difference between using Gini for classiffication and for regression.  Gini for classification is just a Gini for regression, where the outcome can only be 0 or 1.</p>",
          "rawMarkdown": "Using the Gini index in this way seems to be common in the insurance industry, though there is no reason why it has to be specific to insurance.  Here are a couple of competitions where Gini was used:\n\nhttps://www.kaggle.com/c/liberty-mutual-fire-peril\n\nhttps://www.kaggle.com/c/liberty-mutual-group-property-inspection-prediction\n\nThere is really no difference between using Gini for classiffication and for regression.  Gini for classification is just a Gini for regression, where the outcome can only be 0 or 1.",
          "votes": 4
        },
        {
          "id": 389478,
          "postDate": "2018-09-18T19:32:47.640Z",
          "content": "<p>Thanks again, I got what you mean. I knew someting new today.</p>",
          "rawMarkdown": "Thanks again, I got what you mean. I knew someting new today."
        },
        {
          "id": 389722,
          "postDate": "2018-09-19T05:56:23.690Z",
          "content": "<p><a href=\"/dmitriyguller\">@dmitriyguller</a>, thanks for this. I never thought about using Gini as a metric for regression. </p>\n\n<p>I suppose the problem with Gini (like AUC for classification) is that you can end up with ranks for predictions instead of real values. If you're interested in prices (or probabilities in classification) then this won't help.</p>\n\n<p>In Gini and AUC competitions you're pretty sure to end up with ranks since weight averaging ranks almost always gives better results. </p>",
          "rawMarkdown": "@dmitriyguller, thanks for this. I never thought about using Gini as a metric for regression. \n\nI suppose the problem with Gini (like AUC for classification) is that you can end up with ranks for predictions instead of real values. If you're interested in prices (or probabilities in classification) then this won't help.\n\nIn Gini and AUC competitions you're pretty sure to end up with ranks since weight averaging ranks almost always gives better results. ",
          "votes": 1
        },
        {
          "id": 389885,
          "postDate": "2018-09-19T11:47:06.967Z",
          "content": "<p><a href=\"/olivier\">@olivier</a>, it's not really a big issue that you have ranks instead of predictions.  Converting ranks to predictions is a simple one-dimensional problem that in all practical situations you can do even without any models, should you need to \"put a price\" on your ranks.  In practice it's even less of an issue, because models that perform well on rank measures are usually well-calibrated by default anyway, unless you deliberate obfuscate them (such as by squaring every prediction).  </p>\n\n<p>In my experience, a model with a better rank metric score can always be made to give better calibrated predictions than a model with a worse rank score, and there is probably a mathematical way to prove that.  It's just that you're taking the calibration stuff out of the differentiation task.  In Kaggle context, having a non-rank metric means that you have to do some boring meaningless stuff like probing the leaderboard for the average level of the target in the test set, or gaming the metric in other ways.</p>",
          "rawMarkdown": "@olivier, it's not really a big issue that you have ranks instead of predictions.  Converting ranks to predictions is a simple one-dimensional problem that in all practical situations you can do even without any models, should you need to \"put a price\" on your ranks.  In practice it's even less of an issue, because models that perform well on rank measures are usually well-calibrated by default anyway, unless you deliberate obfuscate them (such as by squaring every prediction).  \n\nIn my experience, a model with a better rank metric score can always be made to give better calibrated predictions than a model with a worse rank score, and there is probably a mathematical way to prove that.  It's just that you're taking the calibration stuff out of the differentiation task.  In Kaggle context, having a non-rank metric means that you have to do some boring meaningless stuff like probing the leaderboard for the average level of the target in the test set, or gaming the metric in other ways.",
          "votes": 5
        },
        {
          "id": 396049,
          "postDate": "2018-09-29T23:46:58.393Z",
          "content": "<p><a href=\"/dmitriyguller\">@dmitriyguller</a>, could you help me get there please? \nWe apply gini index on the transactionRevenue with unique counts of user in this training data.\nThen predict using regression method with objective function <strong>normalized_gini</strong> that gini and normalized gini is defined in <a href=\"https://www.kaggle.com/oxofff/gini-scorer-cv-gridsearch\">this script</a>.\nFinally, take out the prediction revenue and do the RMSE log ( sum + 1 ) stuff.\nIt might be good.\nBecause we deal with the highly skew problem properly using gini and normalized gini.</p>\n\n<p>Do I get there???</p>\n\n<p>Appreciate your comment.</p>",
          "rawMarkdown": "@dmitriyguller, could you help me get there please? \nWe apply gini index on the transactionRevenue with unique counts of user in this training data.\nThen predict using regression method with objective function **normalized_gini** that gini and normalized gini is defined in [this script](https://www.kaggle.com/oxofff/gini-scorer-cv-gridsearch).\nFinally, take out the prediction revenue and do the RMSE log ( sum + 1 ) stuff.\nIt might be good.\nBecause we deal with the highly skew problem properly using gini and normalized gini.\n\nDo I get there???\n\nAppreciate your comment."
        }
      ]
    },
    {
      "id": 388700,
      "postDate": "2018-09-17T12:41:49.663Z",
      "content": "<p>As a quick approximation, these numbers suggest that there are less than 300 (.30 * 968 = ~290) visitors in the public test set who have at least 2 non-zero transaction revenues. Those are the users affected by the sum(log) vs. log(sum) issue, so it seems very plausible that there's some small sample size noise happening. RMSE is also sensitive to outliers - it could be that a few unusually high spend users are throwing off the numbers / encouraging the inflated predictions that result from sum(log) over log(sum).</p>",
      "rawMarkdown": "As a quick approximation, these numbers suggest that there are less than 300 (.30 * 968 = ~290) visitors in the public test set who have at least 2 non-zero transaction revenues. Those are the users affected by the sum(log) vs. log(sum) issue, so it seems very plausible that there's some small sample size noise happening. RMSE is also sensitive to outliers - it could be that a few unusually high spend users are throwing off the numbers / encouraging the inflated predictions that result from sum(log) over log(sum).",
      "votes": 10,
      "replies": [
        {
          "id": 390732,
          "postDate": "2018-09-20T18:12:57.077Z",
          "content": "<p>Hi Eddy, sorry but I don't get this...  how did you find those 290 visitors ? In the train.csv there are 11515 rows with Revenue &gt;0 and 2486 of them for users that have more than one transaction, so that's 20% of the data ? Thanks for your time !</p>",
          "rawMarkdown": "Hi Eddy, sorry but I don't get this...  how did you find those 290 visitors ? In the train.csv there are 11515 rows with Revenue &gt;0 and 2486 of them for users that have more than one transaction, so that's 20% of the data ? Thanks for your time !",
          "votes": 1
        },
        {
          "id": 390749,
          "postDate": "2018-09-20T19:07:57.367Z",
          "content": "<p>Hi Antoine, it's a very rough guess based on the figure that olivier mentions in the original post: \"968 visitors with at least 2 non zero transactionRevenue\" in the train data. Pretending that the size and patterns of test are about the same as train, since the public leaderboard is 30% of the test data (presumably it must be <em>30% of users</em>), .30 * 968 ~ 290 users. I wouldn't make anything of the exact number, I just wanted to make the point that the number of users in the public leaderboard data that have more than 1 non-zero revenue visit is likely very small.</p>",
          "rawMarkdown": "Hi Antoine, it's a very rough guess based on the figure that olivier mentions in the original post: \"968 visitors with at least 2 non zero transactionRevenue\" in the train data. Pretending that the size and patterns of test are about the same as train, since the public leaderboard is 30% of the test data (presumably it must be *30% of users*), .30 * 968 ~ 290 users. I wouldn't make anything of the exact number, I just wanted to make the point that the number of users in the public leaderboard data that have more than 1 non-zero revenue visit is likely very small.",
          "votes": 2
        },
        {
          "id": 390787,
          "postDate": "2018-09-20T20:45:28.830Z",
          "content": "<p>Ok Thanks ! I just wanted to confirm this... I'm finding a different number in my case... 2486, so 20% of users with revenue have more than 1 purchase. Which is a significant part. </p>\n\n<p>Also the fact that we are using log(sum) instead of sum(log) also applies to user that have 1 transaction but many lines with zero transaction in the file... I added the details in: <a href=\"https://www.kaggle.com/ogrellier/user-level-lightgbm-lb-1-7081#390786\">Olivier's kernel</a>... I don't get why by optimizing the wrong variable we get better results here... ? Is it just that train and test set are very different ?</p>",
          "rawMarkdown": "Ok Thanks ! I just wanted to confirm this... I'm finding a different number in my case... 2486, so 20% of users with revenue have more than 1 purchase. Which is a significant part. \n\nAlso the fact that we are using log(sum) instead of sum(log) also applies to user that have 1 transaction but many lines with zero transaction in the file... I added the details in: [Olivier's kernel][1]... I don't get why by optimizing the wrong variable we get better results here... ? Is it just that train and test set are very different ?\n\n  [1]: https://www.kaggle.com/ogrellier/user-level-lightgbm-lb-1-7081#390786"
        }
      ]
    },
    {
      "id": 388634,
      "postDate": "2018-09-17T10:19:57.170Z",
      "content": "<p>I also did some experiments. What I can say is, when I have an improvement in my local validation, LB with log(sum(target)) follows quite properly. However, sum(log(target)) does not do so even I have a significant validation improvement. I think we might have a really unreliable public LB.</p>",
      "rawMarkdown": "I also did some experiments. What I can say is, when I have an improvement in my local validation, LB with log(sum(target)) follows quite properly. However, sum(log(target)) does not do so even I have a significant validation improvement. I think we might have a really unreliable public LB.",
      "votes": 7,
      "replies": [
        {
          "id": 388639,
          "postDate": "2018-09-17T10:38:07.813Z",
          "content": "<p>Thanks fatihöztürk. On my side at user level the log of sum has a reasonably reliable CV/LB correlation (at least for now) even though CV and LB are far from each other.</p>",
          "rawMarkdown": "Thanks fatihöztürk. On my side at user level the log of sum has a reasonably reliable CV/LB correlation (at least for now) even though CV and LB are far from each other.",
          "votes": 3
        },
        {
          "id": 388641,
          "postDate": "2018-09-17T10:40:16.520Z",
          "content": "<p>yeap. completely the same. </p>",
          "rawMarkdown": "yeap. completely the same. ",
          "votes": 2
        },
        {
          "id": 388667,
          "postDate": "2018-09-17T11:39:04.563Z",
          "content": "<p><a href=\"/fatihozturk\">@fatihozturk</a>, the opposite happens at my end.  My CV is pretty consistent with LB when i use sum of log; I even tried log of sum and it also follows my CV but the gap is bigger. </p>",
          "rawMarkdown": "@fatihozturk, the opposite happens at my end.  My CV is pretty consistent with LB when i use sum of log; I even tried log of sum and it also follows my CV but the gap is bigger. ",
          "votes": 2
        },
        {
          "id": 388675,
          "postDate": "2018-09-17T11:52:11.457Z",
          "content": "<p>Hi Kain. I think we have experienced the same. Till some point, the sum of logs also followed my CV improvement constantly. But, with some new feature engineerings, LB started to fail to show the same improvement with the sum of logs. Sometimes improve but other times not. However, log of sums keeps showing improvement even LB came as 1.7X.</p>",
          "rawMarkdown": "Hi Kain. I think we have experienced the same. Till some point, the sum of logs also followed my CV improvement constantly. But, with some new feature engineerings, LB started to fail to show the same improvement with the sum of logs. Sometimes improve but other times not. However, log of sums keeps showing improvement even LB came as 1.7X.",
          "votes": 1
        },
        {
          "id": 388680,
          "postDate": "2018-09-17T12:04:24.233Z",
          "content": "<p>Yeah; the same here, for some set of features my CV and LB go different ways (both sum(log) and log(sum))!</p>",
          "rawMarkdown": " Yeah; the same here, for some set of features my CV and LB go different ways (both sum(log) and log(sum))!",
          "votes": 2
        }
      ]
    },
    {
      "id": 392698,
      "postDate": "2018-09-24T07:54:24.860Z",
      "content": "<p>I created a new kernel <a href=\"https://www.kaggle.com/ogrellier/using-classification-for-predictions\">here</a> and really the last plot shows that the competition metric is the sum of np.log1p... All other ways to aggregate is simply way off.</p>",
      "rawMarkdown": "I created a new kernel [here](https://www.kaggle.com/ogrellier/using-classification-for-predictions) and really the last plot shows that the competition metric is the sum of np.log1p... All other ways to aggregate is simply way off.",
      "votes": 3
    },
    {
      "id": 388620,
      "postDate": "2018-09-17T09:46:20.323Z",
      "content": "<p>I have also tried: np.clip(test_preds, 0, train[TARGET].max()). This also worsened my score by around 0.01. So it either means that scale of values changed in test set, or it is really sum of logs.</p>",
      "rawMarkdown": "I have also tried: np.clip(test_preds, 0, train[TARGET].max()). This also worsened my score by around 0.01. So it either means that scale of values changed in test set, or it is really sum of logs.",
      "votes": 3,
      "replies": [
        {
          "id": 388628,
          "postDate": "2018-09-17T10:05:33.230Z",
          "content": "<p>Thanks for sharing your experiment <a href=\"/divrikwicky\">@divrikwicky</a>. I believe there are revenues in test somewhat higher than in train. </p>",
          "rawMarkdown": "Thanks for sharing your experiment @divrikwicky. I believe there are revenues in test somewhat higher than in train. "
        }
      ]
    },
    {
      "id": 390700,
      "postDate": "2018-09-20T16:56:28.567Z",
      "content": "<p>What if you convert to log1p, make the predictions and convert back? like:</p>\n\n<ul>\n<li>apply log1p(revenue)</li>\n<li>make predictions</li>\n<li>convert back to original values with values = np.expm1(predictions)</li>\n<li>sum and apply log: log1p(sum(values))</li>\n</ul>",
      "rawMarkdown": "What if you convert to log1p, make the predictions and convert back? like:\n\n - apply log1p(revenue)\n - make predictions\n - convert back to original values with values = np.expm1(predictions)\n - sum and apply log: log1p(sum(values))",
      "votes": 4,
      "replies": [
        {
          "id": 392272,
          "postDate": "2018-09-23T12:38:19.680Z",
          "content": "<p>Hi <a href=\"/jsaguiar\">@jsaguiar</a>,</p>\n\n<p>I guess there is a potential problem with the approach you shared. First we will try to predict Expected values of log1p(revenues) and then we will take expm1(log1p(revenues)) and then sum them up, and then again take log1p, now, here we would be underestimating because our approach is, we our doing f(Expectations (X)), whereas we should try to do Expectations(f(X)), where f is expm1 and X is log1p(revenue). By <a href=\"https://en.wikipedia.org/wiki/Jensen%27s_inequality\">Jensen's inequality</a>, f(Expectation(X)) is less than Expectation(f(X)) when f is convex, here f being expm1, I am afraid, we would be underestimating by this approach.</p>",
          "rawMarkdown": "Hi @jsaguiar,\n\nI guess there is a potential problem with the approach you shared. First we will try to predict Expected values of log1p(revenues) and then we will take expm1(log1p(revenues)) and then sum them up, and then again take log1p, now, here we would be underestimating because our approach is, we our doing f(Expectations (X)), whereas we should try to do Expectations(f(X)), where f is expm1 and X is log1p(revenue). By [Jensen's inequality][1], f(Expectation(X)) is less than Expectation(f(X)) when f is convex, here f being expm1, I am afraid, we would be underestimating by this approach.\n\n\n  [1]: https://en.wikipedia.org/wiki/Jensen%27s_inequality",
          "votes": 3
        },
        {
          "id": 392388,
          "postDate": "2018-09-23T17:21:06.287Z",
          "content": "<p>I didn't know that, thanks for the explanation. Is there a way to know how much we are underestimating? The only other approach I can think of is making predictions on the user-level</p>",
          "rawMarkdown": "I didn't know that, thanks for the explanation. Is there a way to know how much we are underestimating? The only other approach I can think of is making predictions on the user-level",
          "votes": 1
        },
        {
          "id": 392772,
          "postDate": "2018-09-24T10:22:19.723Z",
          "content": "<p>Hi <a href=\"/jsaguiar\">@jsaguiar</a>,</p>\n\n<p>I have no idea how to measure the underestimation, I have tried one thing which gave me decent result though - I created few blocks - like users with 1 visit, user with 2 visits and so on, and then multiplied the predictions with a factor <code>f</code> , where <code>f</code> is the ratio of average original revenue divided by the average predicted revenue for that group. However, I am not at all sure if this is the correct way. :(</p>",
          "rawMarkdown": "Hi @jsaguiar,\n\nI have no idea how to measure the underestimation, I have tried one thing which gave me decent result though - I created few blocks - like users with 1 visit, user with 2 visits and so on, and then multiplied the predictions with a factor `f` , where `f` is the ratio of average original revenue divided by the average predicted revenue for that group. However, I am not at all sure if this is the correct way. :("
        }
      ]
    },
    {
      "id": 390482,
      "postDate": "2018-09-20T09:27:31.527Z",
      "content": "<p>This is doing my head in ...</p>\n\n<p>If I have a user with zero revenue log(0) screws up the metric</p>\n\n<p>Olivier could you please tell me if you think these are ignored or default to 0 or should we be using log1p</p>",
      "rawMarkdown": "This is doing my head in ...\n\nIf I have a user with zero revenue log(0) screws up the metric\n\nOlivier could you please tell me if you think these are ignored or default to 0 or should we be using log1p",
      "votes": 1,
      "replies": [
        {
          "id": 390489,
          "postDate": "2018-09-20T09:38:43.573Z",
          "content": "<p>Log1p, check <a href=\"https://www.kaggle.com/c/ga-customer-revenue-prediction/discussion/65691#387242\">Phil's answer</a></p>",
          "rawMarkdown": "Log1p, check [Phil's answer][1]\n\n\n  [1]: https://www.kaggle.com/c/ga-customer-revenue-prediction/discussion/65691#387242",
          "votes": 2
        },
        {
          "id": 390569,
          "postDate": "2018-09-20T12:49:23.677Z",
          "content": "<p>Hey <a href=\"/scirpus\">@scirpus</a> sorry I arrived here too late but I 100% support <a href=\"/alexfir\">@alexfir</a> 's answer. We should use np.log1p of the sum of revenues.</p>",
          "rawMarkdown": "Hey @scirpus sorry I arrived here too late but I 100% support @alexfir 's answer. We should use np.log1p of the sum of revenues.",
          "votes": 1
        },
        {
          "id": 390667,
          "postDate": "2018-09-20T15:58:50.983Z",
          "content": "<p>Thanks for the answer Olivier - I am much calmer now !</p>\n\n<p>Congrats on you master x3 !</p>",
          "rawMarkdown": "Thanks for the answer Olivier - I am much calmer now !\n\nCongrats on you master x3 !",
          "votes": 1
        }
      ]
    },
    {
      "id": 388574,
      "postDate": "2018-09-17T07:46:10.970Z",
      "content": "<p>For me summing predictions also worked much better:</p>\n\n<pre><code>You advanced 145 places on the leaderboard!\nYour submission scored 1.6699, which is an improvement of your previous score of 1.7811\n</code></pre>\n\n<p>I changed only calculation of submit value.</p>",
      "rawMarkdown": "For me summing predictions also worked much better:\n\n    You advanced 145 places on the leaderboard!\n    Your submission scored 1.6699, which is an improvement of your previous score of 1.7811\n\nI changed only calculation of submit value.",
      "votes": 2,
      "replies": [
        {
          "id": 388577,
          "postDate": "2018-09-17T07:56:58.120Z",
          "content": "<p><a href=\"/artgor\">@artgor</a>, be aware this may just be a public LB artifact ;-) as <a href=\"/ca3295\">@ca3295</a> (craig) said shake up may be sort of fun...</p>",
          "rawMarkdown": "@artgor, be aware this may just be a public LB artifact ;-) as @ca3295 (craig) said shake up may be sort of fun...",
          "votes": 2
        },
        {
          "id": 388585,
          "postDate": "2018-09-17T08:08:56.120Z",
          "content": "<p>Thanks for the warning :) Currently I'm trying to find a CV which I can trust.</p>",
          "rawMarkdown": "Thanks for the warning :) Currently I'm trying to find a CV which I can trust."
        },
        {
          "id": 388663,
          "postDate": "2018-09-17T11:37:08.597Z",
          "content": "<p>Same experiment as Andrew; same result:</p>\n\n<p>Your Best Entry \nYou advanced 206 places on the leaderboard!\nYour submission scored 1.6535, which is an improvement of your previous score of 1.7404. Great job!</p>",
          "rawMarkdown": "Same experiment as Andrew; same result:\n\nYour Best Entry \nYou advanced 206 places on the leaderboard!\nYour submission scored 1.6535, which is an improvement of your previous score of 1.7404. Great job!",
          "votes": 1
        }
      ]
    },
    {
      "id": 388553,
      "postDate": "2018-09-17T07:23:12.370Z",
      "content": "<p>Very interesting that your CV moves the opposite way to the leaderboard - my CV being consistent with the leaderboard was the main thing leading me to think the metric must be sum of logs.</p>\n\n<p>I suppose that in addition to the point you explain about log(sum) = sum(log) for most entries, the skew of the data towards no spend at all might make it likely that the public LB has lower spend purely because spend is very noisey.</p>\n\n<p>I'm less bullish now about the metric being wrong, but I'll keep an eye on how the discussion plays out. Thanks for the insight!</p>",
      "rawMarkdown": "Very interesting that your CV moves the opposite way to the leaderboard - my CV being consistent with the leaderboard was the main thing leading me to think the metric must be sum of logs.\n\nI suppose that in addition to the point you explain about log(sum) = sum(log) for most entries, the skew of the data towards no spend at all might make it likely that the public LB has lower spend purely because spend is very noisey.\n\nI'm less bullish now about the metric being wrong, but I'll keep an eye on how the discussion plays out. Thanks for the insight!",
      "votes": 2,
      "replies": [
        {
          "id": 388564,
          "postDate": "2018-09-17T07:35:31.463Z",
          "content": "<p>Craig I had my conclusions wrong. Sum of log is higher than log of sum so if my LB improved, the public LB should have higher spendings.</p>\n\n<p>CV RMSE is higher because <code>sum of logs</code> is higher than <code>log of sum</code> so rmse should be higher. It's not the same for you since you perform the sum as a post processing operation.  </p>\n\n<p>In the end I don't think scoring is at fault I just think it is due to the revenues in Public LB.</p>",
          "rawMarkdown": "Craig I had my conclusions wrong. Sum of log is higher than log of sum so if my LB improved, the public LB should have higher spendings.\n\nCV RMSE is higher because `sum of logs` is higher than `log of sum` so rmse should be higher. It's not the same for you since you perform the sum as a post processing operation.  \n\nIn the end I don't think scoring is at fault I just think it is due to the revenues in Public LB.",
          "votes": 2
        },
        {
          "id": 388569,
          "postDate": "2018-09-17T07:39:44.977Z",
          "content": "<p>Yes - which ever direction, noise (or systematic differences) could well be the answer. The private leaderboard shake-up is already looking like it could be brutal!</p>",
          "rawMarkdown": "Yes - which ever direction, noise (or systematic differences) could well be the answer. The private leaderboard shake-up is already looking like it could be brutal!",
          "votes": 2
        }
      ]
    },
    {
      "id": 388549,
      "postDate": "2018-09-17T07:14:04.047Z",
      "content": "<p>The requirement was </p>\n\n<blockquote>\n  <p>&gt; you must predict the natural log of their <strong>total revenue</strong></p>\n</blockquote>\n\n<p>That is, natural log of aggregated values.\nYou seem to be aggregating <strong>natural log of revenues.</strong> Did I understand it wrong? </p>",
      "rawMarkdown": "The requirement was \n\n&gt; &gt; you must predict the natural log of their **total revenue**\n\nThat is, natural log of aggregated values.\nYou seem to be aggregating **natural log of revenues.** Did I understand it wrong? ",
      "votes": 2,
      "replies": [
        {
          "id": 388568,
          "postDate": "2018-09-17T07:39:04.357Z",
          "content": "<p>Krishna Kammaje you are right our job is to predict log of sum.</p>\n\n<p>But as I said in the beginning of the post, this is part of a discussion on the scoring metric and public LB scores being better if you sum the logs. </p>\n\n<p>From this I try to understand why this is the case.</p>",
          "rawMarkdown": "Krishna Kammaje you are right our job is to predict log of sum.\n\nBut as I said in the beginning of the post, this is part of a discussion on the scoring metric and public LB scores being better if you sum the logs. \n\nFrom this I try to understand why this is the case.",
          "votes": 2
        }
      ]
    },
    {
      "id": 390318,
      "postDate": "2018-09-20T03:04:20.757Z",
      "content": "<p>ok</p>",
      "rawMarkdown": "ok"
    },
    {
      "id": 389375,
      "postDate": "2018-09-18T16:00:10.890Z",
      "content": "<p>Good job buddy</p>",
      "rawMarkdown": "Good job buddy"
    },
    {
      "id": 388654,
      "postDate": "2018-09-17T11:13:52.547Z",
      "content": "<p><code>max=77113430000</code>  for transactionRevenue looks weird, it seems to be an outlier. Am I wrong?</p>",
      "rawMarkdown": "`max=77113430000`  for transactionRevenue looks weird, it seems to be an outlier. Am I wrong?",
      "replies": [
        {
          "id": 388658,
          "postDate": "2018-09-17T11:17:45.760Z",
          "content": "<p>This is the sum of revenues, haven't checked the distribution yet. In any case after 10 I don't have enough fingers to count on ;-)</p>",
          "rawMarkdown": "This is the sum of revenues, haven't checked the distribution yet. In any case after 10 I don't have enough fingers to count on ;-)",
          "votes": 1
        }
      ]
    },
    {
      "id": 389969,
      "postDate": "2018-09-19T14:27:24.110Z",
      "content": "<p>thanks</p>",
      "rawMarkdown": "thanks"
    }
  ],
  "comments": [
    {
      "id": 389172,
      "author_name": "MrOoijer",
      "author_url": "",
      "post_date": "2018-09-18T09:37:54.230000",
      "content": "<p>My two cents on the original topic. One of my first test was with the the logp1 s of the sum customer revenue as estimates per customer. That scored 1.74. The sum of logp1's scored 1.71. Observing that the mean estimate was hogher than the mean actual on train, I multiplied the latter with mean(train)/mean(test) and that scored 1.66. The conclusio is obvious: this has nothing to do with the sum of logs versus the logs of sums; it is caused by the extremely unbalanced target variable.</p>\n\n<p>Almost all scripts here convert the target variable to logp1(y) almost without considering other options, But you have to decouple this from the conversion in the objective function.  Why not log1p(log1p(y))? Why not y** 1/8? Other competitions have shown me that (for xgboost) it can make a real difference and it is worth experimenting with.</p>\n\n<p>Any thoughts on the CV? Should we stratify on fullVisionId?</p>",
      "votes": 27,
      "replies": [
        {
          "id": 389179,
          "author_name": "olivier",
          "author_url": "",
          "post_date": "2018-09-18T09:51:38.150000",
          "content": "<p>Good to see you around <a href=\"/mrooijer\">@mrooijer</a> ! Thanks for sharing your findings with us.</p>\n\n<p>You're absolutely right on the metrics used for modeling. And I guess <code>GroupKFold</code> on <code>fullVisitorId</code> would avoid having same visitors spanning different folds.</p>",
          "votes": 4,
          "replies": []
        },
        {
          "id": 389527,
          "author_name": "Max Halford",
          "author_url": "",
          "post_date": "2018-09-18T21:23:02.503000",
          "content": "<p>Hey Olivier,</p>\n\n<p>The thing is that some of the users in the training set are also in the test set. If you do a time-based cross-validation this property holds. In my opinion going with a time-based cross-validation is closer to reality. What's more it seems to work slightly better for me, at least locally.</p>",
          "votes": 6,
          "replies": []
        },
        {
          "id": 389728,
          "author_name": "",
          "author_url": "",
          "post_date": "2018-09-19T06:15:28.090000",
          "content": "",
          "votes": 0,
          "replies": []
        },
        {
          "id": 389735,
          "author_name": "Max Halford",
          "author_url": "",
          "post_date": "2018-09-19T06:29:18.090000",
          "content": "<p>Hi Zijun, no I am not.</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 389747,
          "author_name": "",
          "author_url": "",
          "post_date": "2018-09-19T06:38:56.520000",
          "content": "",
          "votes": 0,
          "replies": []
        },
        {
          "id": 389924,
          "author_name": "Mhamed Jabri",
          "author_url": "",
          "post_date": "2018-09-19T13:19:06.283000",
          "content": "<p>Hello, could anyone explain what does * multiply by mean(train)/mean(test)* mean ? :) Thank you !</p>",
          "votes": 2,
          "replies": []
        },
        {
          "id": 390108,
          "author_name": "MrOoijer",
          "author_url": "",
          "post_date": "2018-09-19T18:14:48.717000",
          "content": "<p>mean of all log1p target values / mean of all predicted log1p observation values. It was something like 0.23/0.27</p>",
          "votes": 3,
          "replies": []
        },
        {
          "id": 390509,
          "author_name": "Misha Lisovyi",
          "author_url": "",
          "post_date": "2018-09-20T10:09:31.680000",
          "content": "<p><a href=\"/mrooijer\">@mrooijer</a>\nCould you please elaborate on what do you mean by </p>\n\n<blockquote>\n  <p>But you have to decouple this from the conversion in the objective function.</p>\n</blockquote>\n\n<p>To a beginner like me it is natural to consider only log1p(sum(revenue)) or sum(log1p(revenue)), as the first is what the organisers claim to be used in eveluation and the second is a guesstimate of what can be used in practice due to some unclear feature. What is the motivation to try something else?</p>\n\n<p>Do you make an implied assumption that the distribution of target in the submission sample is very different and some other (i.e. non-adverticed) target can empirically lead to a better submission metric?</p>\n\n<p>Or is motivation something different?</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 390615,
          "author_name": "maverick",
          "author_url": "",
          "post_date": "2018-09-20T14:15:33.723000",
          "content": "<p>Hi <a href=\"/maxhalford\">@maxhalford</a>, could you elaborate on what did you mean by time-based cross validation (I guess this not the same as sklearn's TimeSeriesSplit)</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 390631,
          "author_name": "Max Halford",
          "author_url": "",
          "post_date": "2018-09-20T14:54:00.150000",
          "content": "<p>No it's not the same as <code>TimeSeriesSplit</code> which, if I am not mistaken, uses only one row in the validation set. The idea of time-based cross-validation is simply to use the last 20% of your data as the validation set. Then you can apply this recursively so that effectively you're walking back in time.</p>\n\n<p>Talk is cheap, <a href=\"https://github.com/MaxHalford/xam/blob/master/docs/model-selection.md#ordered-cross-validation\">show me the code</a>.</p>",
          "votes": 4,
          "replies": []
        },
        {
          "id": 391043,
          "author_name": "Misha Lisovyi",
          "author_url": "",
          "post_date": "2018-09-21T07:12:44.037000",
          "content": "<p><a href=\"/mrooijer\">@mrooijer</a> actually, is i scale predictions by mean(train_target)/mean(train_oof_prediction), then my score gets worse by 0.02. The other two observations are similar</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 391232,
      "author_name": "Anthony Chiu",
      "author_url": "",
      "post_date": "2018-09-21T12:45:27.380000",
      "content": "<p>My guess:</p>\n\n<p><img src=\"https://i.imgur.com/CVcd1mJ.png\" alt=\"enter image description here\"></p>\n\n<p>I plotted the train_targets (blue line), the train_predictions (green line) in the above image. What I can see is my predictions are always lower than the actual target. Also, my model failed completely in predicting the revenue spike. </p>\n\n<p>I assume most of our predictions are similar (lower than the actual target &amp; failed to predict spike). </p>\n\n<p>Since sum of logs is larger than log of sums: log(a) + log(b) = log(ab) is always &gt;= log(a+b), where a &amp; b &gt;= 1, it is like scaling our \"under-shotting\" predictions up to fit those revenue spike. And I guess that's why most people found that sum of logs gave them better lb result. </p>",
      "votes": 19,
      "replies": []
    },
    {
      "id": 388723,
      "author_name": "Maxwell",
      "author_url": "",
      "post_date": "2018-09-17T13:45:40.563000",
      "content": "<p>At least, transaction frequency in test looks higher than that in train. <br>\nBut it does not directly mean higher spending in test data.  </p>\n\n<p><img src=\"https://cdn-ak.f.st-hatena.com/images/fotolife/g/greenwind120170/20180917/20180917230237.png\" alt=\"transaction comparison\"></p>",
      "votes": 16,
      "replies": [
        {
          "id": 389085,
          "author_name": "Misha Lisovyi",
          "author_url": "",
          "post_date": "2018-09-18T05:59:38.703000",
          "content": "<p>Indeed. What also puzzles me i that in 2017 there was only a narrow spike next to Christmas, whereas in 2016 it was a 1-2-month-wide bump</p>",
          "votes": 3,
          "replies": []
        },
        {
          "id": 389092,
          "author_name": "Maxwell",
          "author_url": "",
          "post_date": "2018-09-18T06:24:05.543000",
          "content": "<p>Yes. <br>\nI think this spike came from special event related to Christmas, IT tech, GMS itself or others. <br>\nFollowing picture is the comparison between train and test from <code>US</code> only. <br>\n( Some other countries would have the same tendency, I think. ) <br>\nSo exploring events in US around the spike time may unwrap the reason. <br>\n<img src=\"https://cdn-ak.f.st-hatena.com/images/fotolife/g/greenwind120170/20180918/20180918151936.png\" alt=\"US transactions\"></p>",
          "votes": 3,
          "replies": []
        },
        {
          "id": 389120,
          "author_name": "Craig Armstrong",
          "author_url": "",
          "post_date": "2018-09-18T07:28:04.160000",
          "content": "<p><a href=\"https://trends.google.com/trends/explore?q=Google%20home&amp;geo=US\">https://trends.google.com/trends/explore?q=Google%20home&amp;geo=US</a></p>\n\n<p>Google Home? I guess a lot of search terms spike around that time of year but it seems a likely candidate.</p>",
          "votes": 2,
          "replies": []
        },
        {
          "id": 389129,
          "author_name": "Maxwell",
          "author_url": "",
          "post_date": "2018-09-18T07:53:30.463000",
          "content": "<p>Hi <a href=\"/ca3295\">@ca3295</a></p>\n\n<p>Thank you for your insight. <br>\nWe can see the same sign in other countries. <br>\nIf Google Home was released in those countries where we can see the spikes, <br>\nyou may be right.  </p>\n\n<p><img src=\"https://cdn-ak.f.st-hatena.com/images/fotolife/g/greenwind120170/20180918/20180918164807.png\" alt=\"5 countries\"></p>",
          "votes": 4,
          "replies": []
        },
        {
          "id": 389133,
          "author_name": "Craig Armstrong",
          "author_url": "",
          "post_date": "2018-09-18T07:59:17.120000",
          "content": "<p>I'm starting to doubt it now, as I'm not sure whether we are looking only at stores that sell t-shirts and mugs etc (the Google \"swag\" mentioned in the competition description i.e. <a href=\"https://shop.googlemerchandisestore.com/\">https://shop.googlemerchandisestore.com/</a> ) or whether it includes things like Google Home (<a href=\"https://store.google.com/\">https://store.google.com/</a> )</p>\n\n<p>Also the search interest in India seems to spike in April 2018, not late December as per your charts.</p>",
          "votes": 3,
          "replies": []
        },
        {
          "id": 389528,
          "author_name": "Ptolemy",
          "author_url": "",
          "post_date": "2018-09-18T21:30:33.230000",
          "content": "<p>Nice graph. Also notice the bias (trend)!\n@Maxwell @\nLooks like spending increases over time, so its not surprising that the test set has more spenders.</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 390663,
          "author_name": "Maxwell",
          "author_url": "",
          "post_date": "2018-09-20T15:47:59.407000",
          "content": "<p>I noticed that <code>trafficSource_adContent</code> has  <code>Ad from 12/13/16</code> category.  </p>\n\n<pre><code>train.trafficSource_adContent.value_counts(dropna=False)\n\nNaN                                            892707\nGoogle Merchandise Collection                    5122\nGoogle Online Store                              1245\nDisplay Ad created 3/11/14                        967\nFull auto ad IMAGE ONLY                           822\nAd from 12/13/16                                  610\nAd from 11/3/16                                   489\nDisplay Ad created 3/11/15                        392\n{KeyWord:Google Brand Items}                      251\n{KeyWord:Google Merchandise}                      155\nAd from 11/7/16                                   123\nFirst Full Auto Template Test Ad                   87\nGoogle Merchandise                                 87  \n</code></pre>\n\n<p>Unfortunately this can not explain the boost next to Christmas day. <br>\n(The number of records is not enough. )  </p>\n\n<pre><code>train[(train.trafficSource_adContent == 'Ad from 12/13/16') &amp; (train.date &lt; dt.date(2016, 12, 30))].shape[0]\n\n185\n</code></pre>\n\n<p>But at least there might be some campaign, <br>\nand I doubt a lot of records were allocated to <code>NaN</code> category.  </p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 392005,
          "author_name": "Cihan Gunes",
          "author_url": "",
          "post_date": "2018-09-22T20:40:26.673000",
          "content": "<p>Nice work!</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 388711,
      "author_name": "Dmitriy Guller",
      "author_url": "",
      "post_date": "2018-09-17T13:29:19.813000",
      "content": "<p>When will people learn to just always use rank metrics like the Gini index?  It amazes me how so many data scientists do not appreciate how important the choice of metric is in the modeling process, and how most metrics reflect an implicit assumption about the distribution of data.  It seems like the thought process usually goes like:</p>\n\n<ol>\n<li>Let's try RMSE, that's simple.  Oh, our target values are highly skewed? </li>\n<li>Let's try logging the target values.  Oh, we have zero targets? </li>\n<li>Let's add one before logging.  Ok, that works, done!</li>\n</ol>",
      "votes": 10,
      "replies": [
        {
          "id": 389145,
          "author_name": "Alexander Firsov",
          "author_url": "",
          "post_date": "2018-09-18T08:18:21.110000",
          "content": "<p>Dmitriy, could you please explain how rank metrics can be used in this competition?</p>",
          "votes": 2,
          "replies": []
        },
        {
          "id": 389415,
          "author_name": "Dmitriy Guller",
          "author_url": "",
          "post_date": "2018-09-18T17:32:19.387000",
          "content": "<blockquote>\n  <p><strong>Alexander Firsov wrote</strong></p>\n  \n  <blockquote>\n    <p>Dmitriy, could you please explain how rank metrics can be used in this competition?</p>\n  </blockquote>\n</blockquote>\n\n<p>You use a Gini index.  You sort your clients by their predicted total spend, and you accumulate their actual spend.  Only the ordering of prediction matters, not the distance between actual and predicted.  By taking this distance out of evaluation metric, you're taking subjectivity and implicit distributional assumptions out.</p>",
          "votes": 2,
          "replies": []
        },
        {
          "id": 389461,
          "author_name": "Alexander Firsov",
          "author_url": "",
          "post_date": "2018-09-18T18:55:26.853000",
          "content": "<p>Dmitriy, thanks for reply, what you described sounds interesting. I used Gini and ROC AUC in binary classification only, but have no idea of how this can be done in regression. \nDo you happen to have links to explanation or samples where Gini is used for regression, as I cannot google anything relevant.</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 389465,
          "author_name": "Dmitriy Guller",
          "author_url": "",
          "post_date": "2018-09-18T19:06:29.237000",
          "content": "<p>Using the Gini index in this way seems to be common in the insurance industry, though there is no reason why it has to be specific to insurance.  Here are a couple of competitions where Gini was used:</p>\n\n<p><a href=\"https://www.kaggle.com/c/liberty-mutual-fire-peril\">https://www.kaggle.com/c/liberty-mutual-fire-peril</a></p>\n\n<p><a href=\"https://www.kaggle.com/c/liberty-mutual-group-property-inspection-prediction\">https://www.kaggle.com/c/liberty-mutual-group-property-inspection-prediction</a></p>\n\n<p>There is really no difference between using Gini for classiffication and for regression.  Gini for classification is just a Gini for regression, where the outcome can only be 0 or 1.</p>",
          "votes": 4,
          "replies": []
        },
        {
          "id": 389478,
          "author_name": "Alexander Firsov",
          "author_url": "",
          "post_date": "2018-09-18T19:32:47.640000",
          "content": "<p>Thanks again, I got what you mean. I knew someting new today.</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 389722,
          "author_name": "olivier",
          "author_url": "",
          "post_date": "2018-09-19T05:56:23.690000",
          "content": "<p><a href=\"/dmitriyguller\">@dmitriyguller</a>, thanks for this. I never thought about using Gini as a metric for regression. </p>\n\n<p>I suppose the problem with Gini (like AUC for classification) is that you can end up with ranks for predictions instead of real values. If you're interested in prices (or probabilities in classification) then this won't help.</p>\n\n<p>In Gini and AUC competitions you're pretty sure to end up with ranks since weight averaging ranks almost always gives better results. </p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 389885,
          "author_name": "Dmitriy Guller",
          "author_url": "",
          "post_date": "2018-09-19T11:47:06.967000",
          "content": "<p><a href=\"/olivier\">@olivier</a>, it's not really a big issue that you have ranks instead of predictions.  Converting ranks to predictions is a simple one-dimensional problem that in all practical situations you can do even without any models, should you need to \"put a price\" on your ranks.  In practice it's even less of an issue, because models that perform well on rank measures are usually well-calibrated by default anyway, unless you deliberate obfuscate them (such as by squaring every prediction).  </p>\n\n<p>In my experience, a model with a better rank metric score can always be made to give better calibrated predictions than a model with a worse rank score, and there is probably a mathematical way to prove that.  It's just that you're taking the calibration stuff out of the differentiation task.  In Kaggle context, having a non-rank metric means that you have to do some boring meaningless stuff like probing the leaderboard for the average level of the target in the test set, or gaming the metric in other ways.</p>",
          "votes": 5,
          "replies": []
        },
        {
          "id": 396049,
          "author_name": "YLT0609",
          "author_url": "",
          "post_date": "2018-09-29T23:46:58.393000",
          "content": "<p><a href=\"/dmitriyguller\">@dmitriyguller</a>, could you help me get there please? \nWe apply gini index on the transactionRevenue with unique counts of user in this training data.\nThen predict using regression method with objective function <strong>normalized_gini</strong> that gini and normalized gini is defined in <a href=\"https://www.kaggle.com/oxofff/gini-scorer-cv-gridsearch\">this script</a>.\nFinally, take out the prediction revenue and do the RMSE log ( sum + 1 ) stuff.\nIt might be good.\nBecause we deal with the highly skew problem properly using gini and normalized gini.</p>\n\n<p>Do I get there???</p>\n\n<p>Appreciate your comment.</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 388700,
      "author_name": "Joe Eddy",
      "author_url": "",
      "post_date": "2018-09-17T12:41:49.663000",
      "content": "<p>As a quick approximation, these numbers suggest that there are less than 300 (.30 * 968 = ~290) visitors in the public test set who have at least 2 non-zero transaction revenues. Those are the users affected by the sum(log) vs. log(sum) issue, so it seems very plausible that there's some small sample size noise happening. RMSE is also sensitive to outliers - it could be that a few unusually high spend users are throwing off the numbers / encouraging the inflated predictions that result from sum(log) over log(sum).</p>",
      "votes": 10,
      "replies": [
        {
          "id": 390732,
          "author_name": "Antoine",
          "author_url": "",
          "post_date": "2018-09-20T18:12:57.077000",
          "content": "<p>Hi Eddy, sorry but I don't get this...  how did you find those 290 visitors ? In the train.csv there are 11515 rows with Revenue &gt;0 and 2486 of them for users that have more than one transaction, so that's 20% of the data ? Thanks for your time !</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 390749,
          "author_name": "Joe Eddy",
          "author_url": "",
          "post_date": "2018-09-20T19:07:57.367000",
          "content": "<p>Hi Antoine, it's a very rough guess based on the figure that olivier mentions in the original post: \"968 visitors with at least 2 non zero transactionRevenue\" in the train data. Pretending that the size and patterns of test are about the same as train, since the public leaderboard is 30% of the test data (presumably it must be <em>30% of users</em>), .30 * 968 ~ 290 users. I wouldn't make anything of the exact number, I just wanted to make the point that the number of users in the public leaderboard data that have more than 1 non-zero revenue visit is likely very small.</p>",
          "votes": 2,
          "replies": []
        },
        {
          "id": 390787,
          "author_name": "Antoine",
          "author_url": "",
          "post_date": "2018-09-20T20:45:28.830000",
          "content": "<p>Ok Thanks ! I just wanted to confirm this... I'm finding a different number in my case... 2486, so 20% of users with revenue have more than 1 purchase. Which is a significant part. </p>\n\n<p>Also the fact that we are using log(sum) instead of sum(log) also applies to user that have 1 transaction but many lines with zero transaction in the file... I added the details in: <a href=\"https://www.kaggle.com/ogrellier/user-level-lightgbm-lb-1-7081#390786\">Olivier's kernel</a>... I don't get why by optimizing the wrong variable we get better results here... ? Is it just that train and test set are very different ?</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 388634,
      "author_name": "Fatih Öztürk",
      "author_url": "",
      "post_date": "2018-09-17T10:19:57.170000",
      "content": "<p>I also did some experiments. What I can say is, when I have an improvement in my local validation, LB with log(sum(target)) follows quite properly. However, sum(log(target)) does not do so even I have a significant validation improvement. I think we might have a really unreliable public LB.</p>",
      "votes": 7,
      "replies": [
        {
          "id": 388639,
          "author_name": "olivier",
          "author_url": "",
          "post_date": "2018-09-17T10:38:07.813000",
          "content": "<p>Thanks fatihöztürk. On my side at user level the log of sum has a reasonably reliable CV/LB correlation (at least for now) even though CV and LB are far from each other.</p>",
          "votes": 3,
          "replies": []
        },
        {
          "id": 388641,
          "author_name": "Fatih Öztürk",
          "author_url": "",
          "post_date": "2018-09-17T10:40:16.520000",
          "content": "<p>yeap. completely the same. </p>",
          "votes": 2,
          "replies": []
        },
        {
          "id": 388667,
          "author_name": "Kain",
          "author_url": "",
          "post_date": "2018-09-17T11:39:04.563000",
          "content": "<p><a href=\"/fatihozturk\">@fatihozturk</a>, the opposite happens at my end.  My CV is pretty consistent with LB when i use sum of log; I even tried log of sum and it also follows my CV but the gap is bigger. </p>",
          "votes": 2,
          "replies": []
        },
        {
          "id": 388675,
          "author_name": "Fatih Öztürk",
          "author_url": "",
          "post_date": "2018-09-17T11:52:11.457000",
          "content": "<p>Hi Kain. I think we have experienced the same. Till some point, the sum of logs also followed my CV improvement constantly. But, with some new feature engineerings, LB started to fail to show the same improvement with the sum of logs. Sometimes improve but other times not. However, log of sums keeps showing improvement even LB came as 1.7X.</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 388680,
          "author_name": "Kain",
          "author_url": "",
          "post_date": "2018-09-17T12:04:24.233000",
          "content": "<p>Yeah; the same here, for some set of features my CV and LB go different ways (both sum(log) and log(sum))!</p>",
          "votes": 2,
          "replies": []
        }
      ]
    },
    {
      "id": 392698,
      "author_name": "olivier",
      "author_url": "",
      "post_date": "2018-09-24T07:54:24.860000",
      "content": "<p>I created a new kernel <a href=\"https://www.kaggle.com/ogrellier/using-classification-for-predictions\">here</a> and really the last plot shows that the competition metric is the sum of np.log1p... All other ways to aggregate is simply way off.</p>",
      "votes": 3,
      "replies": []
    },
    {
      "id": 388620,
      "author_name": "Ahmet Erdem",
      "author_url": "",
      "post_date": "2018-09-17T09:46:20.323000",
      "content": "<p>I have also tried: np.clip(test_preds, 0, train[TARGET].max()). This also worsened my score by around 0.01. So it either means that scale of values changed in test set, or it is really sum of logs.</p>",
      "votes": 3,
      "replies": [
        {
          "id": 388628,
          "author_name": "olivier",
          "author_url": "",
          "post_date": "2018-09-17T10:05:33.230000",
          "content": "<p>Thanks for sharing your experiment <a href=\"/divrikwicky\">@divrikwicky</a>. I believe there are revenues in test somewhat higher than in train. </p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 390700,
      "author_name": "Aguiar",
      "author_url": "",
      "post_date": "2018-09-20T16:56:28.567000",
      "content": "<p>What if you convert to log1p, make the predictions and convert back? like:</p>\n\n<ul>\n<li>apply log1p(revenue)</li>\n<li>make predictions</li>\n<li>convert back to original values with values = np.expm1(predictions)</li>\n<li>sum and apply log: log1p(sum(values))</li>\n</ul>",
      "votes": 4,
      "replies": [
        {
          "id": 392272,
          "author_name": "Indranil Bhattacharya",
          "author_url": "",
          "post_date": "2018-09-23T12:38:19.680000",
          "content": "<p>Hi <a href=\"/jsaguiar\">@jsaguiar</a>,</p>\n\n<p>I guess there is a potential problem with the approach you shared. First we will try to predict Expected values of log1p(revenues) and then we will take expm1(log1p(revenues)) and then sum them up, and then again take log1p, now, here we would be underestimating because our approach is, we our doing f(Expectations (X)), whereas we should try to do Expectations(f(X)), where f is expm1 and X is log1p(revenue). By <a href=\"https://en.wikipedia.org/wiki/Jensen%27s_inequality\">Jensen's inequality</a>, f(Expectation(X)) is less than Expectation(f(X)) when f is convex, here f being expm1, I am afraid, we would be underestimating by this approach.</p>",
          "votes": 3,
          "replies": []
        },
        {
          "id": 392388,
          "author_name": "Aguiar",
          "author_url": "",
          "post_date": "2018-09-23T17:21:06.287000",
          "content": "<p>I didn't know that, thanks for the explanation. Is there a way to know how much we are underestimating? The only other approach I can think of is making predictions on the user-level</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 392772,
          "author_name": "Indranil Bhattacharya",
          "author_url": "",
          "post_date": "2018-09-24T10:22:19.723000",
          "content": "<p>Hi <a href=\"/jsaguiar\">@jsaguiar</a>,</p>\n\n<p>I have no idea how to measure the underestimation, I have tried one thing which gave me decent result though - I created few blocks - like users with 1 visit, user with 2 visits and so on, and then multiplied the predictions with a factor <code>f</code> , where <code>f</code> is the ratio of average original revenue divided by the average predicted revenue for that group. However, I am not at all sure if this is the correct way. :(</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 390482,
      "author_name": "Scirpus",
      "author_url": "",
      "post_date": "2018-09-20T09:27:31.527000",
      "content": "<p>This is doing my head in ...</p>\n\n<p>If I have a user with zero revenue log(0) screws up the metric</p>\n\n<p>Olivier could you please tell me if you think these are ignored or default to 0 or should we be using log1p</p>",
      "votes": 1,
      "replies": [
        {
          "id": 390489,
          "author_name": "Alexander Firsov",
          "author_url": "",
          "post_date": "2018-09-20T09:38:43.573000",
          "content": "<p>Log1p, check <a href=\"https://www.kaggle.com/c/ga-customer-revenue-prediction/discussion/65691#387242\">Phil's answer</a></p>",
          "votes": 2,
          "replies": []
        },
        {
          "id": 390569,
          "author_name": "olivier",
          "author_url": "",
          "post_date": "2018-09-20T12:49:23.677000",
          "content": "<p>Hey <a href=\"/scirpus\">@scirpus</a> sorry I arrived here too late but I 100% support <a href=\"/alexfir\">@alexfir</a> 's answer. We should use np.log1p of the sum of revenues.</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 390667,
          "author_name": "Scirpus",
          "author_url": "",
          "post_date": "2018-09-20T15:58:50.983000",
          "content": "<p>Thanks for the answer Olivier - I am much calmer now !</p>\n\n<p>Congrats on you master x3 !</p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 388574,
      "author_name": "Andrey Lukyanenko",
      "author_url": "",
      "post_date": "2018-09-17T07:46:10.970000",
      "content": "<p>For me summing predictions also worked much better:</p>\n\n<pre><code>You advanced 145 places on the leaderboard!\nYour submission scored 1.6699, which is an improvement of your previous score of 1.7811\n</code></pre>\n\n<p>I changed only calculation of submit value.</p>",
      "votes": 2,
      "replies": [
        {
          "id": 388577,
          "author_name": "olivier",
          "author_url": "",
          "post_date": "2018-09-17T07:56:58.120000",
          "content": "<p><a href=\"/artgor\">@artgor</a>, be aware this may just be a public LB artifact ;-) as <a href=\"/ca3295\">@ca3295</a> (craig) said shake up may be sort of fun...</p>",
          "votes": 2,
          "replies": []
        },
        {
          "id": 388585,
          "author_name": "Andrey Lukyanenko",
          "author_url": "",
          "post_date": "2018-09-17T08:08:56.120000",
          "content": "<p>Thanks for the warning :) Currently I'm trying to find a CV which I can trust.</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 388663,
          "author_name": "GolemMaking",
          "author_url": "",
          "post_date": "2018-09-17T11:37:08.597000",
          "content": "<p>Same experiment as Andrew; same result:</p>\n\n<p>Your Best Entry \nYou advanced 206 places on the leaderboard!\nYour submission scored 1.6535, which is an improvement of your previous score of 1.7404. Great job!</p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 388553,
      "author_name": "Craig Armstrong",
      "author_url": "",
      "post_date": "2018-09-17T07:23:12.370000",
      "content": "<p>Very interesting that your CV moves the opposite way to the leaderboard - my CV being consistent with the leaderboard was the main thing leading me to think the metric must be sum of logs.</p>\n\n<p>I suppose that in addition to the point you explain about log(sum) = sum(log) for most entries, the skew of the data towards no spend at all might make it likely that the public LB has lower spend purely because spend is very noisey.</p>\n\n<p>I'm less bullish now about the metric being wrong, but I'll keep an eye on how the discussion plays out. Thanks for the insight!</p>",
      "votes": 2,
      "replies": [
        {
          "id": 388564,
          "author_name": "olivier",
          "author_url": "",
          "post_date": "2018-09-17T07:35:31.463000",
          "content": "<p>Craig I had my conclusions wrong. Sum of log is higher than log of sum so if my LB improved, the public LB should have higher spendings.</p>\n\n<p>CV RMSE is higher because <code>sum of logs</code> is higher than <code>log of sum</code> so rmse should be higher. It's not the same for you since you perform the sum as a post processing operation.  </p>\n\n<p>In the end I don't think scoring is at fault I just think it is due to the revenues in Public LB.</p>",
          "votes": 2,
          "replies": []
        },
        {
          "id": 388569,
          "author_name": "Craig Armstrong",
          "author_url": "",
          "post_date": "2018-09-17T07:39:44.977000",
          "content": "<p>Yes - which ever direction, noise (or systematic differences) could well be the answer. The private leaderboard shake-up is already looking like it could be brutal!</p>",
          "votes": 2,
          "replies": []
        }
      ]
    },
    {
      "id": 388549,
      "author_name": "Krishna Kishor Kammaje",
      "author_url": "",
      "post_date": "2018-09-17T07:14:04.047000",
      "content": "<p>The requirement was </p>\n\n<blockquote>\n  <p>&gt; you must predict the natural log of their <strong>total revenue</strong></p>\n</blockquote>\n\n<p>That is, natural log of aggregated values.\nYou seem to be aggregating <strong>natural log of revenues.</strong> Did I understand it wrong? </p>",
      "votes": 2,
      "replies": [
        {
          "id": 388568,
          "author_name": "olivier",
          "author_url": "",
          "post_date": "2018-09-17T07:39:04.357000",
          "content": "<p>Krishna Kammaje you are right our job is to predict log of sum.</p>\n\n<p>But as I said in the beginning of the post, this is part of a discussion on the scoring metric and public LB scores being better if you sum the logs. </p>\n\n<p>From this I try to understand why this is the case.</p>",
          "votes": 2,
          "replies": []
        }
      ]
    },
    {
      "id": 390318,
      "author_name": "JasonLZQ",
      "author_url": "",
      "post_date": "2018-09-20T03:04:20.757000",
      "content": "<p>ok</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 389375,
      "author_name": "Sokratis Kouvaras",
      "author_url": "",
      "post_date": "2018-09-18T16:00:10.890000",
      "content": "<p>Good job buddy</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 388654,
      "author_name": "Max Lukyanenko",
      "author_url": "",
      "post_date": "2018-09-17T11:13:52.547000",
      "content": "<p><code>max=77113430000</code>  for transactionRevenue looks weird, it seems to be an outlier. Am I wrong?</p>",
      "votes": 0,
      "replies": [
        {
          "id": 388658,
          "author_name": "olivier",
          "author_url": "",
          "post_date": "2018-09-17T11:17:45.760000",
          "content": "<p>This is the sum of revenues, haven't checked the distribution yet. In any case after 10 I don't have enough fingers to count on ;-)</p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 389969,
      "author_name": "Shadowfax",
      "author_url": "",
      "post_date": "2018-09-19T14:27:24.110000",
      "content": "<p>thanks</p>",
      "votes": 0,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "388546": "There has been some discussion on the fact the competition metric/scoring was unclear...\n\nCraig Armstrong and yilisg requested I do an experiment with my [public kernel](https://www.kaggle.com/ogrellier/user-level-lightgbm) and to my surprise the LB score improved substantially to around 1.70 when CV increased to above 2 !\n\nThe kernel trains a model at user level and not session level. The experiment was to aggregate `np.log1p(transcationRevenue)` instead of `transactionRevenue`.\n\nMost of the visitors (620675 in train) have only one session and  summing up before of after going to log does not change anything for them.\n\nFor the others, if they only have 1 session with a non zero revenue it does not change anything either. \n\nThere are 93492 Visitors with more than 1 session in train. On these, 86616 have 0 revenue and 5909 only have 1 session with a non zero revenue !\n\nThis means you only have 968 visitors with at least 2 non zero transactionRevenue. That's not a lot but these guys have very high spendings : `min=10000, max=77113430000, median=57195000`\n\nThe question then becomes : does public LB have higher spending ? \n\n```\nnp.log1p(1 + 20000) = 9.90..  and  2 * np.log1p(1 + 10000) = 18.42..\n```\n\n",
    "389172": "My two cents on the original topic. One of my first test was with the the logp1 s of the sum customer revenue as estimates per customer. That scored 1.74. The sum of logp1's scored 1.71. Observing that the mean estimate was hogher than the mean actual on train, I multiplied the latter with mean(train)/mean(test) and that scored 1.66. The conclusio is obvious: this has nothing to do with the sum of logs versus the logs of sums; it is caused by the extremely unbalanced target variable.\n\nAlmost all scripts here convert the target variable to logp1(y) almost without considering other options, But you have to decouple this from the conversion in the objective function.  Why not log1p(log1p(y))? Why not y** 1/8? Other competitions have shown me that (for xgboost) it can make a real difference and it is worth experimenting with.\n\nAny thoughts on the CV? Should we stratify on fullVisionId?",
    "391232": "My guess:\n\n![enter image description here][1]\n\nI plotted the train_targets (blue line), the train_predictions (green line) in the above image. What I can see is my predictions are always lower than the actual target. Also, my model failed completely in predicting the revenue spike. \n\nI assume most of our predictions are similar (lower than the actual target &amp; failed to predict spike). \n\nSince sum of logs is larger than log of sums: log(a) + log(b) = log(ab) is always &gt;= log(a+b), where a &amp; b &gt;= 1, it is like scaling our \"under-shotting\" predictions up to fit those revenue spike. And I guess that's why most people found that sum of logs gave them better lb result. \n\n  [1]: https://i.imgur.com/CVcd1mJ.png",
    "388723": "At least, transaction frequency in test looks higher than that in train.  \nBut it does not directly mean higher spending in test data.  \n  \n![transaction comparison][1]\n\n\n  [1]: https://cdn-ak.f.st-hatena.com/images/fotolife/g/greenwind120170/20180917/20180917230237.png",
    "388711": "When will people learn to just always use rank metrics like the Gini index?  It amazes me how so many data scientists do not appreciate how important the choice of metric is in the modeling process, and how most metrics reflect an implicit assumption about the distribution of data.  It seems like the thought process usually goes like:\n\n 1.  Let's try RMSE, that's simple.  Oh, our target values are highly skewed? \n 2.  Let's try logging the target values.  Oh, we have zero targets? \n 3. Let's add one before logging.  Ok, that works, done!",
    "388700": "As a quick approximation, these numbers suggest that there are less than 300 (.30 * 968 = ~290) visitors in the public test set who have at least 2 non-zero transaction revenues. Those are the users affected by the sum(log) vs. log(sum) issue, so it seems very plausible that there's some small sample size noise happening. RMSE is also sensitive to outliers - it could be that a few unusually high spend users are throwing off the numbers / encouraging the inflated predictions that result from sum(log) over log(sum).",
    "388634": "I also did some experiments. What I can say is, when I have an improvement in my local validation, LB with log(sum(target)) follows quite properly. However, sum(log(target)) does not do so even I have a significant validation improvement. I think we might have a really unreliable public LB.",
    "392698": "I created a new kernel [here](https://www.kaggle.com/ogrellier/using-classification-for-predictions) and really the last plot shows that the competition metric is the sum of np.log1p... All other ways to aggregate is simply way off.",
    "388620": "I have also tried: np.clip(test_preds, 0, train[TARGET].max()). This also worsened my score by around 0.01. So it either means that scale of values changed in test set, or it is really sum of logs.",
    "390700": "What if you convert to log1p, make the predictions and convert back? like:\n\n - apply log1p(revenue)\n - make predictions\n - convert back to original values with values = np.expm1(predictions)\n - sum and apply log: log1p(sum(values))",
    "390482": "This is doing my head in ...\n\nIf I have a user with zero revenue log(0) screws up the metric\n\nOlivier could you please tell me if you think these are ignored or default to 0 or should we be using log1p",
    "388574": "For me summing predictions also worked much better:\n\n    You advanced 145 places on the leaderboard!\n    Your submission scored 1.6699, which is an improvement of your previous score of 1.7811\n\nI changed only calculation of submit value.",
    "388553": "Very interesting that your CV moves the opposite way to the leaderboard - my CV being consistent with the leaderboard was the main thing leading me to think the metric must be sum of logs.\n\nI suppose that in addition to the point you explain about log(sum) = sum(log) for most entries, the skew of the data towards no spend at all might make it likely that the public LB has lower spend purely because spend is very noisey.\n\nI'm less bullish now about the metric being wrong, but I'll keep an eye on how the discussion plays out. Thanks for the insight!",
    "388549": "The requirement was \n\n&gt; &gt; you must predict the natural log of their **total revenue**\n\nThat is, natural log of aggregated values.\nYou seem to be aggregating **natural log of revenues.** Did I understand it wrong? ",
    "390318": "ok",
    "389375": "Good job buddy",
    "388654": "`max=77113430000`  for transactionRevenue looks weird, it seems to be an outlier. Am I wrong?",
    "389969": "thanks"
  }
}