{
  "id": 72997,
  "title": "Super duper mega approach",
  "url": "/competitions/ga-customer-revenue-prediction/discussion/72997",
  "author_name": "",
  "post_date": "2018-11-28T23:59:29.848718400Z",
  "votes": -9,
  "comment_count": 13,
  "views": 0,
  "content": "<p>Hello guys,</p>\n\n<p>I'm back to my notebook after short rest.\nIn next 2-3 days I will publish here some tricks I used in my solution.</p>\n\n<p>Please comment, add your thoughts and improvements. Maybe you used some interesting tricks.\nUp to date I have about 0.26 RMSE (0.6 MSE) at CV.</p>",
  "messages": [
    {
      "id": "429488",
      "postDate": "11/28/2018 23:59:29",
      "content": "<p>Hello guys,</p>\n\n<p>I'm back to my notebook after short rest.\nIn next 2-3 days I will publish here some tricks I used in my solution.</p>\n\n<p>Please comment, add your thoughts and improvements. Maybe you used some interesting tricks.\nUp to date I have about 0.26 RMSE (0.6 MSE) at CV.</p>",
      "rawMarkdown": "Hello guys,\n\nI'm back to my notebook after short rest.\nIn next 2-3 days I will publish here some tricks I used in my solution.\n\nPlease comment, add your thoughts and improvements. Maybe you used some interesting tricks.\nUp to date I have about 0.26 RMSE (0.6 MSE) at CV.",
      "votes": null
    },
    {
      "id": "429491",
      "postDate": "11/29/2018 00:04:47",
      "content": "<p>Use regression, no classification + regression</p>\n\n<p>Simple example which shows influence of classification error on MSE. Even assuming regression error(after applying classification) is zero, we have terrible results against dummy mean/median value, which we can get from original dataset. So the way through classification is wrong. Even if you get 99+% accuracy, you should remember, less than 1% of people in original dataset returned after so long gap. <a href=\"http://tinypic.com/r/qobmg3/9\">http://tinypic.com/r/qobmg3/9</a>\n<img src=\"http://oi64.tinypic.com/qobmg3.jpg\" alt=\"clf_vs_reg\"></p>",
      "rawMarkdown": "Use regression, no classification + regression\n\nSimple example which shows influence of classification error on MSE. Even assuming regression error(after applying classification) is zero, we have terrible results against dummy mean/median value, which we can get from original dataset. So the way through classification is wrong. Even if you get 99+% accuracy, you should remember, less than 1% of people in original dataset returned after so long gap. http://tinypic.com/r/qobmg3/9\n![clf_vs_reg][1]\n\n\n  [1]: http://oi64.tinypic.com/qobmg3.jpg",
      "votes": null
    },
    {
      "id": "429613",
      "postDate": "11/29/2018 04:18:44",
      "content": "<p>Hi Egor, \nThanks for sharing. \nplease share what is your rmse when predicting all zeros?</p>\n\n<p>Thanks </p>",
      "rawMarkdown": "Hi Egor, \nThanks for sharing. \nplease share what is your rmse when predicting all zeros?\n\nThanks",
      "votes": null
    },
    {
      "id": "429921",
      "postDate": "11/29/2018 14:54:21",
      "content": "<p>Hi @AmirH,</p>\n\n<p>RMSE for all-zero is 0.29, BUT it depends of the CV way.\nHere is my approach for data augmentation and feature engineering. I shifted my window of observation, gap, and prediction periods across whole time I provided. CV was carried out on the last 20%(last 20% of rows after augmentation and shuffling rows).\n<img src=\"http://oi65.tinypic.com/k54a6g.jpg\" alt=\"clf_vs_reg\"></p>",
      "rawMarkdown": "Hi @AmirH,\n\nRMSE for all-zero is 0.29, BUT it depends of the CV way.\nHere is my approach for data augmentation and feature engineering. I shifted my window of observation, gap, and prediction periods across whole time I provided. CV was carried out on the last 20%(last 20% of rows after augmentation and shuffling rows).\n![clf_vs_reg][1]\n\n  [1]: http://oi65.tinypic.com/k54a6g.jpg",
      "votes": null
    },
    {
      "id": "430034",
      "postDate": "11/29/2018 17:48:18",
      "content": "<p>Hi Egor,  the reported CV error comes from a period of time that overlaps the period you used to train your data? </p>\n\n<p>I did this and the results were great, but when considered a totally disjoint validation period (as we have to do for the competition) the results got very bad again.</p>\n\n<p>I think the most reallistic scenario  is to validate in a data set within a time period that was not used on training, otherwise, i believe the error might be underestimated.</p>",
      "rawMarkdown": "Hi Egor,  the reported CV error comes from a period of time that overlaps the period you used to train your data? \n\nI did this and the results were great, but when considered a totally disjoint validation period (as we have to do for the competition) the results got very bad again.\n\nI think the most reallistic scenario  is to validate in a data set within a time period that was not used on training, otherwise, i believe the error might be underestimated.",
      "votes": null
    },
    {
      "id": "430352",
      "postDate": "11/30/2018 08:53:56",
      "content": "<p>Hi Luis,\nYes I trained model on the random 80% of rows and validated it on the other 20%.</p>\n\n<p>Yes, I do not exclude that the error may be underestimated</p>",
      "rawMarkdown": "Hi Luis,\nYes I trained model on the random 80% of rows and validated it on the other 20%.\n\nYes, I do not exclude that the error may be underestimated",
      "votes": null
    },
    {
      "id": "430387",
      "postDate": "11/30/2018 09:57:45",
      "content": "<p>Hey Egor, what exactly do you mean by duplicates?</p>",
      "rawMarkdown": "Hey Egor, what exactly do you mean by duplicates?",
      "votes": null
    },
    {
      "id": "430398",
      "postDate": "11/30/2018 10:23:26",
      "content": "<p>Egor - thanks for taking the time to share your thoughts on this. Unfortunately doing this in the last day or two of a competition is not a popular choice. The reason for the large number of downvotes is likely because you are sharing this in so late in the competition. If it is effective then someone who has so far made no effort could score highly by implementing your approach. If your approach turns out to be unsound then people will waste valuable time on the last day working on it. This is why for kernels there is a clear notice from Kaggle saying please do not release new kernels that score well during the last week. For future reference, I suspect that if you had put this post out the day after the competition close, or when the results are announced then you would not have got downvotes and you may have got some upvotes.</p>",
      "rawMarkdown": "Egor - thanks for taking the time to share your thoughts on this. Unfortunately doing this in the last day or two of a competition is not a popular choice. The reason for the large number of downvotes is likely because you are sharing this in so late in the competition. If it is effective then someone who has so far made no effort could score highly by implementing your approach. If your approach turns out to be unsound then people will waste valuable time on the last day working on it. This is why for kernels there is a clear notice from Kaggle saying please do not release new kernels that score well during the last week. For future reference, I suspect that if you had put this post out the day after the competition close, or when the results are announced then you would not have got downvotes and you may have got some upvotes.",
      "votes": null
    },
    {
      "id": "430522",
      "postDate": "11/30/2018 14:16:53",
      "content": "<p>Hello Antonio,</p>\n\n<p>Duplicates are users with the same behavior. They have same hits, revenue, browser, OS, etc.</p>",
      "rawMarkdown": "Hello Antonio,\n\nDuplicates are users with the same behavior. They have same hits, revenue, browser, OS, etc.",
      "votes": null
    },
    {
      "id": "430600",
      "postDate": "11/30/2018 17:26:48",
      "content": "<p>Ok, thanks. But just out of curiosity: at which stage do you identify them? The original data has session granularity, so are you saying that there are some users for whom all original session data are identical (so there are identical sessions started at the same time, but belonging to different users)?</p>\n\n<p>Or did you find the duplicates when you aggregated the data? In this case it is not necessary that the original data are identical for some users, but it just turned out that aggregated data are identical for some users with different fullVisitorIds (which seems a bit unlikely, to be honest)?</p>\n\n<p>Can you please clarify this?</p>",
      "rawMarkdown": "Ok, thanks. But just out of curiosity: at which stage do you identify them? The original data has session granularity, so are you saying that there are some users for whom all original session data are identical (so there are identical sessions started at the same time, but belonging to different users)?\n\nOr did you find the duplicates when you aggregated the data? In this case it is not necessary that the original data are identical for some users, but it just turned out that aggregated data are identical for some users with different fullVisitorIds (which seems a bit unlikely, to be honest)?\n\nCan you please clarify this?",
      "votes": null
    },
    {
      "id": "430671",
      "postDate": "11/30/2018 20:33:39",
      "content": "<p>Yes, I mean duplicates after aggregating data.</p>\n\n<p>E.g. one visit with no hints(just go direct to GStore homepage and close tab after 1-3 sec -- case of wrong entry). Or maybe bot entry(1 sec visit, one page view). For me this is just one pattern. So I don't need a lot of data about this patterns.</p>\n\n<p>Moreover, you get a lot of duplicates while shifting window. E.g. activity within one day in the end of observation period leads to ~165 identical rows(yes, I drop userID after aggregating, for me user=pattern of user actions).</p>\n\n<p>Note, that the features are based on user behavior and some users may behave the same way. Thats why we can get duplicates. Especially from visits with zero revenue.</p>\n\n<p>If you have any more questions, I'm ready to clarify.</p>",
      "rawMarkdown": "Yes, I mean duplicates after aggregating data.\n\nE.g. one visit with no hints(just go direct to GStore homepage and close tab after 1-3 sec -- case of wrong entry). Or maybe bot entry(1 sec visit, one page view). For me this is just one pattern. So I don't need a lot of data about this patterns.\n\nMoreover, you get a lot of duplicates while shifting window. E.g. activity within one day in the end of observation period leads to ~165 identical rows(yes, I drop userID after aggregating, for me user=pattern of user actions).\n\nNote, that the features are based on user behavior and some users may behave the same way. Thats why we can get duplicates. Especially from visits with zero revenue.\n\nIf you have any more questions, I'm ready to clarify.",
      "votes": null
    },
    {
      "id": "430963",
      "postDate": "12/01/2018 10:40:53",
      "content": "<p>OK, now I see what you mean. Yes, you have a point when it comes to analysing groups of users (clustering) and to dropping data which is strictly identical, but of course depending on how many features you have,  the same observations will or will not be strictly identical (even if only one feature is different for two observations, this is information that potentially can be valuable). Also the way you aggregate your data matters, even in your example there are ways to aggregate the data for the user with one activity only so that you get different (and different in a meaningful way) aggregated figures for each rolling window. </p>\n\n<p>But of course most of this is just heuristics, and it is hard to know in advance. what element of strategy is really going to give the model a competitive edge. </p>",
      "rawMarkdown": "OK, now I see what you mean. Yes, you have a point when it comes to analysing groups of users (clustering) and to dropping data which is strictly identical, but of course depending on how many features you have,  the same observations will or will not be strictly identical (even if only one feature is different for two observations, this is information that potentially can be valuable). Also the way you aggregate your data matters, even in your example there are ways to aggregate the data for the user with one activity only so that you get different (and different in a meaningful way) aggregated figures for each rolling window. \n\nBut of course most of this is just heuristics, and it is hard to know in advance. what element of strategy is really going to give the model a competitive edge.",
      "votes": null
    },
    {
      "id": "431227",
      "postDate": "12/01/2018 22:03:29",
      "content": "<p>Ok</p>",
      "rawMarkdown": "Ok",
      "votes": null
    },
    {
      "id": "435120",
      "postDate": "12/07/2018 14:31:31",
      "content": "<p>I like some of the things you did and the way that you presented them. I agree with <a href=\"/luisduque440\">@luisduque440</a> that the overlapping time periods could lead to overfitting. Might have been interesting to check if you could have got similar results by doing everything you listed above but not shuffling and instead using a time-series validation approach where you only look forward in time at later time slots. I guess it is too late to do anything about it now though. Lets see where we end up in 2 months time.</p>",
      "rawMarkdown": "I like some of the things you did and the way that you presented them. I agree with @luisduque440 that the overlapping time periods could lead to overfitting. Might have been interesting to check if you could have got similar results by doing everything you listed above but not shuffling and instead using a time-series validation approach where you only look forward in time at later time slots. I guess it is too late to do anything about it now though. Lets see where we end up in 2 months time.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 429491,
      "author_name": "eguliy",
      "author_url": "",
      "post_date": "11/29/2018 00:04:47",
      "content": "<p>Use regression, no classification + regression</p>\n\n<p>Simple example which shows influence of classification error on MSE. Even assuming regression error(after applying classification) is zero, we have terrible results against dummy mean/median value, which we can get from original dataset. So the way through classification is wrong. Even if you get 99+% accuracy, you should remember, less than 1% of people in original dataset returned after so long gap. <a href=\"http://tinypic.com/r/qobmg3/9\">http://tinypic.com/r/qobmg3/9</a>\n<img src=\"http://oi64.tinypic.com/qobmg3.jpg\" alt=\"clf_vs_reg\"></p>",
      "votes": null,
      "replies": [
        {
          "id": 429613,
          "author_name": "tpthegreat",
          "author_url": "",
          "post_date": "11/29/2018 04:18:44",
          "content": "<p>Hi Egor, \nThanks for sharing. \nplease share what is your rmse when predicting all zeros?</p>\n\n<p>Thanks </p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 429921,
          "author_name": "eguliy",
          "author_url": "",
          "post_date": "11/29/2018 14:54:21",
          "content": "<p>Hi @AmirH,</p>\n\n<p>RMSE for all-zero is 0.29, BUT it depends of the CV way.\nHere is my approach for data augmentation and feature engineering. I shifted my window of observation, gap, and prediction periods across whole time I provided. CV was carried out on the last 20%(last 20% of rows after augmentation and shuffling rows).\n<img src=\"http://oi65.tinypic.com/k54a6g.jpg\" alt=\"clf_vs_reg\"></p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 430034,
          "author_name": "luisduque440",
          "author_url": "",
          "post_date": "11/29/2018 17:48:18",
          "content": "<p>Hi Egor,  the reported CV error comes from a period of time that overlaps the period you used to train your data? </p>\n\n<p>I did this and the results were great, but when considered a totally disjoint validation period (as we have to do for the competition) the results got very bad again.</p>\n\n<p>I think the most reallistic scenario  is to validate in a data set within a time period that was not used on training, otherwise, i believe the error might be underestimated.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 430352,
          "author_name": "eguliy",
          "author_url": "",
          "post_date": "11/30/2018 08:53:56",
          "content": "<p>Hi Luis,\nYes I trained model on the random 80% of rows and validated it on the other 20%.</p>\n\n<p>Yes, I do not exclude that the error may be underestimated</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 430387,
          "author_name": "antoniorossi",
          "author_url": "",
          "post_date": "11/30/2018 09:57:45",
          "content": "<p>Hey Egor, what exactly do you mean by duplicates?</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 430522,
          "author_name": "eguliy",
          "author_url": "",
          "post_date": "11/30/2018 14:16:53",
          "content": "<p>Hello Antonio,</p>\n\n<p>Duplicates are users with the same behavior. They have same hits, revenue, browser, OS, etc.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 430600,
          "author_name": "antoniorossi",
          "author_url": "",
          "post_date": "11/30/2018 17:26:48",
          "content": "<p>Ok, thanks. But just out of curiosity: at which stage do you identify them? The original data has session granularity, so are you saying that there are some users for whom all original session data are identical (so there are identical sessions started at the same time, but belonging to different users)?</p>\n\n<p>Or did you find the duplicates when you aggregated the data? In this case it is not necessary that the original data are identical for some users, but it just turned out that aggregated data are identical for some users with different fullVisitorIds (which seems a bit unlikely, to be honest)?</p>\n\n<p>Can you please clarify this?</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 430671,
          "author_name": "eguliy",
          "author_url": "",
          "post_date": "11/30/2018 20:33:39",
          "content": "<p>Yes, I mean duplicates after aggregating data.</p>\n\n<p>E.g. one visit with no hints(just go direct to GStore homepage and close tab after 1-3 sec -- case of wrong entry). Or maybe bot entry(1 sec visit, one page view). For me this is just one pattern. So I don't need a lot of data about this patterns.</p>\n\n<p>Moreover, you get a lot of duplicates while shifting window. E.g. activity within one day in the end of observation period leads to ~165 identical rows(yes, I drop userID after aggregating, for me user=pattern of user actions).</p>\n\n<p>Note, that the features are based on user behavior and some users may behave the same way. Thats why we can get duplicates. Especially from visits with zero revenue.</p>\n\n<p>If you have any more questions, I'm ready to clarify.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 430963,
          "author_name": "antoniorossi",
          "author_url": "",
          "post_date": "12/01/2018 10:40:53",
          "content": "<p>OK, now I see what you mean. Yes, you have a point when it comes to analysing groups of users (clustering) and to dropping data which is strictly identical, but of course depending on how many features you have,  the same observations will or will not be strictly identical (even if only one feature is different for two observations, this is information that potentially can be valuable). Also the way you aggregate your data matters, even in your example there are ways to aggregate the data for the user with one activity only so that you get different (and different in a meaningful way) aggregated figures for each rolling window. </p>\n\n<p>But of course most of this is just heuristics, and it is hard to know in advance. what element of strategy is really going to give the model a competitive edge. </p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 430398,
      "author_name": "datahobbit",
      "author_url": "",
      "post_date": "11/30/2018 10:23:26",
      "content": "<p>Egor - thanks for taking the time to share your thoughts on this. Unfortunately doing this in the last day or two of a competition is not a popular choice. The reason for the large number of downvotes is likely because you are sharing this in so late in the competition. If it is effective then someone who has so far made no effort could score highly by implementing your approach. If your approach turns out to be unsound then people will waste valuable time on the last day working on it. This is why for kernels there is a clear notice from Kaggle saying please do not release new kernels that score well during the last week. For future reference, I suspect that if you had put this post out the day after the competition close, or when the results are announced then you would not have got downvotes and you may have got some upvotes.</p>",
      "votes": null,
      "replies": [
        {
          "id": 431227,
          "author_name": "eguliy",
          "author_url": "",
          "post_date": "12/01/2018 22:03:29",
          "content": "<p>Ok</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 435120,
          "author_name": "datahobbit",
          "author_url": "",
          "post_date": "12/07/2018 14:31:31",
          "content": "<p>I like some of the things you did and the way that you presented them. I agree with <a href=\"/luisduque440\">@luisduque440</a> that the overlapping time periods could lead to overfitting. Might have been interesting to check if you could have got similar results by doing everything you listed above but not shuffling and instead using a time-series validation approach where you only look forward in time at later time slots. I guess it is too late to do anything about it now though. Lets see where we end up in 2 months time.</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "429488": "Hello guys,\n\nI'm back to my notebook after short rest.\nIn next 2-3 days I will publish here some tricks I used in my solution.\n\nPlease comment, add your thoughts and improvements. Maybe you used some interesting tricks.\nUp to date I have about 0.26 RMSE (0.6 MSE) at CV.",
    "429491": "Use regression, no classification + regression\n\nSimple example which shows influence of classification error on MSE. Even assuming regression error(after applying classification) is zero, we have terrible results against dummy mean/median value, which we can get from original dataset. So the way through classification is wrong. Even if you get 99+% accuracy, you should remember, less than 1% of people in original dataset returned after so long gap. http://tinypic.com/r/qobmg3/9\n![clf_vs_reg][1]\n\n\n  [1]: http://oi64.tinypic.com/qobmg3.jpg",
    "429613": "Hi Egor, \nThanks for sharing. \nplease share what is your rmse when predicting all zeros?\n\nThanks",
    "429921": "Hi @AmirH,\n\nRMSE for all-zero is 0.29, BUT it depends of the CV way.\nHere is my approach for data augmentation and feature engineering. I shifted my window of observation, gap, and prediction periods across whole time I provided. CV was carried out on the last 20%(last 20% of rows after augmentation and shuffling rows).\n![clf_vs_reg][1]\n\n  [1]: http://oi65.tinypic.com/k54a6g.jpg",
    "430034": "Hi Egor,  the reported CV error comes from a period of time that overlaps the period you used to train your data? \n\nI did this and the results were great, but when considered a totally disjoint validation period (as we have to do for the competition) the results got very bad again.\n\nI think the most reallistic scenario  is to validate in a data set within a time period that was not used on training, otherwise, i believe the error might be underestimated.",
    "430352": "Hi Luis,\nYes I trained model on the random 80% of rows and validated it on the other 20%.\n\nYes, I do not exclude that the error may be underestimated",
    "430387": "Hey Egor, what exactly do you mean by duplicates?",
    "430398": "Egor - thanks for taking the time to share your thoughts on this. Unfortunately doing this in the last day or two of a competition is not a popular choice. The reason for the large number of downvotes is likely because you are sharing this in so late in the competition. If it is effective then someone who has so far made no effort could score highly by implementing your approach. If your approach turns out to be unsound then people will waste valuable time on the last day working on it. This is why for kernels there is a clear notice from Kaggle saying please do not release new kernels that score well during the last week. For future reference, I suspect that if you had put this post out the day after the competition close, or when the results are announced then you would not have got downvotes and you may have got some upvotes.",
    "430522": "Hello Antonio,\n\nDuplicates are users with the same behavior. They have same hits, revenue, browser, OS, etc.",
    "430600": "Ok, thanks. But just out of curiosity: at which stage do you identify them? The original data has session granularity, so are you saying that there are some users for whom all original session data are identical (so there are identical sessions started at the same time, but belonging to different users)?\n\nOr did you find the duplicates when you aggregated the data? In this case it is not necessary that the original data are identical for some users, but it just turned out that aggregated data are identical for some users with different fullVisitorIds (which seems a bit unlikely, to be honest)?\n\nCan you please clarify this?",
    "430671": "Yes, I mean duplicates after aggregating data.\n\nE.g. one visit with no hints(just go direct to GStore homepage and close tab after 1-3 sec -- case of wrong entry). Or maybe bot entry(1 sec visit, one page view). For me this is just one pattern. So I don't need a lot of data about this patterns.\n\nMoreover, you get a lot of duplicates while shifting window. E.g. activity within one day in the end of observation period leads to ~165 identical rows(yes, I drop userID after aggregating, for me user=pattern of user actions).\n\nNote, that the features are based on user behavior and some users may behave the same way. Thats why we can get duplicates. Especially from visits with zero revenue.\n\nIf you have any more questions, I'm ready to clarify.",
    "430963": "OK, now I see what you mean. Yes, you have a point when it comes to analysing groups of users (clustering) and to dropping data which is strictly identical, but of course depending on how many features you have,  the same observations will or will not be strictly identical (even if only one feature is different for two observations, this is information that potentially can be valuable). Also the way you aggregate your data matters, even in your example there are ways to aggregate the data for the user with one activity only so that you get different (and different in a meaningful way) aggregated figures for each rolling window. \n\nBut of course most of this is just heuristics, and it is hard to know in advance. what element of strategy is really going to give the model a competitive edge.",
    "431227": "Ok",
    "435120": "I like some of the things you did and the way that you presented them. I agree with @luisduque440 that the overlapping time periods could lead to overfitting. Might have been interesting to check if you could have got similar results by doing everything you listed above but not shuffling and instead using a time-series validation approach where you only look forward in time at later time slots. I guess it is too late to do anything about it now though. Lets see where we end up in 2 months time."
  },
  "source": "meta"
}