Just released a user level model scoring 1.72 on LB here :\nhttps://www.kaggle.com/ogrellier/user-level-lightgbm
\n\nUnlike most kernels do, the aggregation at user level is performed before training the model and should therefore create some diversity for stacking ;-)
\n\nLightGBM has not been tuned yet.
\n\nOther than that the kernel uses python logging package so that you can log whatever you like to a file and not depend on scripts showing the logs.
\n\nHope you'll find this useful.
", "messages": [ { "id": "388207", "postDate": "09/16/2018 15:04:29", "content": "Just released a user level model scoring 1.72 on LB here :\nhttps://www.kaggle.com/ogrellier/user-level-lightgbm
\n\nUnlike most kernels do, the aggregation at user level is performed before training the model and should therefore create some diversity for stacking ;-)
\n\nLightGBM has not been tuned yet.
\n\nOther than that the kernel uses python logging package so that you can log whatever you like to a file and not depend on scripts showing the logs.
\n\nHope you'll find this useful.
", "rawMarkdown": "Just released a user level model scoring 1.72 on LB here :\nhttps://www.kaggle.com/ogrellier/user-level-lightgbm\n\nUnlike most kernels do, the aggregation at user level is performed before training the model and should therefore create some diversity for stacking ;-)\n\nLightGBM has not been tuned yet.\n\nOther than that the kernel uses python logging package so that you can log whatever you like to a file and not depend on scripts showing the logs. \n\nHope you'll find this useful.", "votes": null }, { "id": "388252", "postDate": "09/16/2018 16:30:52", "content": "Nice work!
\n\nAre you aware of the debate about whether the scoring metric is specified incorrectly? A few people (myself included) are getting much better leaderboard scores (and scores that are more representative of the local CV) when they submit the sum of the log of per-user predictions instead of the log of the sum. That might mean that there's a problem with the scoring, and so I'd be wary of pursuing the aggregation-first method too far until the mystery is solved as this can only give you the log of the sum. The \"problem\" could of course be a quirk of the noise in the public leaderboard, but I'm leaning towards there being a mistake at the moment.
\n\nThere's a bit of discussion about it here
", "rawMarkdown": "Nice work!\n\nAre you aware of the debate about whether the scoring metric is specified incorrectly? A few people (myself included) are getting much better leaderboard scores (and scores that are more representative of the local CV) when they submit the sum of the log of per-user predictions instead of the log of the sum. That might mean that there's a problem with the scoring, and so I'd be wary of pursuing the aggregation-first method too far until the mystery is solved as this can only give you the log of the sum. The \"problem\" could of course be a quirk of the noise in the public leaderboard, but I'm leaning towards there being a mistake at the moment.\n\nThere's a bit of discussion about it [here](https://www.kaggle.com/c/google-analytics-customer-revenue-prediction/discussion/65806)", "votes": null }, { "id": "388278", "postDate": "09/16/2018 17:16:58", "content": "Thanks Craig for the heads up ! There was already a scoring problem telling competitors the metric was the log instead of log(1 + x) but was not aware of the issue you raised. I guess I can still aggregate on the log of the revenue. I may have a go and see if I get better results.
Of course - I'm not sure why I didn't think of aggregating on the log of the revenue - doh!
\n\nIt will be nice to see if the alternative aggregation gives you a boost as well, especially given your diverse approach
", "rawMarkdown": "Of course - I'm not sure why I didn't think of aggregating on the log of the revenue - doh! \n\nIt will be nice to see if the alternative aggregation gives you a boost as well, especially given your diverse approach", "votes": null }, { "id": "388281", "postDate": "09/16/2018 17:29:34", "content": "I'll keep you posted with that. I should be able to check tomorrow.
", "rawMarkdown": "I'll keep you posted with that. I should be able to check tomorrow.", "votes": null }, { "id": "388289", "postDate": "09/16/2018 17:52:37", "content": "Great work. I am interested too. I suspect moving np.log1p of the revenues before aggregate_sessions should result in better LB (and local CV -- but that's entirely depending on how the \"actual target\" is defined).
", "rawMarkdown": "Great work. I am interested too. I suspect moving np.log1p of the revenues before aggregate_sessions should result in better LB (and local CV -- but that's entirely depending on how the \"actual target\" is defined).", "votes": null }, { "id": "388547", "postDate": "09/17/2018 07:06:17", "content": "@yilihome and @ca3295 Just did the experiment you asked for and LB improved. I made a write up here
", "rawMarkdown": "yilihome and @ca3295 Just did the experiment you asked for and LB improved. I made a write up [here](https://www.kaggle.com/c/google-analytics-customer-revenue-prediction/discussion/65995)", "votes": null }, { "id": "388592", "postDate": "09/17/2018 08:29:31", "content": "Thanks!!
", "rawMarkdown": "Thanks!!", "votes": null } ], "comments": [ { "id": 388252, "author_name": "ca3295", "author_url": "", "post_date": "09/16/2018 16:30:52", "content": "Nice work!
\n\nAre you aware of the debate about whether the scoring metric is specified incorrectly? A few people (myself included) are getting much better leaderboard scores (and scores that are more representative of the local CV) when they submit the sum of the log of per-user predictions instead of the log of the sum. That might mean that there's a problem with the scoring, and so I'd be wary of pursuing the aggregation-first method too far until the mystery is solved as this can only give you the log of the sum. The \"problem\" could of course be a quirk of the noise in the public leaderboard, but I'm leaning towards there being a mistake at the moment.
\n\nThere's a bit of discussion about it here
", "votes": null, "replies": [ { "id": 388278, "author_name": "ogrellier", "author_url": "", "post_date": "09/16/2018 17:16:58", "content": "Thanks Craig for the heads up ! There was already a scoring problem telling competitors the metric was the log instead of log(1 + x) but was not aware of the issue you raised. I guess I can still aggregate on the log of the revenue. I may have a go and see if I get better results.
Of course - I'm not sure why I didn't think of aggregating on the log of the revenue - doh!
\n\nIt will be nice to see if the alternative aggregation gives you a boost as well, especially given your diverse approach
", "votes": null, "replies": [] }, { "id": 388281, "author_name": "ogrellier", "author_url": "", "post_date": "09/16/2018 17:29:34", "content": "I'll keep you posted with that. I should be able to check tomorrow.
", "votes": null, "replies": [] }, { "id": 388289, "author_name": "yilihome", "author_url": "", "post_date": "09/16/2018 17:52:37", "content": "Great work. I am interested too. I suspect moving np.log1p of the revenues before aggregate_sessions should result in better LB (and local CV -- but that's entirely depending on how the \"actual target\" is defined).
", "votes": null, "replies": [] }, { "id": 388547, "author_name": "ogrellier", "author_url": "", "post_date": "09/17/2018 07:06:17", "content": "@yilihome and @ca3295 Just did the experiment you asked for and LB improved. I made a write up here
", "votes": null, "replies": [] }, { "id": 388592, "author_name": "yilihome", "author_url": "", "post_date": "09/17/2018 08:29:31", "content": "Thanks!!
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "388207": "Just released a user level model scoring 1.72 on LB here :\nhttps://www.kaggle.com/ogrellier/user-level-lightgbm\n\nUnlike most kernels do, the aggregation at user level is performed before training the model and should therefore create some diversity for stacking ;-)\n\nLightGBM has not been tuned yet.\n\nOther than that the kernel uses python logging package so that you can log whatever you like to a file and not depend on scripts showing the logs. \n\nHope you'll find this useful.", "388252": "Nice work!\n\nAre you aware of the debate about whether the scoring metric is specified incorrectly? A few people (myself included) are getting much better leaderboard scores (and scores that are more representative of the local CV) when they submit the sum of the log of per-user predictions instead of the log of the sum. That might mean that there's a problem with the scoring, and so I'd be wary of pursuing the aggregation-first method too far until the mystery is solved as this can only give you the log of the sum. The \"problem\" could of course be a quirk of the noise in the public leaderboard, but I'm leaning towards there being a mistake at the moment.\n\nThere's a bit of discussion about it [here](https://www.kaggle.com/c/google-analytics-customer-revenue-prediction/discussion/65806)", "388278": "Thanks Craig for the heads up ! There was already a scoring problem telling competitors the metric was the log instead of `log(1 + x)` but was not aware of the issue you raised. I guess I can still aggregate on the log of the revenue. I may have a go and see if I get better results.", "388280": "Of course - I'm not sure why I didn't think of aggregating on the log of the revenue - doh! \n\nIt will be nice to see if the alternative aggregation gives you a boost as well, especially given your diverse approach", "388281": "I'll keep you posted with that. I should be able to check tomorrow.", "388289": "Great work. I am interested too. I suspect moving np.log1p of the revenues before aggregate_sessions should result in better LB (and local CV -- but that's entirely depending on how the \"actual target\" is defined).", "388547": "yilihome and @ca3295 Just did the experiment you asked for and LB improved. I made a write up [here](https://www.kaggle.com/c/google-analytics-customer-revenue-prediction/discussion/65995)", "388592": "Thanks!!" }, "source": "meta" }