I've done a simple experiment, splitting train into 2 blocks (time based), made a 5 fold OOF on the 1st block, predicted the second block. The OOF rmse score on the sessions was 1.61 and test score was 1.69.
\n\nHere are rmse values on different ways to aggregate things :
\n\nLOG1P of the sum of actual revenues gives the best rmse score and the closest to the OOF fold score.
", "messages": [ { "id": "392758", "postDate": "09/24/2018 09:46:23", "content": "I've done a simple experiment, splitting train into 2 blocks (time based), made a 5 fold OOF on the 1st block, predicted the second block. The OOF rmse score on the sessions was 1.61 and test score was 1.69.
\n\nHere are rmse values on different ways to aggregate things :
\n\nLOG1P of the sum of actual revenues gives the best rmse score and the closest to the OOF fold score.
", "rawMarkdown": "I've done a simple experiment, splitting train into 2 blocks (time based), made a 5 fold OOF on the 1st block, predicted the second block. The OOF rmse score on the sessions was 1.61 and test score was 1.69. \n\nHere are rmse values on different ways to aggregate things : \n\n - LOG1P of SUM aggregated by dates TRN : 7.32\n - LOG1P of SUM aggregated by dates TST : 8.61\n - SUM of LOG1P aggregated by dates TRN : 167.33\n - SUM of LOG1P aggregated by dates TST : 209.78\n - LOG1P of SUM aggregated by visitors TRN : 1.57\n - LOG1P of SUM aggregated by visitors TST : 1.73\n - SUM of LOG1P aggregated by visitors TRN : 1.91\n - SUM of LOG1P aggregated by visitors TST : 2.20\n\nLOG1P of the sum of actual revenues gives the best rmse score and the closest to the OOF fold score.", "votes": null }, { "id": "392778", "postDate": "09/24/2018 10:49:11", "content": "Hence, that means that we should proceed with a user level log1p(transactions) despite the fact that the most models score very poor on the LB? My perspective is yes, due to the fact that Y_evaluation of the private LB is calculated as log1p(sum(transactions))
", "rawMarkdown": "Hence, that means that we should proceed with a user level log1p(transactions) despite the fact that the most models score very poor on the LB? My perspective is yes, due to the fact that Y_evaluation of the private LB is calculated as log1p(sum(transactions))", "votes": null }, { "id": "392783", "postDate": "09/24/2018 11:11:47", "content": "I must admit I still fail to understand why sum of logs gives better LB results although LOG of sum transactions vs sum of log predicted gives a rmse that's not completely off.
", "rawMarkdown": "I must admit I still fail to understand why sum of logs gives better LB results although LOG of sum transactions vs sum of log predicted gives a rmse that's not completely off.", "votes": null }, { "id": "392787", "postDate": "09/24/2018 11:17:49", "content": "Awesome! Do you think it's possible to find public/private split of your 2nd part to emulate situation with our leaderboard?
", "rawMarkdown": "Awesome! Do you think it's possible to find public/private split of your 2nd part to emulate situation with our leaderboard?", "votes": null }, { "id": "392793", "postDate": "09/24/2018 11:34:38", "content": "Hi Sergei, I used train as anything in 2016 and test as anything in 2017. The results I get from that are quite similar to the scores I get from the LB except that sum of logs does not yield better results on my test set.
\n\nI have no idea how we could find a split in my test set that would mimic the public LB :( Maybe finding a split that gives better results with sum of logs ;-) lol
", "rawMarkdown": "Hi Sergei, I used train as anything in 2016 and test as anything in 2017. The results I get from that are quite similar to the scores I get from the LB except that sum of logs does not yield better results on my test set.\n\nI have no idea how we could find a split in my test set that would mimic the public LB :( Maybe finding a split that gives better results with sum of logs ;-) lol", "votes": null }, { "id": "392803", "postDate": "09/24/2018 11:48:46", "content": "It's a good idea in general as for me. But it needs a computing power to find such split. We need multiple splits to check the private part score deviation.
", "rawMarkdown": "It's a good idea in general as for me. But it needs a computing power to find such split. We need multiple splits to check the private part score deviation.", "votes": null }, { "id": "392804", "postDate": "09/24/2018 11:48:54", "content": "Concerning this difference, i believe that there many facts.\n1.) the 98.7% of transactions have zero value\n2.) the 13% of distinct users has more than one transaction\n3.) the LB is on 30% of test set.\n4.) the users with only one transaction(87%) have no effect regarding the log of sum or the sum of log.\n5.) I tried to train my best model (1.59 CV,1.6536 LB only on individual users with CV of ~0.9 and only on users with multiple transactions with CV ~2.4. The combined model had a little worse results.
\n\nIn conclusion, maybe, the 30% LB with such an imbalanced dataset has not so many cases with users of multiple transactions. Hence, the LB is unreliable.\nFurthermore, consider that the users with multiple transactions have the majority amount of the total transactions.
", "rawMarkdown": "Concerning this difference, i believe that there many facts.\n1.) the 98.7% of transactions have zero value\n2.) the 13% of distinct users has more than one transaction\n3.) the LB is on 30% of test set.\n4.) the users with only one transaction(87%) have no effect regarding the log of sum or the sum of log.\n5.) I tried to train my best model (1.59 CV,1.6536 LB only on individual users with CV of ~0.9 and only on users with multiple transactions with CV ~2.4. The combined model had a little worse results.\n\nIn conclusion, maybe, the 30% LB with such an imbalanced dataset has not so many cases with users of multiple transactions. Hence, the LB is unreliable.\nFurthermore, consider that the users with multiple transactions have the majority amount of the total transactions.", "votes": null }, { "id": "392805", "postDate": "09/24/2018 11:50:30", "content": "Thinking about it, I suppose this means you need visitors with more non-zero sessions in Public LB. As suggested by the last plot of this kernel
", "rawMarkdown": "Thinking about it, I suppose this means you need visitors with more non-zero sessions in Public LB. As suggested by the last plot of [this kernel](https://www.kaggle.com/ogrellier/using-classification-for-predictions)", "votes": null }, { "id": "392818", "postDate": "09/24/2018 12:20:50", "content": "Maybe in the test set, new users are introduced who buy items regularly. So they were underestimated since there is no such buyers in train set and they were corrected thanks to sum of logs. This is just one of my theories, I will spend some submissions on it.
", "rawMarkdown": "Maybe in the test set, new users are introduced who buy items regularly. So they were underestimated since there is no such buyers in train set and they were corrected thanks to sum of logs. This is just one of my theories, I will spend some submissions on it.", "votes": null }, { "id": "392823", "postDate": "09/24/2018 12:33:50", "content": "Great analysis, thank you. Do we know how the \"sum of log\" public LB phenomenon was discovered? I'm curious if it was simply an erroneous interpretation of the performance metric to start with (it must have been?).
", "rawMarkdown": "Great analysis, thank you. Do we know how the \"sum of log\" public LB phenomenon was discovered? I'm curious if it was simply an erroneous interpretation of the performance metric to start with (it must have been?).", "votes": null }, { "id": "392959", "postDate": "09/24/2018 16:22:22", "content": "Yeah, someone erroneously submitted sum of logs and got good results. After sharing it with others many of us are using it ;).
", "rawMarkdown": "Yeah, someone erroneously submitted sum of logs and got good results. After sharing it with others many of us are using it ;).", "votes": null }, { "id": "392971", "postDate": "09/24/2018 16:37:58", "content": "What does \"aggregated by dates\" and \"aggregated by visitors\" mean?
", "rawMarkdown": "What does \"aggregated by dates\" and \"aggregated by visitors\" mean?", "votes": null }, { "id": "393006", "postDate": "09/24/2018 17:26:50", "content": "Hi Corey, you can either groupby date or groupby fullVisitorId. The groupby date is used to look at the evolution of total transaction over time, the other is used to submit our preductions ;-)
", "rawMarkdown": "Hi Corey, you can either groupby date or groupby fullVisitorId. The groupby date is used to look at the evolution of total transaction over time, the other is used to submit our preductions ;-)", "votes": null }, { "id": "393027", "postDate": "09/24/2018 18:09:47", "content": "I do not understand you. First, why are you doing group by at all? Are you doing groupby in the KFold?
", "rawMarkdown": "I do not understand you. First, why are you doing group by at all? Are you doing groupby in the KFold?", "votes": null }, { "id": "393074", "postDate": "09/24/2018 19:45:07", "content": "I would like to point out if the better score of log(sum(transactions)) instead of sum(log(transactions)) is a fallacy.\nAccidently, i trained with a target of log(1+log(1+transations)) and i got an RMSE of 0.35, but when transforming back to sum of logs per user i got an LB of 1.74. \nI think i will quit the competition, due to metric :D
", "rawMarkdown": "I would like to point out if the better score of log(sum(transactions)) instead of sum(log(transactions)) is a fallacy.\nAccidently, i trained with a target of log(1+log(1+transations)) and i got an RMSE of 0.35, but when transforming back to sum of logs per user i got an LB of 1.74. \nI think i will quit the competition, due to metric :D", "votes": null }, { "id": "393099", "postDate": "09/24/2018 20:20:21", "content": "Corey, the datatset is at session level, the submission is at user level. so at some point to make a submission you will need to groupby fullVisitorId !
\n\nNow at some point you may want to check your predictions, one option is to check if at date level your predictions look the same as the actual revenues per day.
\n\nFor these 2 reasons, I groupby date and I groupby fullVisitorId.
\n\nI don't do the groupby within the KFold I do the groupby and check the RMSE once predictions are made using OOF for the train part and with predictions for the test part.
\n\nHope this is clearer.
", "rawMarkdown": "Corey, the datatset is at session level, the submission is at user level. so at some point to make a submission you will need to groupby fullVisitorId !\n\nNow at some point you may want to check your predictions, one option is to check if at date level your predictions look the same as the actual revenues per day.\n\nFor these 2 reasons, I groupby date and I groupby fullVisitorId.\n\nI don't do the groupby within the KFold I do the groupby and check the RMSE once predictions are made using OOF for the train part and with predictions for the test part.\n\nHope this is clearer.", "votes": null }, { "id": "393126", "postDate": "09/24/2018 21:18:03", "content": "This is a lot clearer, thank you.
\n\nHowever, I have another question now. If you are looking at RMSE aggregated by dates, you are forced to do this at the transaction level, correct? Because, you need to sum the transactions for that date, instead of the user level.
", "rawMarkdown": "This is a lot clearer, thank you.\n\nHowever, I have another question now. If you are looking at RMSE aggregated by dates, you are forced to do this at the transaction level, correct? Because, you need to sum the transactions for that date, instead of the user level.", "votes": null }, { "id": "393367", "postDate": "09/25/2018 08:54:11", "content": "Thanks for the good ideas to bring it. I know a lot more. :) สมัครเอเย่นต์omgbet :J
", "rawMarkdown": "Thanks for the good ideas to bring it. I know a lot more. :) สมัครเอเย่นต์omgbet :J", "votes": null }, { "id": "393595", "postDate": "09/25/2018 16:54:33", "content": "Thanks for all the work you did to shed light on this.
\n\nHow does you local CV/LB score compare now that the scoring metric is fixed? I'm finding the LB score is significantly lower than my local CV, even at the aggregated level.
\n\nI've been messing around trying to work out why this might be, and I've found that summing the squared errors and then dividing by the number of rows in the unaggregated data gets me much closer to the LB score than dividing by the number of rows in the aggregated data (obviously I take the sqrt after dividing). If your CV and LB scores match up nicely I can be confident it's a problem with my script :-)
", "rawMarkdown": "Thanks for all the work you did to shed light on this.\n\nHow does you local CV/LB score compare now that the scoring metric is fixed? I'm finding the LB score is significantly lower than my local CV, even at the aggregated level. \n\nI've been messing around trying to work out why this might be, and I've found that summing the squared errors and then dividing by the number of rows in the unaggregated data gets me much closer to the LB score than dividing by the number of rows in the aggregated data (obviously I take the sqrt after dividing). If your CV and LB scores match up nicely I can be confident it's a problem with my script :-)", "votes": null }, { "id": "393610", "postDate": "09/25/2018 17:11:14", "content": "OMG, don't tell me there's a second problem here .............................
\n\nLB is significantly lower that CV of about 0.16 I think, so it's probably not a problem with your script.
\n\nJust to reword what you wrote you compte the sum of squared error (at the Visitor level) and then divide by the number of sessions or rows in the original file, is that right ?
", "rawMarkdown": "OMG, don't tell me there's a second problem here .............................\n\nLB is significantly lower that CV of about 0.16 I think, so it's probably not a problem with your script.\n\nJust to reword what you wrote you compte the sum of squared error (at the Visitor level) and then divide by the number of sessions or rows in the original file, is that right ?", "votes": null }, { "id": "393640", "postDate": "09/25/2018 17:43:52", "content": "OK - first time using LaTeX markup in Kaggle, so forgive me if this takes a few edits....
\n\nLet n be the total number of visits and m be the number of unique visitors. y is ln1p(sum revenue), and y hat is the estimate.
\n\nThe metric should be:
\n\n$$\n\\sqrt{\n\\frac{\\sum_{i=1}^{m}(y_i-\\hat y_i)^2}{m}\n}\n$$
\n\nBut:
\n\n$$\n\\sqrt{\n\\frac{\\sum_{i=1}^{m}(y_i-\\hat y_i)^2}{n}\n}\n$$
\n\nGives me closer agreement with the leader board. Could be a total coincidence, but is also a plausible mistake to make. If what I've said is right, then LB scores are out by a factor of root(m/n) - which isn't the end of the world. Of course it could easily be be noise.
", "rawMarkdown": "OK - first time using LaTeX markup in Kaggle, so forgive me if this takes a few edits....\n\nLet n be the total number of visits and m be the number of unique visitors. y is ln1p(sum revenue), and y hat is the estimate.\n\nThe metric should be:\n\n$$\n\\sqrt{\n\\frac{\\sum_{i=1}^{m}(y_i-\\hat y_i)^2}{m}\n}\n$$\n\nBut:\n\n$$\n\\sqrt{\n\\frac{\\sum_{i=1}^{m}(y_i-\\hat y_i)^2}{n}\n}\n$$\n\nGives me closer agreement with the leader board. Could be a total coincidence, but is also a plausible mistake to make. If what I've said is right, then LB scores are out by a factor of root(m/n) - which isn't the end of the world. Of course it could easily be be noise.", "votes": null }, { "id": "393649", "postDate": "09/25/2018 18:04:31", "content": "You're not bad with LaTeX for a first time ;-)\nOk that's what I understood!
\n\nThanks for you work Craig ! I let you post yet another topic on the bloody metric ?
\n\nPS: And I assure you you'll get my upvote :)
", "rawMarkdown": "You're not bad with LaTeX for a first time ;-)\nOk that's what I understood! \n\nThanks for you work Craig ! I let you post yet another topic on the bloody metric ?\n\nPS: And I assure you you'll get my upvote :)", "votes": null }, { "id": "393659", "postDate": "09/25/2018 18:17:57", "content": "I've posted here: https://www.kaggle.com/c/ga-customer-revenue-prediction/discussion/66811
\n\nI tried to tag you but unfortunately my tagging skills are not at the same level as my LaTeX skills.
\n\nYes the upvote would be appreciated - posting here does make me nervous. The great thing about Kaggle is that it's full of very smart people that I can learn from. The other thing about Kaggle is that it's full of very smart people that can be intellectually intimidating!
", "rawMarkdown": "I've posted here: https://www.kaggle.com/c/ga-customer-revenue-prediction/discussion/66811\n\nI tried to tag you but unfortunately my tagging skills are not at the same level as my LaTeX skills.\n\nYes the upvote would be appreciated - posting here does make me nervous. The great thing about Kaggle is that it's full of very smart people that I can learn from. The other thing about Kaggle is that it's full of very smart people that can be intellectually intimidating!", "votes": null }, { "id": "393666", "postDate": "09/25/2018 18:29:28", "content": ":facepalm: This is getting ridiculous.
", "rawMarkdown": ":facepalm: This is getting ridiculous.", "votes": null } ], "comments": [ { "id": 392778, "author_name": "dkaraflos", "author_url": "", "post_date": "09/24/2018 10:49:11", "content": "Hence, that means that we should proceed with a user level log1p(transactions) despite the fact that the most models score very poor on the LB? My perspective is yes, due to the fact that Y_evaluation of the private LB is calculated as log1p(sum(transactions))
", "votes": null, "replies": [] }, { "id": 392783, "author_name": "ogrellier", "author_url": "", "post_date": "09/24/2018 11:11:47", "content": "I must admit I still fail to understand why sum of logs gives better LB results although LOG of sum transactions vs sum of log predicted gives a rmse that's not completely off.
", "votes": null, "replies": [ { "id": 392804, "author_name": "dkaraflos", "author_url": "", "post_date": "09/24/2018 11:48:54", "content": "Concerning this difference, i believe that there many facts.\n1.) the 98.7% of transactions have zero value\n2.) the 13% of distinct users has more than one transaction\n3.) the LB is on 30% of test set.\n4.) the users with only one transaction(87%) have no effect regarding the log of sum or the sum of log.\n5.) I tried to train my best model (1.59 CV,1.6536 LB only on individual users with CV of ~0.9 and only on users with multiple transactions with CV ~2.4. The combined model had a little worse results.
\n\nIn conclusion, maybe, the 30% LB with such an imbalanced dataset has not so many cases with users of multiple transactions. Hence, the LB is unreliable.\nFurthermore, consider that the users with multiple transactions have the majority amount of the total transactions.
", "votes": null, "replies": [] } ] }, { "id": 392787, "author_name": "sergeifironov", "author_url": "", "post_date": "09/24/2018 11:17:49", "content": "Awesome! Do you think it's possible to find public/private split of your 2nd part to emulate situation with our leaderboard?
", "votes": null, "replies": [ { "id": 392793, "author_name": "ogrellier", "author_url": "", "post_date": "09/24/2018 11:34:38", "content": "Hi Sergei, I used train as anything in 2016 and test as anything in 2017. The results I get from that are quite similar to the scores I get from the LB except that sum of logs does not yield better results on my test set.
\n\nI have no idea how we could find a split in my test set that would mimic the public LB :( Maybe finding a split that gives better results with sum of logs ;-) lol
", "votes": null, "replies": [] }, { "id": 392803, "author_name": "sergeifironov", "author_url": "", "post_date": "09/24/2018 11:48:46", "content": "It's a good idea in general as for me. But it needs a computing power to find such split. We need multiple splits to check the private part score deviation.
", "votes": null, "replies": [] }, { "id": 392805, "author_name": "ogrellier", "author_url": "", "post_date": "09/24/2018 11:50:30", "content": "Thinking about it, I suppose this means you need visitors with more non-zero sessions in Public LB. As suggested by the last plot of this kernel
", "votes": null, "replies": [] } ] }, { "id": 392818, "author_name": "aerdem4", "author_url": "", "post_date": "09/24/2018 12:20:50", "content": "Maybe in the test set, new users are introduced who buy items regularly. So they were underestimated since there is no such buyers in train set and they were corrected thanks to sum of logs. This is just one of my theories, I will spend some submissions on it.
", "votes": null, "replies": [] }, { "id": 392823, "author_name": "nnnnick", "author_url": "", "post_date": "09/24/2018 12:33:50", "content": "Great analysis, thank you. Do we know how the \"sum of log\" public LB phenomenon was discovered? I'm curious if it was simply an erroneous interpretation of the performance metric to start with (it must have been?).
", "votes": null, "replies": [ { "id": 392959, "author_name": "kainsama", "author_url": "", "post_date": "09/24/2018 16:22:22", "content": "Yeah, someone erroneously submitted sum of logs and got good results. After sharing it with others many of us are using it ;).
", "votes": null, "replies": [] } ] }, { "id": 392971, "author_name": "returnofsputnik", "author_url": "", "post_date": "09/24/2018 16:37:58", "content": "What does \"aggregated by dates\" and \"aggregated by visitors\" mean?
", "votes": null, "replies": [ { "id": 393006, "author_name": "ogrellier", "author_url": "", "post_date": "09/24/2018 17:26:50", "content": "Hi Corey, you can either groupby date or groupby fullVisitorId. The groupby date is used to look at the evolution of total transaction over time, the other is used to submit our preductions ;-)
", "votes": null, "replies": [] }, { "id": 393027, "author_name": "returnofsputnik", "author_url": "", "post_date": "09/24/2018 18:09:47", "content": "I do not understand you. First, why are you doing group by at all? Are you doing groupby in the KFold?
", "votes": null, "replies": [] }, { "id": 393099, "author_name": "ogrellier", "author_url": "", "post_date": "09/24/2018 20:20:21", "content": "Corey, the datatset is at session level, the submission is at user level. so at some point to make a submission you will need to groupby fullVisitorId !
\n\nNow at some point you may want to check your predictions, one option is to check if at date level your predictions look the same as the actual revenues per day.
\n\nFor these 2 reasons, I groupby date and I groupby fullVisitorId.
\n\nI don't do the groupby within the KFold I do the groupby and check the RMSE once predictions are made using OOF for the train part and with predictions for the test part.
\n\nHope this is clearer.
", "votes": null, "replies": [] }, { "id": 393126, "author_name": "returnofsputnik", "author_url": "", "post_date": "09/24/2018 21:18:03", "content": "This is a lot clearer, thank you.
\n\nHowever, I have another question now. If you are looking at RMSE aggregated by dates, you are forced to do this at the transaction level, correct? Because, you need to sum the transactions for that date, instead of the user level.
", "votes": null, "replies": [] } ] }, { "id": 393074, "author_name": "dkaraflos", "author_url": "", "post_date": "09/24/2018 19:45:07", "content": "I would like to point out if the better score of log(sum(transactions)) instead of sum(log(transactions)) is a fallacy.\nAccidently, i trained with a target of log(1+log(1+transations)) and i got an RMSE of 0.35, but when transforming back to sum of logs per user i got an LB of 1.74. \nI think i will quit the competition, due to metric :D
", "votes": null, "replies": [] }, { "id": 393367, "author_name": "seoemlon06", "author_url": "", "post_date": "09/25/2018 08:54:11", "content": "Thanks for the good ideas to bring it. I know a lot more. :) สมัครเอเย่นต์omgbet :J
", "votes": null, "replies": [] }, { "id": 393595, "author_name": "ca3295", "author_url": "", "post_date": "09/25/2018 16:54:33", "content": "Thanks for all the work you did to shed light on this.
\n\nHow does you local CV/LB score compare now that the scoring metric is fixed? I'm finding the LB score is significantly lower than my local CV, even at the aggregated level.
\n\nI've been messing around trying to work out why this might be, and I've found that summing the squared errors and then dividing by the number of rows in the unaggregated data gets me much closer to the LB score than dividing by the number of rows in the aggregated data (obviously I take the sqrt after dividing). If your CV and LB scores match up nicely I can be confident it's a problem with my script :-)
", "votes": null, "replies": [ { "id": 393610, "author_name": "ogrellier", "author_url": "", "post_date": "09/25/2018 17:11:14", "content": "OMG, don't tell me there's a second problem here .............................
\n\nLB is significantly lower that CV of about 0.16 I think, so it's probably not a problem with your script.
\n\nJust to reword what you wrote you compte the sum of squared error (at the Visitor level) and then divide by the number of sessions or rows in the original file, is that right ?
", "votes": null, "replies": [] }, { "id": 393640, "author_name": "ca3295", "author_url": "", "post_date": "09/25/2018 17:43:52", "content": "OK - first time using LaTeX markup in Kaggle, so forgive me if this takes a few edits....
\n\nLet n be the total number of visits and m be the number of unique visitors. y is ln1p(sum revenue), and y hat is the estimate.
\n\nThe metric should be:
\n\n$$\n\\sqrt{\n\\frac{\\sum_{i=1}^{m}(y_i-\\hat y_i)^2}{m}\n}\n$$
\n\nBut:
\n\n$$\n\\sqrt{\n\\frac{\\sum_{i=1}^{m}(y_i-\\hat y_i)^2}{n}\n}\n$$
\n\nGives me closer agreement with the leader board. Could be a total coincidence, but is also a plausible mistake to make. If what I've said is right, then LB scores are out by a factor of root(m/n) - which isn't the end of the world. Of course it could easily be be noise.
", "votes": null, "replies": [] }, { "id": 393649, "author_name": "ogrellier", "author_url": "", "post_date": "09/25/2018 18:04:31", "content": "You're not bad with LaTeX for a first time ;-)\nOk that's what I understood!
\n\nThanks for you work Craig ! I let you post yet another topic on the bloody metric ?
\n\nPS: And I assure you you'll get my upvote :)
", "votes": null, "replies": [] }, { "id": 393659, "author_name": "ca3295", "author_url": "", "post_date": "09/25/2018 18:17:57", "content": "I've posted here: https://www.kaggle.com/c/ga-customer-revenue-prediction/discussion/66811
\n\nI tried to tag you but unfortunately my tagging skills are not at the same level as my LaTeX skills.
\n\nYes the upvote would be appreciated - posting here does make me nervous. The great thing about Kaggle is that it's full of very smart people that I can learn from. The other thing about Kaggle is that it's full of very smart people that can be intellectually intimidating!
", "votes": null, "replies": [] }, { "id": 393666, "author_name": "tunguz", "author_url": "", "post_date": "09/25/2018 18:29:28", "content": ":facepalm: This is getting ridiculous.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "392758": "I've done a simple experiment, splitting train into 2 blocks (time based), made a 5 fold OOF on the 1st block, predicted the second block. The OOF rmse score on the sessions was 1.61 and test score was 1.69. \n\nHere are rmse values on different ways to aggregate things : \n\n - LOG1P of SUM aggregated by dates TRN : 7.32\n - LOG1P of SUM aggregated by dates TST : 8.61\n - SUM of LOG1P aggregated by dates TRN : 167.33\n - SUM of LOG1P aggregated by dates TST : 209.78\n - LOG1P of SUM aggregated by visitors TRN : 1.57\n - LOG1P of SUM aggregated by visitors TST : 1.73\n - SUM of LOG1P aggregated by visitors TRN : 1.91\n - SUM of LOG1P aggregated by visitors TST : 2.20\n\nLOG1P of the sum of actual revenues gives the best rmse score and the closest to the OOF fold score.", "392778": "Hence, that means that we should proceed with a user level log1p(transactions) despite the fact that the most models score very poor on the LB? My perspective is yes, due to the fact that Y_evaluation of the private LB is calculated as log1p(sum(transactions))", "392783": "I must admit I still fail to understand why sum of logs gives better LB results although LOG of sum transactions vs sum of log predicted gives a rmse that's not completely off.", "392787": "Awesome! Do you think it's possible to find public/private split of your 2nd part to emulate situation with our leaderboard?", "392793": "Hi Sergei, I used train as anything in 2016 and test as anything in 2017. The results I get from that are quite similar to the scores I get from the LB except that sum of logs does not yield better results on my test set.\n\nI have no idea how we could find a split in my test set that would mimic the public LB :( Maybe finding a split that gives better results with sum of logs ;-) lol", "392803": "It's a good idea in general as for me. But it needs a computing power to find such split. We need multiple splits to check the private part score deviation.", "392804": "Concerning this difference, i believe that there many facts.\n1.) the 98.7% of transactions have zero value\n2.) the 13% of distinct users has more than one transaction\n3.) the LB is on 30% of test set.\n4.) the users with only one transaction(87%) have no effect regarding the log of sum or the sum of log.\n5.) I tried to train my best model (1.59 CV,1.6536 LB only on individual users with CV of ~0.9 and only on users with multiple transactions with CV ~2.4. The combined model had a little worse results.\n\nIn conclusion, maybe, the 30% LB with such an imbalanced dataset has not so many cases with users of multiple transactions. Hence, the LB is unreliable.\nFurthermore, consider that the users with multiple transactions have the majority amount of the total transactions.", "392805": "Thinking about it, I suppose this means you need visitors with more non-zero sessions in Public LB. As suggested by the last plot of [this kernel](https://www.kaggle.com/ogrellier/using-classification-for-predictions)", "392818": "Maybe in the test set, new users are introduced who buy items regularly. So they were underestimated since there is no such buyers in train set and they were corrected thanks to sum of logs. This is just one of my theories, I will spend some submissions on it.", "392823": "Great analysis, thank you. Do we know how the \"sum of log\" public LB phenomenon was discovered? I'm curious if it was simply an erroneous interpretation of the performance metric to start with (it must have been?).", "392959": "Yeah, someone erroneously submitted sum of logs and got good results. After sharing it with others many of us are using it ;).", "392971": "What does \"aggregated by dates\" and \"aggregated by visitors\" mean?", "393006": "Hi Corey, you can either groupby date or groupby fullVisitorId. The groupby date is used to look at the evolution of total transaction over time, the other is used to submit our preductions ;-)", "393027": "I do not understand you. First, why are you doing group by at all? Are you doing groupby in the KFold?", "393074": "I would like to point out if the better score of log(sum(transactions)) instead of sum(log(transactions)) is a fallacy.\nAccidently, i trained with a target of log(1+log(1+transations)) and i got an RMSE of 0.35, but when transforming back to sum of logs per user i got an LB of 1.74. \nI think i will quit the competition, due to metric :D", "393099": "Corey, the datatset is at session level, the submission is at user level. so at some point to make a submission you will need to groupby fullVisitorId !\n\nNow at some point you may want to check your predictions, one option is to check if at date level your predictions look the same as the actual revenues per day.\n\nFor these 2 reasons, I groupby date and I groupby fullVisitorId.\n\nI don't do the groupby within the KFold I do the groupby and check the RMSE once predictions are made using OOF for the train part and with predictions for the test part.\n\nHope this is clearer.", "393126": "This is a lot clearer, thank you.\n\nHowever, I have another question now. If you are looking at RMSE aggregated by dates, you are forced to do this at the transaction level, correct? Because, you need to sum the transactions for that date, instead of the user level.", "393367": "Thanks for the good ideas to bring it. I know a lot more. :) สมัครเอเย่นต์omgbet :J", "393595": "Thanks for all the work you did to shed light on this.\n\nHow does you local CV/LB score compare now that the scoring metric is fixed? I'm finding the LB score is significantly lower than my local CV, even at the aggregated level. \n\nI've been messing around trying to work out why this might be, and I've found that summing the squared errors and then dividing by the number of rows in the unaggregated data gets me much closer to the LB score than dividing by the number of rows in the aggregated data (obviously I take the sqrt after dividing). If your CV and LB scores match up nicely I can be confident it's a problem with my script :-)", "393610": "OMG, don't tell me there's a second problem here .............................\n\nLB is significantly lower that CV of about 0.16 I think, so it's probably not a problem with your script.\n\nJust to reword what you wrote you compte the sum of squared error (at the Visitor level) and then divide by the number of sessions or rows in the original file, is that right ?", "393640": "OK - first time using LaTeX markup in Kaggle, so forgive me if this takes a few edits....\n\nLet n be the total number of visits and m be the number of unique visitors. y is ln1p(sum revenue), and y hat is the estimate.\n\nThe metric should be:\n\n$$\n\\sqrt{\n\\frac{\\sum_{i=1}^{m}(y_i-\\hat y_i)^2}{m}\n}\n$$\n\nBut:\n\n$$\n\\sqrt{\n\\frac{\\sum_{i=1}^{m}(y_i-\\hat y_i)^2}{n}\n}\n$$\n\nGives me closer agreement with the leader board. Could be a total coincidence, but is also a plausible mistake to make. If what I've said is right, then LB scores are out by a factor of root(m/n) - which isn't the end of the world. Of course it could easily be be noise.", "393649": "You're not bad with LaTeX for a first time ;-)\nOk that's what I understood! \n\nThanks for you work Craig ! I let you post yet another topic on the bloody metric ?\n\nPS: And I assure you you'll get my upvote :)", "393659": "I've posted here: https://www.kaggle.com/c/ga-customer-revenue-prediction/discussion/66811\n\nI tried to tag you but unfortunately my tagging skills are not at the same level as my LaTeX skills.\n\nYes the upvote would be appreciated - posting here does make me nervous. The great thing about Kaggle is that it's full of very smart people that I can learn from. The other thing about Kaggle is that it's full of very smart people that can be intellectually intimidating!", "393666": ":facepalm: This is getting ridiculous." }, "source": "meta" }