We found an error in part of our solution that led to the scoring discrepancies people were seeing with sum(log) vs log(sum). This has been corrected and the leaderboard has been rescored. Thanks for your patience, and your help in tracking down the issue. I apologize for the inconvenience. Please feel free to ask questions about it in this thread.
\n\nAll parts of the solution should now match with the log(sum+1) target expressed in the documentation. Please let us know if you notice anything unexpected.
I've noted the issue 11 days ago here and the organizer gave a rather definitive reply saying \"The correct - and expected - way is as noted in the Evaluation and Data pages. I know, because that's precisely how I set the targets. I would recommend you optimize for the expected target.\" To which I simply replied \"Thanks Phil. That's really good information to know.\" -- assuming you know what you were doing, since you precisely set the target. That said, I did find it rather distasteful that I needed to be told \"It should go without saying, too, that lowering your RMSE on the public test set is NOT the same thing as having a model that is generalizing better.\"
\n\nOkay, people make mistakes -- honestly it's better to find out now and fix it, compared to fixing it close to the deadline or not fixing it at all (!). Apologize and move on, which I was ready to do so until I was disappointed again by reading this thread. You attempted to defend a low-level mistake by saying \">99.9% of targets matched the stated target definition\"... As a Kaggler yourself, Phil, surely you know all of us spent hours trying to gain 0.0001. Heck, the #1 rank score is 1.4155 and the #126 rank (medal cutoff) is 1.4432... the difference is only 0.0277 and the 0.01% \"mistake\" changed our scores by 0.2 and completely shaken up the LB.
\n\nLastly, while indeed there is \"a lot of time left on this competition [...] optimize for the stated target and you should do fine\", I didn't participate to just \"do fine\" and I think it is a very valid concern that those of us who started early have wasted 20-40 submissions trying to figure out what happened to the LB. If you imagine a couple of us forming a team later on, we will be penalized by 80-100 \"submissions\" collectively (or it might prevent us from teaming up all together if our total submissions exceed the 5x day limit). Your argument that \"All submissions have been rescored, so you know how your old submissions did.\" does not make any sense to me. If I had been working towards the wrong objective, I absolutely do not need to know how my old submissions did (esp. since they are well above 1.5 and are completely meaningless).
\n\nPersonally, I think the both the manner/tone here and also the remedial action (i.e. relaunching the competition) by inversion at the Airbus Challenge is exemplary. If you allow me to quote him in parts: \"First and foremost, let me express my sincere apologies for the frustration this situation has caused. I'll provide a full explanation of what happened below, but first want to give a high-level summary of the next steps. [...] I'm sure some of you are scratching your heads over how such an obvious mistake could have been made. It was actually a two-part failure, and I happened to be the direct cause of both. [...] To keep this type of error from happening again, we are updating our independent check process to explicitly include competition-specific watchouts, so a second set of eyes is properly equipped to catch failures such as this. [...] Again, I've very sorry for the frustrations arising from this. As always, I welcome and appreciate your feedback (even when it's brutally blunt)!\"... In fact, I am rather curious about Kaggle's \"independent check process\" since this is clearly not the first time leakage/error has happened in recent memory. While you can't undo what you have said, the very least you could do is to relaunch the competition so that all of us can have a fresh start on equal footing.
\n\nEdit 1: Actually, since 98.7% of the revenues are zero in the training set, you might want to tell us how many of the 1.3% that actually have a value was wrong. Quoting 99.9% \"correct\" while probably 99% is zero is either deliberately misleading or simply poor numeracy.
\n\nEdit 2: In view of Phil's response here, I've removed my comment about boycotting this competition, but unfortunately I'll probably be spending a lot less time on it.
", "rawMarkdown": "I've noted the issue 11 days ago [here][1] and the organizer gave a rather definitive reply saying \"The correct - and expected - way is as noted in the Evaluation and Data pages. **I know, because that's precisely how I set the targets.** I would recommend you optimize for the expected target.\" To which I simply replied \"Thanks Phil. That's really good information to know.\" -- assuming you know what you were doing, since you precisely set the target. That said, I did find it rather distasteful that I needed to be told \"It should go without saying, too, that lowering your RMSE on the public test set is NOT the same thing as having a model that is generalizing better.\"\n\nOkay, people make mistakes -- honestly it's better to find out now and fix it, compared to fixing it close to the deadline or not fixing it at all (!). Apologize and move on, which I was ready to do so until I was disappointed again by reading this thread. You attempted to defend a low-level mistake by saying \"**>99.9% of targets matched the stated target definition**\"... As a Kaggler yourself, Phil, surely you know all of us spent hours trying to gain 0.0001. Heck, the #1 rank score is 1.4155 and the #126 rank (medal cutoff) is 1.4432... the difference is only 0.0277 and the 0.01% \"mistake\" changed our scores by 0.2 and completely shaken up the LB.\n\nLastly, while indeed there is \"a lot of time left on this competition [...] optimize for the stated target and you should do fine\", I didn't participate to just \"do fine\" and I think it is a very valid concern that those of us who started early have wasted 20-40 submissions trying to figure out what happened to the LB. If you imagine a couple of us forming a team later on, we will be penalized by 80-100 \"submissions\" collectively (or it might prevent us from teaming up all together if our total submissions exceed the 5x day limit). Your argument that \"All submissions have been rescored, so you know how your old submissions did.\" does not make any sense to me. If I had been working towards the wrong objective, I absolutely do not need to know how my old submissions did (esp. since they are well above 1.5 and are completely meaningless).\n\nPersonally, I think the both the manner/tone [here][2] and also the **remedial action** (i.e. relaunching the competition) by inversion at the Airbus Challenge is exemplary. If you allow me to quote him in parts: \"First and foremost, let me express my sincere apologies for the frustration this situation has caused. I'll provide a full explanation of what happened below, but first want to give a high-level summary of the next steps. [...] I'm sure some of you are scratching your heads over how such an obvious mistake could have been made. It was actually a two-part failure, and I happened to be the direct cause of both. [...] To keep this type of error from happening again, we are updating our independent check process to explicitly include competition-specific watchouts, so a second set of eyes is properly equipped to catch failures such as this. [...] Again, I've very sorry for the frustrations arising from this. As always, I welcome and appreciate your feedback (even when it's brutally blunt)!\"... In fact, I am rather curious about Kaggle's \"independent check process\" since this is clearly not the first time leakage/error has happened in recent memory. **While you can't undo what you have said, the very least you could do is to relaunch the competition so that all of us can have a fresh start on equal footing.**\n\nEdit 1: Actually, since 98.7% of the revenues are zero in the training set, you might want to tell us how many of the 1.3% that actually have a value was wrong. Quoting 99.9% \"correct\" while probably 99% is zero is either deliberately misleading or simply poor numeracy.\n\nEdit 2: In view of Phil's response [here][3], I've removed my comment about boycotting this competition, but unfortunately I'll probably be spending a lot less time on it.\n\n [1]: https://www.kaggle.com/c/ga-customer-revenue-prediction/discussion/65806\n [2]: https://www.kaggle.com/c/airbus-ship-detection/discussion/64388\n [3]: https://www.kaggle.com/c/ga-customer-revenue-prediction/discussion/66737#393828", "votes": 44 }, { "id": 393131, "postDate": "2018-09-24T21:31:44.003Z", "content": "We found an error in part of our solution that led to the scoring discrepancies people were seeing with sum(log) vs log(sum). This has been corrected and the leaderboard has been rescored. Thanks for your patience, and your help in tracking down the issue. I apologize for the inconvenience. Please feel free to ask questions about it in this thread.
\n\nAll parts of the solution should now match with the log(sum+1) target expressed in the documentation. Please let us know if you notice anything unexpected.
Hi Phil,
\n\nCan I ask why you don't just reset the competition? I think this is fair as all the people who submitting to the LB to figure out how to improve are now at a distinct disadvantage when it comes to team mergers etc. from those who start today.
\n\nLook at the LB shakeup to see what an impact the kaggle scoring bug has made.
\n\nEdit: The evaluation page still states \"you must predict the natural log of their total revenue\" which is wrong?
\n\nEdit x2: You can get 1.4307 (21st as it stands) just by doing an arithmetic mean on the top 3 scripts so you should really think of a reset to stop early adopters getting ruined submissions-wise
", "rawMarkdown": "Hi Phil,\n\nCan I ask why you don't just reset the competition? I think this is fair as all the people who submitting to the LB to figure out how to improve are now at a distinct disadvantage when it comes to team mergers etc. from those who start today.\n\nLook at the LB shakeup to see what an impact the kaggle scoring bug has made.\n\nEdit: The evaluation page still states \"you must predict the natural log of their total revenue\" which is wrong?\n\nEdit x2: You can get 1.4307 (21st as it stands) just by doing an arithmetic mean on the top 3 scripts so you should really think of a reset to stop early adopters getting ruined submissions-wise", "votes": 24, "replies": [ { "id": 393353, "postDate": "2018-09-25T08:15:04.573Z", "content": "I agree with you
", "rawMarkdown": "I agree with you", "votes": 3 }, { "id": 393361, "postDate": "2018-09-25T08:36:40.707Z", "content": "As a member of a team that had a lot submissions in this competition I want to add that change in the LB (fixing the error in LB) also requires us check many of our past failed attempts again and teams like us with many submissions are at a disadvantage compared to newcomers that their competition starts today. For example using the previous LB, time based validation did not look promising but after fixing the bug, it seems it is a good candidate for validation.
", "rawMarkdown": "As a member of a team that had a lot submissions in this competition I want to add that change in the LB (fixing the error in LB) also requires us check many of our past failed attempts again and teams like us with many submissions are at a disadvantage compared to newcomers that their competition starts today. For example using the previous LB, time based validation did not look promising but after fixing the bug, it seems it is a good candidate for validation. ", "votes": 5 }, { "id": 393362, "postDate": "2018-09-25T08:39:16.107Z", "content": "I agree, there is a lot of confusion now for teams that were highly involved on the competition
", "rawMarkdown": "I agree, there is a lot of confusion now for teams that were highly involved on the competition", "votes": 3 }, { "id": 393431, "postDate": "2018-09-25T11:05:25.370Z", "content": "Seconded! I dropped some lines of investigations becouse they were performing badly in public lb
", "rawMarkdown": "Seconded! I dropped some lines of investigations becouse they were performing badly in public lb", "votes": 2 }, { "id": 393444, "postDate": "2018-09-25T11:38:40.287Z", "content": "Previously >99.9% of targets matched the stated target definition. Now 100% do. The data and evaluation pages have always described the given target, so no change is required there.
\n\nAll submissions have been rescored, so you know how your old submissions did.
\n\nBasing any decisions on the public LB in any competition is inadvisable. Also, optimizing for anything but the stated target of a competition is really inadvisable.
\n\nIn sum, this was a mistake and again, I am sorry for the inconvenience, but the target has not changed from the consistently stated target. A small number of rows have been brought in line with that target. There is no need to reset the competition.
\n\nThere is a lot of time left on this competition. Optimize for the stated target and you should do fine.
", "rawMarkdown": "Previously >99.9% of targets matched the stated target definition. Now 100% do. The data and evaluation pages have always described the given target, so no change is required there.\n\nAll submissions have been rescored, so you know how your old submissions did.\n\nBasing *any* decisions on the public LB in any competition is inadvisable. Also, optimizing for anything but the stated target of a competition is *really* inadvisable.\n\nIn sum, this was a mistake and again, I am sorry for the inconvenience, but the target has *not* changed from the consistently stated target. A small number of rows have been brought in line with that target. There is no need to reset the competition.\n\nThere is a lot of time left on this competition. Optimize for the stated target and you should do fine.", "votes": -16 }, { "id": 393471, "postDate": "2018-09-25T12:38:20.800Z", "content": "With all due respect Phil, your comments are really callous and show total lack of respect and insight into how Kaggle competition dynamics evolve. Not to mention total lack of ownership of your mistake and any sense of genuine remorse and compunction.
", "rawMarkdown": "With all due respect Phil, your comments are really callous and show total lack of respect and insight into how Kaggle competition dynamics evolve. Not to mention total lack of ownership of **your** mistake and any sense of genuine remorse and compunction.", "votes": 31 }, { "id": 393477, "postDate": "2018-09-25T12:55:25.957Z", "content": "Dear Phil,\nI'm fine with mistakes... They happen. I'm less fine with the tone and content of your replay. Saying \"just go on kids\" doesn't seems an acceptable answer to me.
", "rawMarkdown": "Dear Phil,\nI'm fine with mistakes... They happen. I'm less fine with the tone and content of your replay. Saying \"just go on kids\" doesn't seems an acceptable answer to me.", "votes": 14 }, { "id": 393484, "postDate": "2018-09-25T13:05:37.057Z", "content": "Phil, as an experienced and somewhat successful Kaggler, I beg to differ slightly. While it is true that relying on Public LB is dangerous, the first and crucial task for most experienced competitors is to line up their CV and LB scores, to make sure that their CV is working.
\n\nI was trying really hard to figure out what the hell I was missing, because my CV-LB relationship was all over the place, and that certainly wasted my time and was very discouraging. I was close to quitting this competition, because I do not have a lot of free time these days, and I only compete if I think I have something to learn. I don't have a lot to learn if I have no understanding of why results come out the way they do.
\n\nI'm not sure whether it's necessary to restart the competition or add time to it, but I think your dismissal is a little off the mark.
", "rawMarkdown": "Phil, as an experienced and somewhat successful Kaggler, I beg to differ slightly. While it is true that relying on Public LB is dangerous, the first and crucial task for most experienced competitors is to line up their CV and LB scores, to make sure that their CV is working. \n\nI was trying really hard to figure out what the hell I was missing, because my CV-LB relationship was all over the place, and that certainly wasted my time and was very discouraging. I was close to quitting this competition, because I do not have a lot of free time these days, and I only compete if I think I have something to learn. I don't have a lot to learn if I have no understanding of why results come out the way they do. \n\nI'm not sure whether it's necessary to restart the competition or add time to it, but I think your dismissal is a little off the mark.", "votes": 16 }, { "id": 393485, "postDate": "2018-09-25T13:06:09.067Z", "content": "I agree with Bojan. The two suggestions that 1) the error was insignificant because it affected only 0.1% of the test rows and 2) that Kagglers should always ignore the LB, are frankly bizarre and indicate a deep disrespect for the amount of time that we collectively spent trying to understand the data based on false LB scores. Personally, I don't care whether the competition is reset or not, but Kaggle should have the decency to acknowledge that the error caused a lot of people to waste a lot of time.
", "rawMarkdown": "I agree with Bojan. The two suggestions that 1) the error was insignificant because it affected only 0.1% of the test rows and 2) that Kagglers should always ignore the LB, are frankly bizarre and indicate a deep disrespect for the amount of time that we collectively spent trying to understand the data based on false LB scores. Personally, I don't care whether the competition is reset or not, but Kaggle should have the decency to acknowledge that the error caused a lot of people to waste a lot of time.", "votes": 27 }, { "id": 393499, "postDate": "2018-09-25T13:30:42.437Z", "content": "I respect that @philculliton has come up with the fact that there was a mistake and also he admits that. However, the following points are hard to ignore.
\n\nGiven we have very less non zero test labels, 0.1% of test labels with high transaction values would significantly change the score. Also, sometimes, I prefer playing with the high transaction values, which would give me whether I am underestimating or overestimating based on the public LB scores, so these two are very important aspects in this competitions.
\n\nAlso, in the evaluation section, you can edit that it is log(1+x).
If 0.1% of test rows was insignificant, it wouldn't affect LB as if another Porto competition has just finished. It's so sad many people wasted their submissions in the wrong way and a newcomer can place at top 50 by simply blending best kernels.
", "rawMarkdown": "If 0.1% of test rows was insignificant, it wouldn't affect LB as if another [Porto][1] competition has just finished. It's so sad many people wasted their submissions in the wrong way and a newcomer can place at top 50 by simply blending best kernels.\n\n\n [1]: http://www.kaggle.com/c/porto-seguro-safe-driver-prediction/leaderboard", "votes": 7 }, { "id": 393559, "postDate": "2018-09-25T15:20:44.237Z", "content": "I think I and other Kagglers put much more effort to solve the issue compared to Phil, which is sad. And him ignoring our wasted time is even more sad.
", "rawMarkdown": "I think I and other Kagglers put much more effort to solve the issue compared to Phil, which is sad. And him ignoring our wasted time is even more sad.", "votes": 12 }, { "id": 393828, "postDate": "2018-09-26T00:16:52.530Z", "content": "All,
\n\nThanks for your responses to this thread. I want to apologize. I did not mean to sound dismissive in my earlier response. I attempted to lay out the bare facts of the matter and instead got caught up in defensiveness. It was not fair to you and your efforts, and I am sorry for that. Thank you all for calling me out.
\n\nThis is one of my first competitions as a data scientist at Kaggle, and as you can imagine I have put myself under an incredible amount of pressure to, frankly, not screw up. Due to technical issues with the data and BigQuery, this competition has taken months of work to prepare, and realizing that I have failed after all of that effort is horrifying. I wanted to do a good job for the community that I was so recently a part of, and instead I failed you. I made a mistake, and then compounded it by letting my insecurities get the better of me when responding to your honest questions.
\n\nIn terms of the questions around relaunching, we’ve arrived at the decision not to relaunch due to the length of time still remaining in the competition. I very much understand that this is suboptimal for some of the competitors, but we deliberated the options and tried to reasonably balance a number of considerations.
\n\nAs yilisg and others have noted, I am a Kaggler myself, and I am aware that a great deal of work went into addressing my mistake on your end. I am sorry for minimizing that, and for making members of our community feel unheard and unappreciated. I do appreciate greatly the help that many of you provided in helping me catch my mistake.
Best,\nPhil
", "rawMarkdown": "All,\n\nThanks for your responses to this thread. I want to apologize. I did not mean to sound dismissive in my earlier response. I attempted to lay out the bare facts of the matter and instead got caught up in defensiveness. It was not fair to you and your efforts, and I am sorry for that. Thank you all for calling me out.\n\nThis is one of my first competitions as a data scientist at Kaggle, and as you can imagine I have put myself under an incredible amount of pressure to, frankly, not screw up. Due to technical issues with the data and BigQuery, this competition has taken months of work to prepare, and realizing that I have failed after all of that effort is horrifying. I wanted to do a good job for the community that I was so recently a part of, and instead I failed you. I made a mistake, and then compounded it by letting my insecurities get the better of me when responding to your honest questions.\n\nIn terms of the questions around relaunching, we’ve arrived at the decision not to relaunch due to the length of time still remaining in the competition. I very much understand that this is suboptimal for some of the competitors, but we deliberated the options and tried to reasonably balance a number of considerations.\n\nAs `yilisg` and others have noted, I am a Kaggler myself, and I am aware that a great deal of work went into addressing my mistake on your end. I am sorry for minimizing that, and for making members of our community feel unheard and unappreciated. I do appreciate greatly the help that many of you provided in helping me catch my mistake.\n\nBest,\nPhil", "votes": 55 }, { "id": 402037, "postDate": "2018-10-11T04:25:39.133Z", "content": "@phil Culliton have a look at the leak ...
", "rawMarkdown": "@phil Culliton have a look at the leak ...", "votes": 5 } ] }, { "id": 393738, "postDate": "2018-09-25T20:24:48.523Z", "content": "BTW there seems to be yet another issue with the evaluation metric, I mean the one you use at kaggle to evaluate the LB.
\n\nhttps://www.kaggle.com/c/ga-customer-revenue-prediction/discussion/66811
\n\nWould be good if this could be clarified ASAP.
", "rawMarkdown": "BTW there seems to be yet another issue with the evaluation metric, I mean the one you use at kaggle to evaluate the LB.\n\nhttps://www.kaggle.com/c/ga-customer-revenue-prediction/discussion/66811\n\nWould be good if this could be clarified ASAP.", "votes": 21, "replies": [ { "id": 397477, "postDate": "2018-10-02T14:33:02.417Z", "content": "Was this ever resolved?
", "rawMarkdown": "Was this ever resolved?" }, { "id": 397523, "postDate": "2018-10-02T15:54:10.383Z", "content": "Let's maintain an optimistic outlook :-D
", "rawMarkdown": "Let's maintain an optimistic outlook :-D" } ] }, { "id": 393155, "postDate": "2018-09-24T22:35:15.683Z", "content": "This may be an unpopular opinion, but as someone who has been in the workforce for a decade and has made a mistake or five, I appreciate the candor and the correction.
", "rawMarkdown": "This may be an unpopular opinion, but as someone who has been in the workforce for a decade and has made a mistake or five, I appreciate the candor and the correction.", "votes": 19 }, { "id": 402170, "postDate": "2018-10-11T09:31:10.413Z", "content": "Is it possible to have an official word from the admins as to whether this contest will be continued or reset or abandoned or whatever?
", "rawMarkdown": "Is it possible to have an official word from the admins as to whether this contest will be continued or reset or abandoned or whatever?", "votes": 17, "replies": [ { "id": 402368, "postDate": "2018-10-11T15:19:20.320Z", "content": "All they have to do is ban the use of external data and reset the competition. Simple solution.
", "rawMarkdown": "All they have to do is ban the use of external data and reset the competition. Simple solution. " }, { "id": 402370, "postDate": "2018-10-11T15:23:00.083Z", "content": "It also would be so great to hear words like 'We are aware of the current situation and discussing the next steps with the staff.' So we can stop refreshing the Discussion tab like we do normally for LB.
", "rawMarkdown": "It also would be so great to hear words like 'We are aware of the current situation and discussing the next steps with the staff.' So we can stop refreshing the Discussion tab like we do normally for LB.", "votes": 11 }, { "id": 402465, "postDate": "2018-10-11T19:03:56.327Z", "content": "It has now been 24 hours since Jerome's submission - come on Kaggle - an update just to say you are looking into it shouldn't take this long. :(
", "rawMarkdown": "It has now been 24 hours since Jerome's submission - come on Kaggle - an update just to say you are looking into it shouldn't take this long. :(", "votes": 4 }, { "id": 402507, "postDate": "2018-10-11T20:03:07.307Z", "content": "Maybe, the competition's goal is to find the customer IDs from the google analytics and get their expenses, in order to obtain an RMSE of 0 as fast as you can !!!!! :D :D
", "rawMarkdown": "Maybe, the competition's goal is to find the customer IDs from the google analytics and get their expenses, in order to obtain an RMSE of 0 as fast as you can !!!!! :D :D", "votes": 1 } ] }, { "id": 393298, "postDate": "2018-09-25T06:03:19.460Z", "content": "@philculliton thanks for correcting the evaluation metric and updating the LB, I can now stop bashing my head against the wall and writing new posts on the subject every day !
\n\nWe'll now have to regain insights for the CV/LB relationship.
", "rawMarkdown": "@philculliton thanks for correcting the evaluation metric and updating the LB, I can now stop bashing my head against the wall and writing new posts on the subject every day !\n\nWe'll now have to regain insights for the CV/LB relationship.", "votes": 13, "replies": [ { "id": 393319, "postDate": "2018-09-25T06:57:01.790Z", "content": "Haha! To be honest I am glad that I have been a bit slow in completing my EDA, and have not tried any modeling yet. I can imagine that a lottery with LB and CV scores must have been really frustrating....
", "rawMarkdown": "Haha! To be honest I am glad that I have been a bit slow in completing my EDA, and have not tried any modeling yet. I can imagine that a lottery with LB and CV scores must have been really frustrating....", "votes": 4 } ] }, { "id": 393179, "postDate": "2018-09-24T23:42:12.673Z", "content": "Thank you for fixing this. My sanity was starting to take quite a beating with each submission.
", "rawMarkdown": "Thank you for fixing this. My sanity was starting to take quite a beating with each submission.", "votes": 6, "replies": [ { "id": 393184, "postDate": "2018-09-24T23:56:37.080Z", "content": "Haha, yes, I definitely began repeating \"trust the CV, trust the CV\" to myself
", "rawMarkdown": "Haha, yes, I definitely began repeating \"trust the CV, trust the CV\" to myself", "votes": 1 } ] }, { "id": 394075, "postDate": "2018-09-26T10:35:59.450Z", "content": "Nice
", "rawMarkdown": "Nice", "votes": 3 }, { "id": 393144, "postDate": "2018-09-24T22:13:04.360Z", "content": "Will you reset number of submission?
", "rawMarkdown": "Will you reset number of submission?", "votes": 3, "replies": [ { "id": 393147, "postDate": "2018-09-24T22:22:28.120Z", "content": "No. All old submissions have been rescored, though. The target - now with a small number of rows updated - still reflects the target from the documentation. If you've been building your model based on the documented target, your model should be performing better now.
", "rawMarkdown": "No. All old submissions have been rescored, though. The target - now with a small number of rows updated - still reflects the target from the documentation. If you've been building your model based on the documented target, your model should be performing better now." }, { "id": 393158, "postDate": "2018-09-24T22:38:48.783Z", "content": "Sorry for misleading, I mean “the” number of submissions. We have to consider that when merging teams.
", "rawMarkdown": "Sorry for misleading, I mean “the” number of submissions. We have to consider that when merging teams.", "votes": 3 }, { "id": 393259, "postDate": "2018-09-25T04:36:34.870Z", "content": "Shucks, all them submissions gone to waste ;-).
", "rawMarkdown": "Shucks, all them submissions gone to waste ;-).", "votes": 1 }, { "id": 396803, "postDate": "2018-10-01T11:56:56.467Z", "content": "Followed what you said, then, any update data we need to load? \nOr just modeling with ALL ORIGINAL DATA ? \nthanks for the reply.
", "rawMarkdown": "Followed what you said, then, any update data we need to load? \nOr just modeling with **ALL ORIGINAL DATA** ? \nthanks for the reply." } ] }, { "id": 393133, "postDate": "2018-09-24T21:35:11.417Z", "content": "Log(1 + sum)?
", "rawMarkdown": "Log(1 + sum)?", "votes": 3, "replies": [ { "id": 393137, "postDate": "2018-09-24T21:37:54.640Z", "content": "Correct.
", "rawMarkdown": "Correct." } ] }, { "id": 402521, "postDate": "2018-10-11T20:28:13.517Z", "content": "The lights are on but nobody's home ;-)
", "rawMarkdown": "The lights are on but nobody's home ;-)", "votes": 4 }, { "id": 393575, "postDate": "2018-09-25T16:07:08.383Z", "content": "Maybe you can give some swag to the people that found the error?
", "rawMarkdown": "Maybe you can give some swag to the people that found the error?", "votes": 2 }, { "id": 393603, "postDate": "2018-09-25T17:05:28.807Z", "content": "just one question. do i need to re-download the data for modeling?
", "rawMarkdown": "just one question. do i need to re-download the data for modeling? ", "votes": 1, "replies": [ { "id": 393605, "postDate": "2018-09-25T17:07:14.867Z", "content": "nope
", "rawMarkdown": "nope", "votes": 1 }, { "id": 393608, "postDate": "2018-09-25T17:10:13.013Z", "content": "Thanks :)
", "rawMarkdown": "Thanks :)" } ] }, { "id": 393531, "postDate": "2018-09-25T14:27:26.053Z", "content": "wow
", "rawMarkdown": "wow\n", "votes": 1 }, { "id": 393243, "postDate": "2018-09-25T03:09:05.073Z", "content": "Thanx Phil, sanity is restored!
", "rawMarkdown": "Thanx Phil, sanity is restored!", "votes": 1 }, { "id": 393186, "postDate": "2018-09-24T23:58:22.483Z", "content": "Now that makes sense!!! my only user level validated model on log1p(sum(transactions)) performed better than other models of sum(log(transaction))
", "rawMarkdown": "Now that makes sense!!! my only user level validated model on log1p(sum(transactions)) performed better than other models of sum(log(transaction))", "votes": 1 }, { "id": 393150, "postDate": "2018-09-24T22:27:33.487Z", "rawMarkdown": "", "votes": -1 }, { "id": 402726, "postDate": "2018-10-12T07:48:42.507Z", "content": "Hi Phil Culliton.. A small suggestion you may consider towards the Data Leakage:
\n\nThe leak (to some extent) has been published and right now there is no point to continue the competition with current leakage data. I think host should re publish the data-mart and replace it with the current one and extend the timeline. May change the eval metric or even the problem statement.
\n\nI have seen a leakage trend over the years in different Kaggle competitions that whenever it's minimizing the error (based on MAE, RMSE, MSE, SMAPE etc) i.e. the lower the better eval statistic, the leakage for those competitions are more than wherever there is higher the better ( AUC, Accuracy etc) statistic competition. But it's completely based on problem statement /business objective also.
\n\nSo I may suggest to change the eval metric as well to a higher the better one, it may prevent another leak to some extent with newly published data.
\n\nIn that way all the submitted solutions will be invalid and everyone's effort will be wasted and the competition will be fair for everybody.
", "rawMarkdown": "Hi [Phil Culliton][1].. A small suggestion you may consider towards the Data Leakage:\n\nThe **leak** (to some extent) has been published and right now there is no point to continue the competition with current leakage data. I think host should re publish the data-mart and replace it with the current one and extend the timeline. May change the eval metric or even the problem statement.\n\nI have seen a leakage trend over the years in different Kaggle competitions that whenever it's minimizing the error (based on MAE, RMSE, MSE, SMAPE etc) i.e. the lower the better eval statistic, the leakage for those competitions are more than wherever there is higher the better ( AUC, Accuracy etc) statistic competition. But it's completely based on problem statement /business objective also.\n\nSo I may suggest to change the eval metric as well to a higher the better one, it may prevent another leak to some extent with newly published data.\n\nIn that way all the submitted solutions will be invalid and everyone's effort will be wasted and the competition will be fair for everybody.\n\n\n [1]: https://www.kaggle.com/philculliton" }, { "id": 393272, "postDate": "2018-09-25T05:00:17.767Z", "content": "I thought I was wrong at the beginning. Turns out I'm right. Thanks for the correction.
", "rawMarkdown": "I thought I was wrong at the beginning. Turns out I'm right. Thanks for the correction." }, { "id": 393148, "postDate": "2018-09-24T22:22:39.727Z", "content": "What was the change? Were you actually doing sum(log)? Or did you just have log(sum) AND replace NA with 0? there was already big shakeup and it turns out some kernels that were performing the best, ended up getting lower scores now
We were in fact doing log(sum+1) except on a small portion of the test set that needed to be updated.
\n\nIf you were creating your target in a fashion counter to the documentation - using sum(log) - your submission scores will be worse.
", "rawMarkdown": "We were in fact doing log(sum+1) except on a small portion of the test set that needed to be updated.\n\nIf you were creating your target in a fashion counter to the documentation - using sum(log) - your submission scores will be worse." }, { "id": 393151, "postDate": "2018-09-24T22:28:32.687Z", "content": "Out of curiosity, what was the metric you were performing on that small portion of the test set that needed to be updated? It would seem to me that you would just run a metric compute on all of the test set, as opposed to having it split somehow as you've implied in your post.
", "rawMarkdown": "Out of curiosity, *what* was the metric you were performing on that small portion of the test set that needed to be updated? It would seem to me that you would just run a metric compute on *all* of the test set, as opposed to having it split somehow as you've implied in your post.", "votes": 2 }, { "id": 393167, "postDate": "2018-09-24T23:15:10.717Z", "content": "Basically, I did an early transformation that only affected one part of the data - a mistake in my math prior to the metric compute. The end result was, essentially, sum(log) on a few of the rows, and log(sum+1) on the rest.
", "rawMarkdown": "Basically, I did an early transformation that only affected one part of the data - a mistake in my math prior to the metric compute. The end result was, essentially, sum(log) on a few of the rows, and log(sum+1) on the rest.", "votes": 2 }, { "id": 393253, "postDate": "2018-09-25T04:16:49.033Z", "content": "Sound like doing log1p on visits first, then doing sum on user level, where small portion of data is users with multiple visits.
", "rawMarkdown": "Sound like doing log1p on visits first, then doing sum on user level, where small portion of data is users with multiple visits.", "votes": 2 }, { "id": 396063, "postDate": "2018-09-30T00:45:15.210Z", "content": "Can you please answer this Phil - https://www.kaggle.com/c/ga-customer-revenue-prediction/discussion/67203
", "rawMarkdown": "Can you please answer this Phil - https://www.kaggle.com/c/ga-customer-revenue-prediction/discussion/67203", "votes": -1 } ] }, { "id": 655278, "postDate": "2019-10-22T21:39:28.187Z", "rawMarkdown": "", "isDeleted": true }, { "id": 394205, "postDate": "2018-09-26T14:01:40.410Z", "content": "Thanks
", "rawMarkdown": "Thanks" } ], "comments": [ { "id": 393728, "author_name": "yilisg", "author_url": "", "post_date": "2018-09-25T20:09:24.117000", "content": "I've noted the issue 11 days ago here and the organizer gave a rather definitive reply saying \"The correct - and expected - way is as noted in the Evaluation and Data pages. I know, because that's precisely how I set the targets. I would recommend you optimize for the expected target.\" To which I simply replied \"Thanks Phil. That's really good information to know.\" -- assuming you know what you were doing, since you precisely set the target. That said, I did find it rather distasteful that I needed to be told \"It should go without saying, too, that lowering your RMSE on the public test set is NOT the same thing as having a model that is generalizing better.\"
\n\nOkay, people make mistakes -- honestly it's better to find out now and fix it, compared to fixing it close to the deadline or not fixing it at all (!). Apologize and move on, which I was ready to do so until I was disappointed again by reading this thread. You attempted to defend a low-level mistake by saying \">99.9% of targets matched the stated target definition\"... As a Kaggler yourself, Phil, surely you know all of us spent hours trying to gain 0.0001. Heck, the #1 rank score is 1.4155 and the #126 rank (medal cutoff) is 1.4432... the difference is only 0.0277 and the 0.01% \"mistake\" changed our scores by 0.2 and completely shaken up the LB.
\n\nLastly, while indeed there is \"a lot of time left on this competition [...] optimize for the stated target and you should do fine\", I didn't participate to just \"do fine\" and I think it is a very valid concern that those of us who started early have wasted 20-40 submissions trying to figure out what happened to the LB. If you imagine a couple of us forming a team later on, we will be penalized by 80-100 \"submissions\" collectively (or it might prevent us from teaming up all together if our total submissions exceed the 5x day limit). Your argument that \"All submissions have been rescored, so you know how your old submissions did.\" does not make any sense to me. If I had been working towards the wrong objective, I absolutely do not need to know how my old submissions did (esp. since they are well above 1.5 and are completely meaningless).
\n\nPersonally, I think the both the manner/tone here and also the remedial action (i.e. relaunching the competition) by inversion at the Airbus Challenge is exemplary. If you allow me to quote him in parts: \"First and foremost, let me express my sincere apologies for the frustration this situation has caused. I'll provide a full explanation of what happened below, but first want to give a high-level summary of the next steps. [...] I'm sure some of you are scratching your heads over how such an obvious mistake could have been made. It was actually a two-part failure, and I happened to be the direct cause of both. [...] To keep this type of error from happening again, we are updating our independent check process to explicitly include competition-specific watchouts, so a second set of eyes is properly equipped to catch failures such as this. [...] Again, I've very sorry for the frustrations arising from this. As always, I welcome and appreciate your feedback (even when it's brutally blunt)!\"... In fact, I am rather curious about Kaggle's \"independent check process\" since this is clearly not the first time leakage/error has happened in recent memory. While you can't undo what you have said, the very least you could do is to relaunch the competition so that all of us can have a fresh start on equal footing.
\n\nEdit 1: Actually, since 98.7% of the revenues are zero in the training set, you might want to tell us how many of the 1.3% that actually have a value was wrong. Quoting 99.9% \"correct\" while probably 99% is zero is either deliberately misleading or simply poor numeracy.
\n\nEdit 2: In view of Phil's response here, I've removed my comment about boycotting this competition, but unfortunately I'll probably be spending a lot less time on it.
", "votes": 44, "replies": [] }, { "id": 393348, "author_name": "Scirpus", "author_url": "", "post_date": "2018-09-25T08:04:58.783000", "content": "Hi Phil,
\n\nCan I ask why you don't just reset the competition? I think this is fair as all the people who submitting to the LB to figure out how to improve are now at a distinct disadvantage when it comes to team mergers etc. from those who start today.
\n\nLook at the LB shakeup to see what an impact the kaggle scoring bug has made.
\n\nEdit: The evaluation page still states \"you must predict the natural log of their total revenue\" which is wrong?
\n\nEdit x2: You can get 1.4307 (21st as it stands) just by doing an arithmetic mean on the top 3 scripts so you should really think of a reset to stop early adopters getting ruined submissions-wise
", "votes": 24, "replies": [ { "id": 393353, "author_name": "Marcus Lin", "author_url": "", "post_date": "2018-09-25T08:15:04.573000", "content": "I agree with you
", "votes": 3, "replies": [] }, { "id": 393361, "author_name": "Kain", "author_url": "", "post_date": "2018-09-25T08:36:40.707000", "content": "As a member of a team that had a lot submissions in this competition I want to add that change in the LB (fixing the error in LB) also requires us check many of our past failed attempts again and teams like us with many submissions are at a disadvantage compared to newcomers that their competition starts today. For example using the previous LB, time based validation did not look promising but after fixing the bug, it seems it is a good candidate for validation.
", "votes": 5, "replies": [] }, { "id": 393362, "author_name": "Dimosthenis Karaflos", "author_url": "", "post_date": "2018-09-25T08:39:16.107000", "content": "I agree, there is a lot of confusion now for teams that were highly involved on the competition
", "votes": 3, "replies": [] }, { "id": 393431, "author_name": "MT", "author_url": "", "post_date": "2018-09-25T11:05:25.370000", "content": "Seconded! I dropped some lines of investigations becouse they were performing badly in public lb
", "votes": 2, "replies": [] }, { "id": 393444, "author_name": "Phil Culliton", "author_url": "", "post_date": "2018-09-25T11:38:40.287000", "content": "Previously >99.9% of targets matched the stated target definition. Now 100% do. The data and evaluation pages have always described the given target, so no change is required there.
\n\nAll submissions have been rescored, so you know how your old submissions did.
\n\nBasing any decisions on the public LB in any competition is inadvisable. Also, optimizing for anything but the stated target of a competition is really inadvisable.
\n\nIn sum, this was a mistake and again, I am sorry for the inconvenience, but the target has not changed from the consistently stated target. A small number of rows have been brought in line with that target. There is no need to reset the competition.
\n\nThere is a lot of time left on this competition. Optimize for the stated target and you should do fine.
", "votes": -16, "replies": [] }, { "id": 393471, "author_name": "Bojan Tunguz", "author_url": "", "post_date": "2018-09-25T12:38:20.800000", "content": "With all due respect Phil, your comments are really callous and show total lack of respect and insight into how Kaggle competition dynamics evolve. Not to mention total lack of ownership of your mistake and any sense of genuine remorse and compunction.
", "votes": 31, "replies": [] }, { "id": 393477, "author_name": "MT", "author_url": "", "post_date": "2018-09-25T12:55:25.957000", "content": "Dear Phil,\nI'm fine with mistakes... They happen. I'm less fine with the tone and content of your replay. Saying \"just go on kids\" doesn't seems an acceptable answer to me.
", "votes": 14, "replies": [] }, { "id": 393484, "author_name": "Dmitriy Guller", "author_url": "", "post_date": "2018-09-25T13:05:37.057000", "content": "Phil, as an experienced and somewhat successful Kaggler, I beg to differ slightly. While it is true that relying on Public LB is dangerous, the first and crucial task for most experienced competitors is to line up their CV and LB scores, to make sure that their CV is working.
\n\nI was trying really hard to figure out what the hell I was missing, because my CV-LB relationship was all over the place, and that certainly wasted my time and was very discouraging. I was close to quitting this competition, because I do not have a lot of free time these days, and I only compete if I think I have something to learn. I don't have a lot to learn if I have no understanding of why results come out the way they do.
\n\nI'm not sure whether it's necessary to restart the competition or add time to it, but I think your dismissal is a little off the mark.
", "votes": 16, "replies": [] }, { "id": 393485, "author_name": "Silogram", "author_url": "", "post_date": "2018-09-25T13:06:09.067000", "content": "I agree with Bojan. The two suggestions that 1) the error was insignificant because it affected only 0.1% of the test rows and 2) that Kagglers should always ignore the LB, are frankly bizarre and indicate a deep disrespect for the amount of time that we collectively spent trying to understand the data based on false LB scores. Personally, I don't care whether the competition is reset or not, but Kaggle should have the decency to acknowledge that the error caused a lot of people to waste a lot of time.
", "votes": 27, "replies": [] }, { "id": 393499, "author_name": "Indranil Bhattacharya", "author_url": "", "post_date": "2018-09-25T13:30:42.437000", "content": "I respect that @philculliton has come up with the fact that there was a mistake and also he admits that. However, the following points are hard to ignore.
\n\nGiven we have very less non zero test labels, 0.1% of test labels with high transaction values would significantly change the score. Also, sometimes, I prefer playing with the high transaction values, which would give me whether I am underestimating or overestimating based on the public LB scores, so these two are very important aspects in this competitions.
\n\nAlso, in the evaluation section, you can edit that it is log(1+x).
If 0.1% of test rows was insignificant, it wouldn't affect LB as if another Porto competition has just finished. It's so sad many people wasted their submissions in the wrong way and a newcomer can place at top 50 by simply blending best kernels.
", "votes": 7, "replies": [] }, { "id": 393559, "author_name": "Ahmet Erdem", "author_url": "", "post_date": "2018-09-25T15:20:44.237000", "content": "I think I and other Kagglers put much more effort to solve the issue compared to Phil, which is sad. And him ignoring our wasted time is even more sad.
", "votes": 12, "replies": [] }, { "id": 393828, "author_name": "Phil Culliton", "author_url": "", "post_date": "2018-09-26T00:16:52.530000", "content": "All,
\n\nThanks for your responses to this thread. I want to apologize. I did not mean to sound dismissive in my earlier response. I attempted to lay out the bare facts of the matter and instead got caught up in defensiveness. It was not fair to you and your efforts, and I am sorry for that. Thank you all for calling me out.
\n\nThis is one of my first competitions as a data scientist at Kaggle, and as you can imagine I have put myself under an incredible amount of pressure to, frankly, not screw up. Due to technical issues with the data and BigQuery, this competition has taken months of work to prepare, and realizing that I have failed after all of that effort is horrifying. I wanted to do a good job for the community that I was so recently a part of, and instead I failed you. I made a mistake, and then compounded it by letting my insecurities get the better of me when responding to your honest questions.
\n\nIn terms of the questions around relaunching, we’ve arrived at the decision not to relaunch due to the length of time still remaining in the competition. I very much understand that this is suboptimal for some of the competitors, but we deliberated the options and tried to reasonably balance a number of considerations.
\n\nAs yilisg and others have noted, I am a Kaggler myself, and I am aware that a great deal of work went into addressing my mistake on your end. I am sorry for minimizing that, and for making members of our community feel unheard and unappreciated. I do appreciate greatly the help that many of you provided in helping me catch my mistake.
Best,\nPhil
", "votes": 55, "replies": [] }, { "id": 402037, "author_name": "Xavier Bourret Sicotte", "author_url": "", "post_date": "2018-10-11T04:25:39.133000", "content": "@phil Culliton have a look at the leak ...
", "votes": 5, "replies": [] } ] }, { "id": 393738, "author_name": "olivier", "author_url": "", "post_date": "2018-09-25T20:24:48.523000", "content": "BTW there seems to be yet another issue with the evaluation metric, I mean the one you use at kaggle to evaluate the LB.
\n\nhttps://www.kaggle.com/c/ga-customer-revenue-prediction/discussion/66811
\n\nWould be good if this could be clarified ASAP.
", "votes": 21, "replies": [ { "id": 397477, "author_name": "Allen", "author_url": "", "post_date": "2018-10-02T14:33:02.417000", "content": "Was this ever resolved?
", "votes": 0, "replies": [] }, { "id": 397523, "author_name": "MT", "author_url": "", "post_date": "2018-10-02T15:54:10.383000", "content": "Let's maintain an optimistic outlook :-D
", "votes": 0, "replies": [] } ] }, { "id": 393155, "author_name": "Nick", "author_url": "", "post_date": "2018-09-24T22:35:15.683000", "content": "This may be an unpopular opinion, but as someone who has been in the workforce for a decade and has made a mistake or five, I appreciate the candor and the correction.
", "votes": 19, "replies": [] }, { "id": 402170, "author_name": "Evdilos_Ikaria", "author_url": "", "post_date": "2018-10-11T09:31:10.413000", "content": "Is it possible to have an official word from the admins as to whether this contest will be continued or reset or abandoned or whatever?
", "votes": 17, "replies": [ { "id": 402368, "author_name": "Boris Marjanovic", "author_url": "", "post_date": "2018-10-11T15:19:20.320000", "content": "All they have to do is ban the use of external data and reset the competition. Simple solution.
", "votes": 0, "replies": [] }, { "id": 402370, "author_name": "Fatih Öztürk", "author_url": "", "post_date": "2018-10-11T15:23:00.083000", "content": "It also would be so great to hear words like 'We are aware of the current situation and discussing the next steps with the staff.' So we can stop refreshing the Discussion tab like we do normally for LB.
", "votes": 11, "replies": [] }, { "id": 402465, "author_name": "Scirpus", "author_url": "", "post_date": "2018-10-11T19:03:56.327000", "content": "It has now been 24 hours since Jerome's submission - come on Kaggle - an update just to say you are looking into it shouldn't take this long. :(
", "votes": 4, "replies": [] }, { "id": 402507, "author_name": "Dimosthenis Karaflos", "author_url": "", "post_date": "2018-10-11T20:03:07.307000", "content": "Maybe, the competition's goal is to find the customer IDs from the google analytics and get their expenses, in order to obtain an RMSE of 0 as fast as you can !!!!! :D :D
", "votes": 1, "replies": [] } ] }, { "id": 393298, "author_name": "olivier", "author_url": "", "post_date": "2018-09-25T06:03:19.460000", "content": "@philculliton thanks for correcting the evaluation metric and updating the LB, I can now stop bashing my head against the wall and writing new posts on the subject every day !
\n\nWe'll now have to regain insights for the CV/LB relationship.
", "votes": 13, "replies": [ { "id": 393319, "author_name": "Erik Bruin", "author_url": "", "post_date": "2018-09-25T06:57:01.790000", "content": "Haha! To be honest I am glad that I have been a bit slow in completing my EDA, and have not tried any modeling yet. I can imagine that a lottery with LB and CV scores must have been really frustrating....
", "votes": 4, "replies": [] } ] }, { "id": 393179, "author_name": "Dmitriy Guller", "author_url": "", "post_date": "2018-09-24T23:42:12.673000", "content": "Thank you for fixing this. My sanity was starting to take quite a beating with each submission.
", "votes": 6, "replies": [ { "id": 393184, "author_name": "Nick", "author_url": "", "post_date": "2018-09-24T23:56:37.080000", "content": "Haha, yes, I definitely began repeating \"trust the CV, trust the CV\" to myself
", "votes": 1, "replies": [] } ] }, { "id": 394075, "author_name": "Syam kakarla", "author_url": "", "post_date": "2018-09-26T10:35:59.450000", "content": "Nice
", "votes": 3, "replies": [] }, { "id": 393144, "author_name": "ONODERA", "author_url": "", "post_date": "2018-09-24T22:13:04.360000", "content": "Will you reset number of submission?
", "votes": 3, "replies": [ { "id": 393147, "author_name": "Phil Culliton", "author_url": "", "post_date": "2018-09-24T22:22:28.120000", "content": "No. All old submissions have been rescored, though. The target - now with a small number of rows updated - still reflects the target from the documentation. If you've been building your model based on the documented target, your model should be performing better now.
", "votes": 0, "replies": [] }, { "id": 393158, "author_name": "ONODERA", "author_url": "", "post_date": "2018-09-24T22:38:48.783000", "content": "Sorry for misleading, I mean “the” number of submissions. We have to consider that when merging teams.
", "votes": 3, "replies": [] }, { "id": 393259, "author_name": "Kain", "author_url": "", "post_date": "2018-09-25T04:36:34.870000", "content": "Shucks, all them submissions gone to waste ;-).
", "votes": 1, "replies": [] }, { "id": 396803, "author_name": "YLT0609", "author_url": "", "post_date": "2018-10-01T11:56:56.467000", "content": "Followed what you said, then, any update data we need to load? \nOr just modeling with ALL ORIGINAL DATA ? \nthanks for the reply.
", "votes": 0, "replies": [] } ] }, { "id": 393133, "author_name": "Alexander Firsov", "author_url": "", "post_date": "2018-09-24T21:35:11.417000", "content": "Log(1 + sum)?
", "votes": 3, "replies": [ { "id": 393137, "author_name": "Phil Culliton", "author_url": "", "post_date": "2018-09-24T21:37:54.640000", "content": "Correct.
", "votes": 0, "replies": [] } ] }, { "id": 402521, "author_name": "Kain", "author_url": "", "post_date": "2018-10-11T20:28:13.517000", "content": "The lights are on but nobody's home ;-)
", "votes": 4, "replies": [] }, { "id": 393575, "author_name": "Allen", "author_url": "", "post_date": "2018-09-25T16:07:08.383000", "content": "Maybe you can give some swag to the people that found the error?
", "votes": 2, "replies": [] }, { "id": 393603, "author_name": "Manoj", "author_url": "", "post_date": "2018-09-25T17:05:28.807000", "content": "just one question. do i need to re-download the data for modeling?
", "votes": 1, "replies": [ { "id": 393605, "author_name": "Kain", "author_url": "", "post_date": "2018-09-25T17:07:14.867000", "content": "nope
", "votes": 1, "replies": [] }, { "id": 393608, "author_name": "Manoj", "author_url": "", "post_date": "2018-09-25T17:10:13.013000", "content": "Thanks :)
", "votes": 0, "replies": [] } ] }, { "id": 393531, "author_name": "Jason", "author_url": "", "post_date": "2018-09-25T14:27:26.053000", "content": "wow
", "votes": 1, "replies": [] }, { "id": 393243, "author_name": "Kain", "author_url": "", "post_date": "2018-09-25T03:09:05.073000", "content": "Thanx Phil, sanity is restored!
", "votes": 1, "replies": [] }, { "id": 393186, "author_name": "Dimosthenis Karaflos", "author_url": "", "post_date": "2018-09-24T23:58:22.483000", "content": "Now that makes sense!!! my only user level validated model on log1p(sum(transactions)) performed better than other models of sum(log(transaction))
", "votes": 1, "replies": [] }, { "id": 393150, "author_name": "randommodnar", "author_url": "", "post_date": "2018-09-24T22:27:33.487000", "content": "", "votes": -1, "replies": [] }, { "id": 402726, "author_name": "Arindam", "author_url": "", "post_date": "2018-10-12T07:48:42.507000", "content": "Hi Phil Culliton.. A small suggestion you may consider towards the Data Leakage:
\n\nThe leak (to some extent) has been published and right now there is no point to continue the competition with current leakage data. I think host should re publish the data-mart and replace it with the current one and extend the timeline. May change the eval metric or even the problem statement.
\n\nI have seen a leakage trend over the years in different Kaggle competitions that whenever it's minimizing the error (based on MAE, RMSE, MSE, SMAPE etc) i.e. the lower the better eval statistic, the leakage for those competitions are more than wherever there is higher the better ( AUC, Accuracy etc) statistic competition. But it's completely based on problem statement /business objective also.
\n\nSo I may suggest to change the eval metric as well to a higher the better one, it may prevent another leak to some extent with newly published data.
\n\nIn that way all the submitted solutions will be invalid and everyone's effort will be wasted and the competition will be fair for everybody.
", "votes": 0, "replies": [] }, { "id": 393272, "author_name": "dexter6855", "author_url": "", "post_date": "2018-09-25T05:00:17.767000", "content": "I thought I was wrong at the beginning. Turns out I'm right. Thanks for the correction.
", "votes": 0, "replies": [] }, { "id": 393148, "author_name": "CoreyJamesLevinson", "author_url": "", "post_date": "2018-09-24T22:22:39.727000", "content": "What was the change? Were you actually doing sum(log)? Or did you just have log(sum) AND replace NA with 0? there was already big shakeup and it turns out some kernels that were performing the best, ended up getting lower scores now
We were in fact doing log(sum+1) except on a small portion of the test set that needed to be updated.
\n\nIf you were creating your target in a fashion counter to the documentation - using sum(log) - your submission scores will be worse.
", "votes": 0, "replies": [] }, { "id": 393151, "author_name": "CoreyJamesLevinson", "author_url": "", "post_date": "2018-09-24T22:28:32.687000", "content": "Out of curiosity, what was the metric you were performing on that small portion of the test set that needed to be updated? It would seem to me that you would just run a metric compute on all of the test set, as opposed to having it split somehow as you've implied in your post.
", "votes": 2, "replies": [] }, { "id": 393167, "author_name": "Phil Culliton", "author_url": "", "post_date": "2018-09-24T23:15:10.717000", "content": "Basically, I did an early transformation that only affected one part of the data - a mistake in my math prior to the metric compute. The end result was, essentially, sum(log) on a few of the rows, and log(sum+1) on the rest.
", "votes": 2, "replies": [] }, { "id": 393253, "author_name": "Alexander Firsov", "author_url": "", "post_date": "2018-09-25T04:16:49.033000", "content": "Sound like doing log1p on visits first, then doing sum on user level, where small portion of data is users with multiple visits.
", "votes": 2, "replies": [] }, { "id": 396063, "author_name": "Neeraj", "author_url": "", "post_date": "2018-09-30T00:45:15.210000", "content": "Can you please answer this Phil - https://www.kaggle.com/c/ga-customer-revenue-prediction/discussion/67203
", "votes": -1, "replies": [] } ] }, { "id": 655278, "author_name": "", "author_url": "", "post_date": "2019-10-22T21:39:28.187000", "content": "", "votes": 0, "replies": [] }, { "id": 394205, "author_name": "Andrey Anisimov", "author_url": "", "post_date": "2018-09-26T14:01:40.410000", "content": "Thanks
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "393728": "I've noted the issue 11 days ago [here][1] and the organizer gave a rather definitive reply saying \"The correct - and expected - way is as noted in the Evaluation and Data pages. **I know, because that's precisely how I set the targets.** I would recommend you optimize for the expected target.\" To which I simply replied \"Thanks Phil. That's really good information to know.\" -- assuming you know what you were doing, since you precisely set the target. That said, I did find it rather distasteful that I needed to be told \"It should go without saying, too, that lowering your RMSE on the public test set is NOT the same thing as having a model that is generalizing better.\"\n\nOkay, people make mistakes -- honestly it's better to find out now and fix it, compared to fixing it close to the deadline or not fixing it at all (!). Apologize and move on, which I was ready to do so until I was disappointed again by reading this thread. You attempted to defend a low-level mistake by saying \"**>99.9% of targets matched the stated target definition**\"... As a Kaggler yourself, Phil, surely you know all of us spent hours trying to gain 0.0001. Heck, the #1 rank score is 1.4155 and the #126 rank (medal cutoff) is 1.4432... the difference is only 0.0277 and the 0.01% \"mistake\" changed our scores by 0.2 and completely shaken up the LB.\n\nLastly, while indeed there is \"a lot of time left on this competition [...] optimize for the stated target and you should do fine\", I didn't participate to just \"do fine\" and I think it is a very valid concern that those of us who started early have wasted 20-40 submissions trying to figure out what happened to the LB. If you imagine a couple of us forming a team later on, we will be penalized by 80-100 \"submissions\" collectively (or it might prevent us from teaming up all together if our total submissions exceed the 5x day limit). Your argument that \"All submissions have been rescored, so you know how your old submissions did.\" does not make any sense to me. If I had been working towards the wrong objective, I absolutely do not need to know how my old submissions did (esp. since they are well above 1.5 and are completely meaningless).\n\nPersonally, I think the both the manner/tone [here][2] and also the **remedial action** (i.e. relaunching the competition) by inversion at the Airbus Challenge is exemplary. If you allow me to quote him in parts: \"First and foremost, let me express my sincere apologies for the frustration this situation has caused. I'll provide a full explanation of what happened below, but first want to give a high-level summary of the next steps. [...] I'm sure some of you are scratching your heads over how such an obvious mistake could have been made. It was actually a two-part failure, and I happened to be the direct cause of both. [...] To keep this type of error from happening again, we are updating our independent check process to explicitly include competition-specific watchouts, so a second set of eyes is properly equipped to catch failures such as this. [...] Again, I've very sorry for the frustrations arising from this. As always, I welcome and appreciate your feedback (even when it's brutally blunt)!\"... In fact, I am rather curious about Kaggle's \"independent check process\" since this is clearly not the first time leakage/error has happened in recent memory. **While you can't undo what you have said, the very least you could do is to relaunch the competition so that all of us can have a fresh start on equal footing.**\n\nEdit 1: Actually, since 98.7% of the revenues are zero in the training set, you might want to tell us how many of the 1.3% that actually have a value was wrong. Quoting 99.9% \"correct\" while probably 99% is zero is either deliberately misleading or simply poor numeracy.\n\nEdit 2: In view of Phil's response [here][3], I've removed my comment about boycotting this competition, but unfortunately I'll probably be spending a lot less time on it.\n\n [1]: https://www.kaggle.com/c/ga-customer-revenue-prediction/discussion/65806\n [2]: https://www.kaggle.com/c/airbus-ship-detection/discussion/64388\n [3]: https://www.kaggle.com/c/ga-customer-revenue-prediction/discussion/66737#393828", "393131": "We found an error in part of our solution that led to the scoring discrepancies people were seeing with sum(log) vs log(sum). This has been corrected and the leaderboard has been rescored. Thanks for your patience, and your help in tracking down the issue. I apologize for the inconvenience. Please feel free to ask questions about it in this thread.\n\nAll parts of the solution should now match with the `log(sum+1)` target expressed in the documentation. Please let us know if you notice anything unexpected.", "393348": "Hi Phil,\n\nCan I ask why you don't just reset the competition? I think this is fair as all the people who submitting to the LB to figure out how to improve are now at a distinct disadvantage when it comes to team mergers etc. from those who start today.\n\nLook at the LB shakeup to see what an impact the kaggle scoring bug has made.\n\nEdit: The evaluation page still states \"you must predict the natural log of their total revenue\" which is wrong?\n\nEdit x2: You can get 1.4307 (21st as it stands) just by doing an arithmetic mean on the top 3 scripts so you should really think of a reset to stop early adopters getting ruined submissions-wise", "393738": "BTW there seems to be yet another issue with the evaluation metric, I mean the one you use at kaggle to evaluate the LB.\n\nhttps://www.kaggle.com/c/ga-customer-revenue-prediction/discussion/66811\n\nWould be good if this could be clarified ASAP.", "393155": "This may be an unpopular opinion, but as someone who has been in the workforce for a decade and has made a mistake or five, I appreciate the candor and the correction.", "402170": "Is it possible to have an official word from the admins as to whether this contest will be continued or reset or abandoned or whatever?", "393298": "@philculliton thanks for correcting the evaluation metric and updating the LB, I can now stop bashing my head against the wall and writing new posts on the subject every day !\n\nWe'll now have to regain insights for the CV/LB relationship.", "393179": "Thank you for fixing this. My sanity was starting to take quite a beating with each submission.", "394075": "Nice", "393144": "Will you reset number of submission?", "393133": "Log(1 + sum)?", "402521": "The lights are on but nobody's home ;-)", "393575": "Maybe you can give some swag to the people that found the error?", "393603": "just one question. do i need to re-download the data for modeling? ", "393531": "wow\n", "393243": "Thanx Phil, sanity is restored!", "393186": "Now that makes sense!!! my only user level validated model on log1p(sum(transactions)) performed better than other models of sum(log(transaction))", "393150": "", "402726": "Hi [Phil Culliton][1].. A small suggestion you may consider towards the Data Leakage:\n\nThe **leak** (to some extent) has been published and right now there is no point to continue the competition with current leakage data. I think host should re publish the data-mart and replace it with the current one and extend the timeline. May change the eval metric or even the problem statement.\n\nI have seen a leakage trend over the years in different Kaggle competitions that whenever it's minimizing the error (based on MAE, RMSE, MSE, SMAPE etc) i.e. the lower the better eval statistic, the leakage for those competitions are more than wherever there is higher the better ( AUC, Accuracy etc) statistic competition. But it's completely based on problem statement /business objective also.\n\nSo I may suggest to change the eval metric as well to a higher the better one, it may prevent another leak to some extent with newly published data.\n\nIn that way all the submitted solutions will be invalid and everyone's effort will be wasted and the competition will be fair for everybody.\n\n\n [1]: https://www.kaggle.com/philculliton", "393272": "I thought I was wrong at the beginning. Turns out I'm right. Thanks for the correction.", "393148": "What was the change? Were you actually doing `sum(log)`? Or did you just have `log(sum) AND replace NA with 0`? there was already big shakeup and it turns out some kernels that were performing the best, ended up getting lower scores now", "655278": "", "394205": "Thanks" } }