Now, I analyzed final-all-scores.csv and learned some part of the reason of my shake down. (Public 1st -> Private 497th?😥)
\n\nOther than site 0,1,2,(3?),4,15, some buildings in other sites seem to be excluded in private.\nI think my good public score relied too much on a few \"difficult and influential\" time-series for example building_id=1250 and meter=2 those are not in private.\nIf any other ideas of the reason of your/our shake, please let me know.
\n\nAnyway, congratulations on winners!
", "messages": [ { "id": "704820", "postDate": "12/28/2019 03:39:07", "content": "Now, I analyzed final-all-scores.csv and learned some part of the reason of my shake down. (Public 1st -> Private 497th?😥)
\n\nOther than site 0,1,2,(3?),4,15, some buildings in other sites seem to be excluded in private.\nI think my good public score relied too much on a few \"difficult and influential\" time-series for example building_id=1250 and meter=2 those are not in private.\nIf any other ideas of the reason of your/our shake, please let me know.
\n\nAnyway, congratulations on winners!
", "rawMarkdown": "Now, I analyzed final-all-scores.csv and learned some part of the reason of my shake down. (Public 1st -> Private 497th?😥)\n\nOther than site 0,1,2,(3?),4,15, some buildings in other sites seem to be excluded in private.\nI think my good public score relied too much on a few \"difficult and influential\" time-series for example building_id=1250 and meter=2 those are not in private.\nIf any other ideas of the reason of your/our shake, please let me know.\n\nAnyway, congratulations on winners!", "votes": null }, { "id": "704858", "postDate": "12/28/2019 05:11:56", "content": "That was unlucky. but you did great job regardless of result itself! ;-) I choose one of my submission with 4 models (my best, middle, middle, somewhat bad) blending in which I set coefficients aiming at trying to maximize the correlations in meter-wise, and that was my best private score. That might make my private score not too low also not too high.
", "rawMarkdown": "That was unlucky. but you did great job regardless of result itself! ;-) I choose one of my submission with 4 models (my best, middle, middle, somewhat bad) blending in which I set coefficients aiming at trying to maximize the correlations in meter-wise, and that was my best private score. That might make my private score not too low also not too high.", "votes": null }, { "id": "704879", "postDate": "12/28/2019 05:51:44", "content": "I got shake down to Rank120 from top30 probably because I use leakage. I accept the final score, and congrats to all winners too.
", "rawMarkdown": "I got shake down to Rank120 from top30 probably because I use leakage. I accept the final score, and congrats to all winners too.", "votes": null }, { "id": "704963", "postDate": "12/28/2019 08:10:16", "content": "I was submitting after the leak data merge and made a mistake choosing the final submission. So I guess that is because you were evaluating the algorithm with the public score after the leak data merge. For example, in a private test, this divergence is likely to occur when weather features are more important than building features. However, I think that the evaluation of the algorithm was difficult for everyone in this competition situation. This is the opinion imagined from my results.
", "rawMarkdown": "I was submitting after the leak data merge and made a mistake choosing the final submission. So I guess that is because you were evaluating the algorithm with the public score after the leak data merge. For example, in a private test, this divergence is likely to occur when weather features are more important than building features. However, I think that the evaluation of the algorithm was difficult for everyone in this competition situation. This is the opinion imagined from my results.", "votes": null }, { "id": "705054", "postDate": "12/28/2019 11:04:30", "content": "Thanks for your comment. I agree with your idea that weather features are more important than building features in private LB! But I mainly used 2018 (private) for leak validation and that validation score was correlated with public LB.
", "rawMarkdown": "Thanks for your comment. I agree with your idea that weather features are more important than building features in private LB! But I mainly used 2018 (private) for leak validation and that validation score was correlated with public LB.", "votes": null }, { "id": "705116", "postDate": "12/28/2019 13:06:12", "content": "Our team has the worst drop after you (Public 9th - Private 378th). I have no idea what caused it. We didn't use leaks for training or validation. I hate this competition so much :D
", "rawMarkdown": "Our team has the worst drop after you (Public 9th - Private 378th). I have no idea what caused it. We didn't use leaks for training or validation. I hate this competition so much :D", "votes": null }, { "id": "705129", "postDate": "12/28/2019 13:40:39", "content": "I am very glad to get comment from you. We are in similar situation.(Outlier removal strategy and shake down) Maybe, main reason is not related to leak but the restricted number of buildings in private LB.\nOur outlier removal strategy may work great in some restricted buildings (not in site 0,1,2,3,4,15!) that are somehow removed from private LB. But even considering the above reason, I estimated you and I are in silver zone at worst. Though some terrible things happened, I learned a lot from this competition😄
", "rawMarkdown": "I am very glad to get comment from you. We are in similar situation.(Outlier removal strategy and shake down) Maybe, main reason is not related to leak but the restricted number of buildings in private LB.\nOur outlier removal strategy may work great in some restricted buildings (not in site 0,1,2,3,4,15!) that are somehow removed from private LB. But even considering the above reason, I estimated you and I are in silver zone at worst. Though some terrible things happened, I learned a lot from this competition😄", "votes": null }, { "id": "705153", "postDate": "12/28/2019 14:39:06", "content": "if i understand correctly you also relied heavily on removing outliers. maybe that caused you to overfit to the public lb part
", "rawMarkdown": "if i understand correctly you also relied heavily on removing outliers. maybe that caused you to overfit to the public lb part", "votes": null }, { "id": "705232", "postDate": "12/28/2019 17:15:37", "content": "Reason of shake up or down is very clear - whether you used leakage or not, or how much leakage did you use for both your submission and model training. It’s worth to dive deep why it shakedown, if you haven’t used any leak data.
", "rawMarkdown": "Reason of shake up or down is very clear - whether you used leakage or not, or how much leakage did you use for both your submission and model training. It’s worth to dive deep why it shakedown, if you haven’t used any leak data.", "votes": null }, { "id": "705408", "postDate": "12/28/2019 22:30:13", "content": "@gunesevitan I looked at your 1.048 kernel and found out your prediction on building #778 is low. I guess this is one of the reasons your team shaked down. The meter reading of this building is huge so it can be dominant on the final rmse.
", "rawMarkdown": "gunesevitan I looked at your 1.048 kernel and found out your prediction on building #778 is low. I guess this is one of the reasons your team shaked down. The meter reading of this building is huge so it can be dominant on the final rmse.", "votes": null }, { "id": "705477", "postDate": "12/29/2019 00:33:39", "content": "I agree with you on shake up: Not replacing leakage leads to shake up.\nI disagree with you on big shake down: I think I do not use leakage heavily comparing with other teams. And @gunesevitan do not use leakage for validation.
", "rawMarkdown": "I agree with you on shake up: Not replacing leakage leads to shake up.\nI disagree with you on big shake down: I think I do not use leakage heavily comparing with other teams. And @gunesevitan do not use leakage for validation.", "votes": null }, { "id": "705536", "postDate": "12/29/2019 02:30:36", "content": "@hiroyukiex, thanks for opening this discussion so that we can learn more.
\n\nFor those interested, check my kernel for comparison of team positions on the LB with medal ranges based on the original 3675 teams public LB before rescoring.
", "rawMarkdown": "hiroyukiex, thanks for opening this discussion so that we can learn more. \n\nFor those interested, check [my kernel](https://www.kaggle.com/sheriytm/ashrae-public-lb-before-rescoring) for comparison of team positions on the LB with medal ranges based on the original 3675 teams [public LB before rescoring](https://www.kaggle.com/c/ashrae-energy-prediction/discussion/123497).", "votes": null }, { "id": "705539", "postDate": "12/29/2019 02:42:11", "content": "@gunesevitan, looking at the final_all_scores.csv file, my best score was a public LB=0.951836, private LB=1.248859 from a submission of almost 2 weeks ago. If I am not mistaken, I think we were in lockstep around that time. I am going to look into that when I get sometime but you may want to check that as well.
\n\nI dropped 9 places from 63rd to #72 according to the shared LB scores. You can check a pdf of the public LB before rescoring that I shared if anyone wants to analyse it.
", "rawMarkdown": "gunesevitan, looking at the final_all_scores.csv file, my best score was a public LB=0.951836, private LB=1.248859 from a submission of almost 2 weeks ago. If I am not mistaken, I think we were in lockstep around that time. I am going to look into that when I get sometime but you may want to check that as well.\n\nI dropped 9 places from 63rd to #72 according to the shared LB scores. You can check a [pdf of the public LB before rescoring](https://www.kaggle.com/sheriytm/ashrae-public-lb-before-rescoring) that I shared if anyone wants to analyse it.", "votes": null }, { "id": "705566", "postDate": "12/29/2019 04:19:14", "content": "I didn't use leaks for training or validation because I thought it will lead to overfitting. @yamsam actually published kernel about it and they got 1st place.
", "rawMarkdown": "I didn't use leaks for training or validation because I thought it will lead to overfitting. @yamsam actually published kernel about it and they got 1st place.", "votes": null }, { "id": "705594", "postDate": "12/29/2019 05:43:21", "content": "@gunesevitan Oh feel sorry to hear that. I have expected your team to get a gold medal or top 5... Learn a lot from your outlier detection strategy! Thank you very much!
", "rawMarkdown": "gunesevitan Oh feel sorry to hear that. I have expected your team to get a gold medal or top 5... Learn a lot from your outlier detection strategy! Thank you very much!", "votes": null }, { "id": "705821", "postDate": "12/29/2019 14:22:33", "content": "@gunesevitan You are still one of the heroes of this competition.
", "rawMarkdown": "gunesevitan You are still one of the heroes of this competition.", "votes": null }, { "id": "705848", "postDate": "12/29/2019 15:15:16", "content": "Who is the downvoter?
\n\nWhy don't you reply to my post and say what you found offensive about my post? I added the link of my other post and kernel because this is a discussion about finding out what most of us got wrong about solving this problem.
\n\nA downvote does not tell me or anybody else what is wrong.
\n\nMy adressing @gunesevitan is to indicate where I took a wrong direction in my modelling and thought it might be helpful for him in reviewing his case as well.
", "rawMarkdown": "Who is the downvoter? \n\nWhy don't you reply to my post and say what you found offensive about my post? I added the link of my other post and kernel because this is a discussion about finding out what most of us got wrong about solving this problem. \n\nA downvote does not tell me or anybody else what is wrong.\n\nMy adressing @gunesevitan is to indicate where I took a wrong direction in my modelling and thought it might be helpful for him in reviewing his case as well.", "votes": null }, { "id": "705900", "postDate": "12/29/2019 16:45:56", "content": "Thanks @rohanrao, but I guess that was my punishment for starting the leaks :D
\n\n@waylongo The thing is my 1.048 model had a coefficient of 0.3-0.5 in our blend. @divrikwicky's NN also had the same coefficients and the other 0.2 were public models. I expected our submissions to be robust but I was wrong.
\n\n@sheriytm I checked previous submissions but I can't figure out at which point I started doing wrong. I'm not the downvoter btw.
", "rawMarkdown": "Thanks @rohanrao, but I guess that was my punishment for starting the leaks :D\n\n@waylongo The thing is my 1.048 model had a coefficient of 0.3-0.5 in our blend. @divrikwicky's NN also had the same coefficients and the other 0.2 were public models. I expected our submissions to be robust but I was wrong.\n\n@sheriytm I checked previous submissions but I can't figure out at which point I started doing wrong. I'm not the downvoter btw.", "votes": null }, { "id": "705916", "postDate": "12/29/2019 17:01:31", "content": "\n\n\nI'm not the downvoter btw.
\n
@gunesevitan, I did not for a second believe your are. I think somebody may have wrongly thought I was disrespecting you or something. I know you long enough via discussions on Kaggle to know u would not do that.
\n\n\n\n\nThanks @rohanrao, but I guess that was my punishment for starting the leaks :D
\n
Please do not feel bad about this. I think it was brave of you to be the 1st one to reveal the leak and you are a hero in my eyes. Thank you very much for all that you have shared here and in previous comeptitions.
\n\nFor me, the reason I enjoyed this competition is because of the messy nature of the data but hated the leak just like most people.
\n\nAbout checking back my previous submissions to determine where I went wrong, it will have to wait a bit for now. I do have a rough idea but it is all inference at this point. When I have a clear picture I will share.
", "rawMarkdown": "> I'm not the downvoter btw.\n\n@gunesevitan, I did not for a second believe your are. I think somebody may have wrongly thought I was disrespecting you or something. I know you long enough via discussions on Kaggle to know u would not do that.\n\n> Thanks @rohanrao, but I guess that was my punishment for starting the leaks :D\n\nPlease do not feel bad about this. I think it was brave of you to be the 1st one to reveal the leak and you are a hero in my eyes. Thank you very much for all that you have shared here and in previous comeptitions.\n\nFor me, the reason I enjoyed this competition is because of the messy nature of the data but hated the leak just like most people.\n\nAbout checking back my previous submissions to determine where I went wrong, it will have to wait a bit for now. I do have a rough idea but it is all inference at this point. When I have a clear picture I will share.", "votes": null }, { "id": "705928", "postDate": "12/29/2019 17:07:56", "content": "One more question. Does anybody else experience issue with upvoting discussions?
\n\nI had to vote so many times to @gunesevitan's comment before it held this time. I tried the last 2 times I posted but would not hold until now. Even now I could not upvote @gunesevitan's last reply to me.
\n\nUpdate:\nOk it showed up. May be Kaggle upvote button is shaky today.
", "rawMarkdown": "One more question. Does anybody else experience issue with upvoting discussions? \n\nI had to vote so many times to @gunesevitan's comment before it held this time. I tried the last 2 times I posted but would not hold until now. Even now I could not upvote @gunesevitan's last reply to me.\n\nUpdate:\nOk it showed up. May be Kaggle upvote button is shaky today.", "votes": null }, { "id": "705970", "postDate": "12/29/2019 18:29:56", "content": "@gunesevitan I personally feel you were one of the leaders along with @nroman @rohanrao in the competition , and these results are just a part of it .
", "rawMarkdown": "gunesevitan I personally feel you were one of the leaders along with @nroman @rohanrao in the competition , and these results are just a part of it .", "votes": null }, { "id": "705984", "postDate": "12/29/2019 19:21:49", "content": "I worked on the last week of the competition and improve 0.02 in PL.
\n\nThe previous submissions without this last improve reach top 80 in private, but with this improvement I down to 1000+
\n\nBut ok, it's was my first compettion, lets keep improving..
", "rawMarkdown": "I worked on the last week of the competition and improve 0.02 in PL.\n\nThe previous submissions without this last improve reach top 80 in private, but with this improvement I down to 1000+ \n\nBut ok, it's was my first compettion, lets keep improving..", "votes": null }, { "id": "706190", "postDate": "12/30/2019 03:37:25", "content": ">One more question. Does anybody else experience issue with upvoting discussions?
\n\nYes, sometimes it is 'shaky'. I faced issues yesterday too. I find it particularly bad while using phone.
", "rawMarkdown": ">One more question. Does anybody else experience issue with upvoting discussions?\n\nYes, sometimes it is 'shaky'. I faced issues yesterday too. I find it particularly bad while using phone.", "votes": null }, { "id": "706202", "postDate": "12/30/2019 04:10:39", "content": "Thanks @rohanrao for confirming. I experienced it when upvoting a discussion post in which case I can only vote outside of the post. I mean I could not upvote once I am in the post but can from the list of discussion posts. Very strange.
", "rawMarkdown": "Thanks @rohanrao for confirming. I experienced it when upvoting a discussion post in which case I can only vote outside of the post. I mean I could not upvote once I am in the post but can from the list of discussion posts. Very strange.", "votes": null }, { "id": "706308", "postDate": "12/30/2019 08:26:31", "content": "Never considered myself as a leader of the competition, since I haven't made anything significant to climb the LB.
\n\nCurrent score shows that we dropped around 40 positions. Which is 'not great, not terrible'. Will analyze it after the NY.
", "rawMarkdown": "Never considered myself as a leader of the competition, since I haven't made anything significant to climb the LB.\n\nCurrent score shows that we dropped around 40 positions. Which is 'not great, not terrible'. Will analyze it after the NY.", "votes": null }, { "id": "706806", "postDate": "12/30/2019 20:55:35", "content": "My team's score had shake down, too. (108 -> 534)\nI think the cause of this problem is how to CV.
\n\nWe use KFold model & Stratified KFold model in ensemble.(shake down submit)\nHowever, when I checked other submissions, the model using only KFold was the highest PrivateLB score.
\n\nAlso, some solution said '4-fold and 5-fold CV', 'different time split, x3, x4 and x6'.\nhttps://www.kaggle.com/c/ashrae-energy-prediction/discussion/123481\nhttps://www.kaggle.com/c/ashrae-energy-prediction/discussion/123525
\n\nI think winner's CV strategy is KFold(not Stratified KFold) and num>2.
", "rawMarkdown": "My team's score had shake down, too. (108 -> 534)\nI think the cause of this problem is how to CV.\n\nWe use KFold model & Stratified KFold model in ensemble.(shake down submit)\nHowever, when I checked other submissions, the model using only KFold was the highest PrivateLB score.\n\nAlso, some solution said '4-fold and 5-fold CV', 'different time split, x3, x4 and x6'.\nhttps://www.kaggle.com/c/ashrae-energy-prediction/discussion/123481\nhttps://www.kaggle.com/c/ashrae-energy-prediction/discussion/123525\n\nI think winner's CV strategy is KFold(not Stratified KFold) and num>2.", "votes": null }, { "id": "706977", "postDate": "12/31/2019 05:26:45", "content": "Thanks for comment. Maybe you are right. I mainly used 2-fold time-series based (not stratified) CV. But my CV score and leak validation score and LB were highly correlated, so I did not doubt my CV strategy.
", "rawMarkdown": "Thanks for comment. Maybe you are right. I mainly used 2-fold time-series based (not stratified) CV. But my CV score and leak validation score and LB were highly correlated, so I did not doubt my CV strategy.", "votes": null }, { "id": "706980", "postDate": "12/31/2019 05:31:57", "content": "That shouldn't be true because I was using 5 kfold
", "rawMarkdown": "That shouldn't be true because I was using 5 kfold", "votes": null }, { "id": "707025", "postDate": "12/31/2019 07:15:39", "content": "Oh really...\nDid any of the blend kernels have other way of cv?\n(ex.Stratified KFold...)
", "rawMarkdown": "Oh really...\nDid any of the blend kernels have other way of cv?\n(ex.Stratified KFold...)", "votes": null }, { "id": "707027", "postDate": "12/31/2019 07:26:32", "content": "while using Kfold, did you use shuffle=True?
", "rawMarkdown": "while using Kfold, did you use shuffle=True?", "votes": null } ], "comments": [ { "id": 704858, "author_name": "hanjoonchoe", "author_url": "", "post_date": "12/28/2019 05:11:56", "content": "That was unlucky. but you did great job regardless of result itself! ;-) I choose one of my submission with 4 models (my best, middle, middle, somewhat bad) blending in which I set coefficients aiming at trying to maximize the correlations in meter-wise, and that was my best private score. That might make my private score not too low also not too high.
", "votes": null, "replies": [] }, { "id": 704879, "author_name": "klloke", "author_url": "", "post_date": "12/28/2019 05:51:44", "content": "I got shake down to Rank120 from top30 probably because I use leakage. I accept the final score, and congrats to all winners too.
", "votes": null, "replies": [] }, { "id": 704963, "author_name": "kueno55", "author_url": "", "post_date": "12/28/2019 08:10:16", "content": "I was submitting after the leak data merge and made a mistake choosing the final submission. So I guess that is because you were evaluating the algorithm with the public score after the leak data merge. For example, in a private test, this divergence is likely to occur when weather features are more important than building features. However, I think that the evaluation of the algorithm was difficult for everyone in this competition situation. This is the opinion imagined from my results.
", "votes": null, "replies": [ { "id": 705054, "author_name": "hiroyukiex", "author_url": "", "post_date": "12/28/2019 11:04:30", "content": "Thanks for your comment. I agree with your idea that weather features are more important than building features in private LB! But I mainly used 2018 (private) for leak validation and that validation score was correlated with public LB.
", "votes": null, "replies": [] } ] }, { "id": 705116, "author_name": "gunesevitan", "author_url": "", "post_date": "12/28/2019 13:06:12", "content": "Our team has the worst drop after you (Public 9th - Private 378th). I have no idea what caused it. We didn't use leaks for training or validation. I hate this competition so much :D
", "votes": null, "replies": [ { "id": 705129, "author_name": "hiroyukiex", "author_url": "", "post_date": "12/28/2019 13:40:39", "content": "I am very glad to get comment from you. We are in similar situation.(Outlier removal strategy and shake down) Maybe, main reason is not related to leak but the restricted number of buildings in private LB.\nOur outlier removal strategy may work great in some restricted buildings (not in site 0,1,2,3,4,15!) that are somehow removed from private LB. But even considering the above reason, I estimated you and I are in silver zone at worst. Though some terrible things happened, I learned a lot from this competition😄
", "votes": null, "replies": [] }, { "id": 705153, "author_name": "proletheus", "author_url": "", "post_date": "12/28/2019 14:39:06", "content": "if i understand correctly you also relied heavily on removing outliers. maybe that caused you to overfit to the public lb part
", "votes": null, "replies": [] }, { "id": 705408, "author_name": "waylongo", "author_url": "", "post_date": "12/28/2019 22:30:13", "content": "@gunesevitan I looked at your 1.048 kernel and found out your prediction on building #778 is low. I guess this is one of the reasons your team shaked down. The meter reading of this building is huge so it can be dominant on the final rmse.
", "votes": null, "replies": [] }, { "id": 705539, "author_name": "sheriytm", "author_url": "", "post_date": "12/29/2019 02:42:11", "content": "@gunesevitan, looking at the final_all_scores.csv file, my best score was a public LB=0.951836, private LB=1.248859 from a submission of almost 2 weeks ago. If I am not mistaken, I think we were in lockstep around that time. I am going to look into that when I get sometime but you may want to check that as well.
\n\nI dropped 9 places from 63rd to #72 according to the shared LB scores. You can check a pdf of the public LB before rescoring that I shared if anyone wants to analyse it.
", "votes": null, "replies": [] }, { "id": 705594, "author_name": "uestcblackstar", "author_url": "", "post_date": "12/29/2019 05:43:21", "content": "@gunesevitan Oh feel sorry to hear that. I have expected your team to get a gold medal or top 5... Learn a lot from your outlier detection strategy! Thank you very much!
", "votes": null, "replies": [] }, { "id": 705821, "author_name": "rohanrao", "author_url": "", "post_date": "12/29/2019 14:22:33", "content": "@gunesevitan You are still one of the heroes of this competition.
", "votes": null, "replies": [] }, { "id": 705848, "author_name": "sheriytm", "author_url": "", "post_date": "12/29/2019 15:15:16", "content": "Who is the downvoter?
\n\nWhy don't you reply to my post and say what you found offensive about my post? I added the link of my other post and kernel because this is a discussion about finding out what most of us got wrong about solving this problem.
\n\nA downvote does not tell me or anybody else what is wrong.
\n\nMy adressing @gunesevitan is to indicate where I took a wrong direction in my modelling and thought it might be helpful for him in reviewing his case as well.
", "votes": null, "replies": [] }, { "id": 705900, "author_name": "gunesevitan", "author_url": "", "post_date": "12/29/2019 16:45:56", "content": "Thanks @rohanrao, but I guess that was my punishment for starting the leaks :D
\n\n@waylongo The thing is my 1.048 model had a coefficient of 0.3-0.5 in our blend. @divrikwicky's NN also had the same coefficients and the other 0.2 were public models. I expected our submissions to be robust but I was wrong.
\n\n@sheriytm I checked previous submissions but I can't figure out at which point I started doing wrong. I'm not the downvoter btw.
", "votes": null, "replies": [] }, { "id": 705916, "author_name": "sheriytm", "author_url": "", "post_date": "12/29/2019 17:01:31", "content": "\n\n\nI'm not the downvoter btw.
\n
@gunesevitan, I did not for a second believe your are. I think somebody may have wrongly thought I was disrespecting you or something. I know you long enough via discussions on Kaggle to know u would not do that.
\n\n\n\n\nThanks @rohanrao, but I guess that was my punishment for starting the leaks :D
\n
Please do not feel bad about this. I think it was brave of you to be the 1st one to reveal the leak and you are a hero in my eyes. Thank you very much for all that you have shared here and in previous comeptitions.
\n\nFor me, the reason I enjoyed this competition is because of the messy nature of the data but hated the leak just like most people.
\n\nAbout checking back my previous submissions to determine where I went wrong, it will have to wait a bit for now. I do have a rough idea but it is all inference at this point. When I have a clear picture I will share.
", "votes": null, "replies": [] }, { "id": 705928, "author_name": "sheriytm", "author_url": "", "post_date": "12/29/2019 17:07:56", "content": "One more question. Does anybody else experience issue with upvoting discussions?
\n\nI had to vote so many times to @gunesevitan's comment before it held this time. I tried the last 2 times I posted but would not hold until now. Even now I could not upvote @gunesevitan's last reply to me.
\n\nUpdate:\nOk it showed up. May be Kaggle upvote button is shaky today.
", "votes": null, "replies": [] }, { "id": 706190, "author_name": "rohanrao", "author_url": "", "post_date": "12/30/2019 03:37:25", "content": ">One more question. Does anybody else experience issue with upvoting discussions?
\n\nYes, sometimes it is 'shaky'. I faced issues yesterday too. I find it particularly bad while using phone.
", "votes": null, "replies": [] }, { "id": 706202, "author_name": "sheriytm", "author_url": "", "post_date": "12/30/2019 04:10:39", "content": "Thanks @rohanrao for confirming. I experienced it when upvoting a discussion post in which case I can only vote outside of the post. I mean I could not upvote once I am in the post but can from the list of discussion posts. Very strange.
", "votes": null, "replies": [] } ] }, { "id": 705232, "author_name": "lftuwujie", "author_url": "", "post_date": "12/28/2019 17:15:37", "content": "Reason of shake up or down is very clear - whether you used leakage or not, or how much leakage did you use for both your submission and model training. It’s worth to dive deep why it shakedown, if you haven’t used any leak data.
", "votes": null, "replies": [ { "id": 705477, "author_name": "hiroyukiex", "author_url": "", "post_date": "12/29/2019 00:33:39", "content": "I agree with you on shake up: Not replacing leakage leads to shake up.\nI disagree with you on big shake down: I think I do not use leakage heavily comparing with other teams. And @gunesevitan do not use leakage for validation.
", "votes": null, "replies": [] }, { "id": 705566, "author_name": "gunesevitan", "author_url": "", "post_date": "12/29/2019 04:19:14", "content": "I didn't use leaks for training or validation because I thought it will lead to overfitting. @yamsam actually published kernel about it and they got 1st place.
", "votes": null, "replies": [] } ] }, { "id": 705536, "author_name": "sheriytm", "author_url": "", "post_date": "12/29/2019 02:30:36", "content": "@hiroyukiex, thanks for opening this discussion so that we can learn more.
\n\nFor those interested, check my kernel for comparison of team positions on the LB with medal ranges based on the original 3675 teams public LB before rescoring.
", "votes": null, "replies": [] }, { "id": 705970, "author_name": "avikrams", "author_url": "", "post_date": "12/29/2019 18:29:56", "content": "@gunesevitan I personally feel you were one of the leaders along with @nroman @rohanrao in the competition , and these results are just a part of it .
", "votes": null, "replies": [ { "id": 706308, "author_name": "nroman", "author_url": "", "post_date": "12/30/2019 08:26:31", "content": "Never considered myself as a leader of the competition, since I haven't made anything significant to climb the LB.
\n\nCurrent score shows that we dropped around 40 positions. Which is 'not great, not terrible'. Will analyze it after the NY.
", "votes": null, "replies": [] } ] }, { "id": 705984, "author_name": "clappis", "author_url": "", "post_date": "12/29/2019 19:21:49", "content": "I worked on the last week of the competition and improve 0.02 in PL.
\n\nThe previous submissions without this last improve reach top 80 in private, but with this improvement I down to 1000+
\n\nBut ok, it's was my first compettion, lets keep improving..
", "votes": null, "replies": [] }, { "id": 706806, "author_name": "yu19900724", "author_url": "", "post_date": "12/30/2019 20:55:35", "content": "My team's score had shake down, too. (108 -> 534)\nI think the cause of this problem is how to CV.
\n\nWe use KFold model & Stratified KFold model in ensemble.(shake down submit)\nHowever, when I checked other submissions, the model using only KFold was the highest PrivateLB score.
\n\nAlso, some solution said '4-fold and 5-fold CV', 'different time split, x3, x4 and x6'.\nhttps://www.kaggle.com/c/ashrae-energy-prediction/discussion/123481\nhttps://www.kaggle.com/c/ashrae-energy-prediction/discussion/123525
\n\nI think winner's CV strategy is KFold(not Stratified KFold) and num>2.
", "votes": null, "replies": [ { "id": 706977, "author_name": "hiroyukiex", "author_url": "", "post_date": "12/31/2019 05:26:45", "content": "Thanks for comment. Maybe you are right. I mainly used 2-fold time-series based (not stratified) CV. But my CV score and leak validation score and LB were highly correlated, so I did not doubt my CV strategy.
", "votes": null, "replies": [] }, { "id": 706980, "author_name": "gunesevitan", "author_url": "", "post_date": "12/31/2019 05:31:57", "content": "That shouldn't be true because I was using 5 kfold
", "votes": null, "replies": [] }, { "id": 707025, "author_name": "yu19900724", "author_url": "", "post_date": "12/31/2019 07:15:39", "content": "Oh really...\nDid any of the blend kernels have other way of cv?\n(ex.Stratified KFold...)
", "votes": null, "replies": [] }, { "id": 707027, "author_name": "lftuwujie", "author_url": "", "post_date": "12/31/2019 07:26:32", "content": "while using Kfold, did you use shuffle=True?
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "704820": "Now, I analyzed final-all-scores.csv and learned some part of the reason of my shake down. (Public 1st -> Private 497th?😥)\n\nOther than site 0,1,2,(3?),4,15, some buildings in other sites seem to be excluded in private.\nI think my good public score relied too much on a few \"difficult and influential\" time-series for example building_id=1250 and meter=2 those are not in private.\nIf any other ideas of the reason of your/our shake, please let me know.\n\nAnyway, congratulations on winners!", "704858": "That was unlucky. but you did great job regardless of result itself! ;-) I choose one of my submission with 4 models (my best, middle, middle, somewhat bad) blending in which I set coefficients aiming at trying to maximize the correlations in meter-wise, and that was my best private score. That might make my private score not too low also not too high.", "704879": "I got shake down to Rank120 from top30 probably because I use leakage. I accept the final score, and congrats to all winners too.", "704963": "I was submitting after the leak data merge and made a mistake choosing the final submission. So I guess that is because you were evaluating the algorithm with the public score after the leak data merge. For example, in a private test, this divergence is likely to occur when weather features are more important than building features. However, I think that the evaluation of the algorithm was difficult for everyone in this competition situation. This is the opinion imagined from my results.", "705054": "Thanks for your comment. I agree with your idea that weather features are more important than building features in private LB! But I mainly used 2018 (private) for leak validation and that validation score was correlated with public LB.", "705116": "Our team has the worst drop after you (Public 9th - Private 378th). I have no idea what caused it. We didn't use leaks for training or validation. I hate this competition so much :D", "705129": "I am very glad to get comment from you. We are in similar situation.(Outlier removal strategy and shake down) Maybe, main reason is not related to leak but the restricted number of buildings in private LB.\nOur outlier removal strategy may work great in some restricted buildings (not in site 0,1,2,3,4,15!) that are somehow removed from private LB. But even considering the above reason, I estimated you and I are in silver zone at worst. Though some terrible things happened, I learned a lot from this competition😄", "705153": "if i understand correctly you also relied heavily on removing outliers. maybe that caused you to overfit to the public lb part", "705232": "Reason of shake up or down is very clear - whether you used leakage or not, or how much leakage did you use for both your submission and model training. It’s worth to dive deep why it shakedown, if you haven’t used any leak data.", "705408": "gunesevitan I looked at your 1.048 kernel and found out your prediction on building #778 is low. I guess this is one of the reasons your team shaked down. The meter reading of this building is huge so it can be dominant on the final rmse.", "705477": "I agree with you on shake up: Not replacing leakage leads to shake up.\nI disagree with you on big shake down: I think I do not use leakage heavily comparing with other teams. And @gunesevitan do not use leakage for validation.", "705536": "hiroyukiex, thanks for opening this discussion so that we can learn more. \n\nFor those interested, check [my kernel](https://www.kaggle.com/sheriytm/ashrae-public-lb-before-rescoring) for comparison of team positions on the LB with medal ranges based on the original 3675 teams [public LB before rescoring](https://www.kaggle.com/c/ashrae-energy-prediction/discussion/123497).", "705539": "gunesevitan, looking at the final_all_scores.csv file, my best score was a public LB=0.951836, private LB=1.248859 from a submission of almost 2 weeks ago. If I am not mistaken, I think we were in lockstep around that time. I am going to look into that when I get sometime but you may want to check that as well.\n\nI dropped 9 places from 63rd to #72 according to the shared LB scores. You can check a [pdf of the public LB before rescoring](https://www.kaggle.com/sheriytm/ashrae-public-lb-before-rescoring) that I shared if anyone wants to analyse it.", "705566": "I didn't use leaks for training or validation because I thought it will lead to overfitting. @yamsam actually published kernel about it and they got 1st place.", "705594": "gunesevitan Oh feel sorry to hear that. I have expected your team to get a gold medal or top 5... Learn a lot from your outlier detection strategy! Thank you very much!", "705821": "gunesevitan You are still one of the heroes of this competition.", "705848": "Who is the downvoter? \n\nWhy don't you reply to my post and say what you found offensive about my post? I added the link of my other post and kernel because this is a discussion about finding out what most of us got wrong about solving this problem. \n\nA downvote does not tell me or anybody else what is wrong.\n\nMy adressing @gunesevitan is to indicate where I took a wrong direction in my modelling and thought it might be helpful for him in reviewing his case as well.", "705900": "Thanks @rohanrao, but I guess that was my punishment for starting the leaks :D\n\n@waylongo The thing is my 1.048 model had a coefficient of 0.3-0.5 in our blend. @divrikwicky's NN also had the same coefficients and the other 0.2 were public models. I expected our submissions to be robust but I was wrong.\n\n@sheriytm I checked previous submissions but I can't figure out at which point I started doing wrong. I'm not the downvoter btw.", "705916": "> I'm not the downvoter btw.\n\n@gunesevitan, I did not for a second believe your are. I think somebody may have wrongly thought I was disrespecting you or something. I know you long enough via discussions on Kaggle to know u would not do that.\n\n> Thanks @rohanrao, but I guess that was my punishment for starting the leaks :D\n\nPlease do not feel bad about this. I think it was brave of you to be the 1st one to reveal the leak and you are a hero in my eyes. Thank you very much for all that you have shared here and in previous comeptitions.\n\nFor me, the reason I enjoyed this competition is because of the messy nature of the data but hated the leak just like most people.\n\nAbout checking back my previous submissions to determine where I went wrong, it will have to wait a bit for now. I do have a rough idea but it is all inference at this point. When I have a clear picture I will share.", "705928": "One more question. Does anybody else experience issue with upvoting discussions? \n\nI had to vote so many times to @gunesevitan's comment before it held this time. I tried the last 2 times I posted but would not hold until now. Even now I could not upvote @gunesevitan's last reply to me.\n\nUpdate:\nOk it showed up. May be Kaggle upvote button is shaky today.", "705970": "gunesevitan I personally feel you were one of the leaders along with @nroman @rohanrao in the competition , and these results are just a part of it .", "705984": "I worked on the last week of the competition and improve 0.02 in PL.\n\nThe previous submissions without this last improve reach top 80 in private, but with this improvement I down to 1000+ \n\nBut ok, it's was my first compettion, lets keep improving..", "706190": ">One more question. Does anybody else experience issue with upvoting discussions?\n\nYes, sometimes it is 'shaky'. I faced issues yesterday too. I find it particularly bad while using phone.", "706202": "Thanks @rohanrao for confirming. I experienced it when upvoting a discussion post in which case I can only vote outside of the post. I mean I could not upvote once I am in the post but can from the list of discussion posts. Very strange.", "706308": "Never considered myself as a leader of the competition, since I haven't made anything significant to climb the LB.\n\nCurrent score shows that we dropped around 40 positions. Which is 'not great, not terrible'. Will analyze it after the NY.", "706806": "My team's score had shake down, too. (108 -> 534)\nI think the cause of this problem is how to CV.\n\nWe use KFold model & Stratified KFold model in ensemble.(shake down submit)\nHowever, when I checked other submissions, the model using only KFold was the highest PrivateLB score.\n\nAlso, some solution said '4-fold and 5-fold CV', 'different time split, x3, x4 and x6'.\nhttps://www.kaggle.com/c/ashrae-energy-prediction/discussion/123481\nhttps://www.kaggle.com/c/ashrae-energy-prediction/discussion/123525\n\nI think winner's CV strategy is KFold(not Stratified KFold) and num>2.", "706977": "Thanks for comment. Maybe you are right. I mainly used 2-fold time-series based (not stratified) CV. But my CV score and leak validation score and LB were highly correlated, so I did not doubt my CV strategy.", "706980": "That shouldn't be true because I was using 5 kfold", "707025": "Oh really...\nDid any of the blend kernels have other way of cv?\n(ex.Stratified KFold...)", "707027": "while using Kfold, did you use shuffle=True?" }, "source": "meta" }