Hey All,
\n\nCross-posting Sohier's post here for visibility:
\n\nQuick update - we have a working draft containing ~40,000 rescores but need to check some edge cases to make sure we didn't miss valid submissions and to run additional tests to ensure that updating the scores kicks off all of the correct jobs downstream, like reinstating the ~1000+ teams that vanished from the LB. This will take additional time, but will not be done until after Christmas. We will post another update or the scores on the PST afternoon of the 26th. Thank you all for your patience!
\n\nTL;DR - We didn't have enough milk and cookies to feed Santa and his reindeer so they skipped over our offices, so we're celebrating Boxing Day instead this year.
", "messages": [ { "id": 702505, "postDate": "2019-12-24T19:10:11.857Z", "content": ".jpeg?generation=1577214609571546&alt=media)
Haha, good one!
", "rawMarkdown": "Haha, good one!", "votes": 2 } ] }, { "id": 702031, "postDate": "2019-12-24T07:17:26.787Z", "content": "I think he skipped because you weren't good kids. Santa hates leaky and unorganized competitions:) Thanks for the update anyway.
", "rawMarkdown": "I think he skipped because you weren't good kids. Santa hates leaky and unorganized competitions:) Thanks for the update anyway.", "votes": 23 }, { "id": 704088, "postDate": "2019-12-27T03:28:05.850Z", "content": "\n\n\nWe will post another update or the scores on the PST afternoon of the 26th.
\n
Shouldn't there have been an update by now? It's already 7:30pm (PST)? @addisonhoward
", "rawMarkdown": "> We will post another update or the scores on the PST afternoon of the 26th.\n\nShouldn't there have been an update by now? It's already 7:30pm (PST)? @addisonhoward ", "votes": 18, "replies": [ { "id": 704120, "postDate": "2019-12-27T04:41:40.613Z", "content": "Probably he is on Christmas holiday๐
", "rawMarkdown": "Probably he is on Christmas holiday๐ ", "votes": 4 }, { "id": 704124, "postDate": "2019-12-27T04:46:49.957Z", "content": "It โs a long long Christmas holiday:)
", "rawMarkdown": "It โs a long long Christmas holiday:)", "votes": 3 }, { "id": 704125, "postDate": "2019-12-27T04:47:50.303Z", "content": "Actually,I also has this question,when can it be posted . afternoon? But it already night
", "rawMarkdown": "Actually,I also has this question,when can it be posted . afternoon? But it already night", "votes": 3 }, { "id": 704126, "postDate": "2019-12-27T04:50:34.043Z", "content": "maybe, 2020/12/26?
", "rawMarkdown": "maybe, 2020/12/26?", "votes": 4 }, { "id": 704128, "postDate": "2019-12-27T04:53:23.053Z", "content": "his \"afternoon\" means \"till 23:59\" ?
", "rawMarkdown": "his \"afternoon\" means \"till 23:59\" ?", "votes": 2 }, { "id": 704148, "postDate": "2019-12-27T05:32:17.120Z", "content": "My year-end is... \n\nfor hour in range(24):
\n press F5
\n
@yoichi7yamakawa this is the funniest thing I've seen today!
", "rawMarkdown": "@yoichi7yamakawa this is the funniest thing I've seen today!", "votes": 1 }, { "id": 704454, "postDate": "2019-12-27T14:02:10.807Z", "content": "Unfortunately the update is that we're still working on it... We've finished a LOT of the rescores, but the rest are still underway. I promise you we are working hard on it, investigate edge cases, and doing what we can to get the results back to you both quickly and with quality.
\n\nThanks for the patience!
", "rawMarkdown": "Unfortunately the update is that we're still working on it... We've finished a LOT of the rescores, but the rest are still underway. I promise you we are working hard on it, investigate edge cases, and doing what we can to get the results back to you both quickly and with quality.\n\nThanks for the patience!", "votes": 4 }, { "id": 704458, "postDate": "2019-12-27T14:07:52.990Z", "content": "@addisonhoward any guesses when it could be finished? Is it today/tomorrow or more likely next week?
", "rawMarkdown": "@addisonhoward any guesses when it could be finished? Is it today/tomorrow or more likely next week?", "votes": 1 }, { "id": 704461, "postDate": "2019-12-27T14:11:04.507Z", "content": "same question,could you please tell us the exact time when it can be finished and when the outcome can be published. \nI hope this can be true
", "rawMarkdown": "same question,could you please tell us the exact time when it can be finished and when the outcome can be published. \nI hope this can be true", "votes": 2 }, { "id": 704464, "postDate": "2019-12-27T14:14:24.887Z", "content": "We're hopeful it will be today! But most of the timing is out of our control, so we're actively monitoring, troubleshooting, and trying to figure out what the hangups are.
\n\nWe'd like to have it wrapped up prior to the weekend just as much as you!
", "rawMarkdown": "We're hopeful it will be today! But most of the timing is out of our control, so we're actively monitoring, troubleshooting, and trying to figure out what the hangups are.\n\nWe'd like to have it wrapped up prior to the weekend just as much as you!", "votes": 7 }, { "id": 704468, "postDate": "2019-12-27T14:18:22.763Z", "content": "Estimating the time shouldn't be so difficult. You know the time needed for each submission and how much resource you have. I really appreciate your effort but Kaggle can actually be more transparent. The problem could be solved in 3500x2x5/60/num_cores hours. If Kaggle has 500 cores, it is roughly 1 hour. I feel like there is something else going on.
", "rawMarkdown": "Estimating the time shouldn't be so difficult. You know the time needed for each submission and how much resource you have. I really appreciate your effort but Kaggle can actually be more transparent. The problem could be solved in 3500x2x5/60/num_cores hours. If Kaggle has 500 cores, it is roughly 1 hour. I feel like there is something else going on.", "votes": 1 }, { "id": 704747, "postDate": "2019-12-28T00:25:37.633Z", "content": "@addisonhoward Thank you for information. I really appreciate your effort. Please take a long time for quality of LB including next year if needed.\nBut I'm sad if\n-Change after opening private LB due to some mistake in rescoring\n-Only rescoring, no check on cheaters\n-Repeating over-promising such as 24๏ฝ48 business hours and update 26th afternoon
", "rawMarkdown": "@addisonhoward Thank you for information. I really appreciate your effort. Please take a long time for quality of LB including next year if needed.\nBut I'm sad if\n-Change after opening private LB due to some mistake in rescoring\n-Only rescoring, no check on cheaters\n-Repeating over-promising such as 24๏ฝ48 business hours and update 26th afternoon", "votes": 2 } ] }, { "id": 704163, "postDate": "2019-12-27T05:56:45.773Z", "content": "ASHRAE competitors be like:\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F113389%2F1bae91d989793a48ef9ee3b16b89014d%2Flaptop.gif?generation=1577426146148882&alt=media\")
I just bought a computer last week๐ญ
", "rawMarkdown": "I just bought a computer last week๐ญ ", "votes": 6 }, { "id": 704186, "postDate": "2019-12-27T06:29:24.797Z", "content": "if computer is broken, I use kaggle_kernel so its alright๐
", "rawMarkdown": "if computer is broken, I use kaggle_kernel so its alright๐", "votes": 4 }, { "id": 704196, "postDate": "2019-12-27T06:40:50.967Z", "content": "I'm replying from my phone now. PC-less
", "rawMarkdown": "I'm replying from my phone now. PC-less", "votes": 1 }, { "id": 704211, "postDate": "2019-12-27T07:18:46.300Z", "content": "![\"\"](\"https://media1.tenor.com/images/c1515d8886faabb4282bed20d07da002/tenor.gif?itemid=11792220\")
This competition has been a frustration for me for multiple reasons. \nI think leak cases could have been handled better, sooner. \nCheaters, probers could have been addressed sooner.\nAnd when we think at last the competition is over, waiting days, or probably weeks for results added insult to injury.\nMaybe simply prioritizing selected submits of teams if they have, or simply going late from early could have been a relief. Most of -serious- teams handpick their submits. We have only 2 rescores both of irrelevant and early tries.\nOn the other hand, I realize this is a demanding community and we will find a reason to whine about anyways,\nso good luck and happy new year to you all :)\n.
", "rawMarkdown": "This competition has been a frustration for me for multiple reasons. \nI think leak cases could have been handled better, sooner. \nCheaters, probers could have been addressed sooner.\nAnd when we think at last the competition is over, waiting days, or probably weeks for results added insult to injury.\nMaybe simply prioritizing selected submits of teams if they have, or simply going late from early could have been a relief. Most of -serious- teams handpick their submits. We have only 2 rescores both of irrelevant and early tries.\nOn the other hand, I realize this is a demanding community and we will find a reason to whine about anyways,\nso good luck and happy new year to you all :)\n.", "votes": 14, "replies": [ { "id": 703813, "postDate": "2019-12-26T17:01:27.497Z", "content": "yeah we need private LB, not \"Leaderboard Reveal - Update 4\" ๐
", "rawMarkdown": "yeah we need private LB, not \"Leaderboard Reveal - Update 4\" ๐ ", "votes": 6 } ] }, { "id": 703774, "postDate": "2019-12-26T15:57:34.647Z", "content": "![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1956324%2Fc40d8ff7afe1ccf3c240711360f8cad2%2Fjust_checking.jpg?generation=1576939907198897&alt=media\")
I hope you are trying to catch those who identified a public data source and did not disclose it properly if it wasn't already disclosed. This is easy to find: look for perfect prediction...
\n\nI feel the agony of all the participants in the meantime...
", "rawMarkdown": "I hope you are trying to catch those who identified a public data source and did not disclose it properly if it wasn't already disclosed. This is easy to find: look for perfect prediction...\n\nI feel the agony of all the participants in the meantime...", "votes": 12, "replies": [ { "id": 702188, "postDate": "2019-12-24T11:15:41.590Z", "content": "I agree, but what happens if they trained on it instead? their scores would be much better, but would not result in perfect predictions. Or what happens if they blended their model predictions with perfect predictions? Reverse out the blend by dividing by ground truth labels and see if there is a consistent coefficient?
", "rawMarkdown": "I agree, but what happens if they trained on it instead? their scores would be much better, but would not result in perfect predictions. Or what happens if they blended their model predictions with perfect predictions? Reverse out the blend by dividing by ground truth labels and see if there is a consistent coefficient?", "votes": 1 }, { "id": 702191, "postDate": "2019-12-24T11:22:41.313Z", "content": "They may avoid making perfect predictions. I would rank teams site by site and further investigate the teams which have much higher rank in some sites compared to other sites.
", "rawMarkdown": "They may avoid making perfect predictions. I would rank teams site by site and further investigate the teams which have much higher rank in some sites compared to other sites.", "votes": 5 }, { "id": 702201, "postDate": "2019-12-24T11:41:00.187Z", "content": "Time to launch another competition - ASHRAE leak users detection. The winner gets his LB score in this one. In 3 months :D
", "rawMarkdown": "Time to launch another competition - ASHRAE leak users detection. The winner gets his LB score in this one. In 3 months :D", "votes": 6 }, { "id": 702275, "postDate": "2019-12-24T13:37:32.637Z", "content": "So there will be s private sharing on private sharing and a leakage on leakage
", "rawMarkdown": "So there will be s private sharing on private sharing and a leakage on leakage", "votes": 1 }, { "id": 702305, "postDate": "2019-12-24T14:25:07.070Z", "content": "Identifying severe difference of model performance on various site-ids is not that complicated. That's a great idea @cpmpml.
", "rawMarkdown": "Identifying severe difference of model performance on various site-ids is not that complicated. That's a great idea @cpmpml.", "votes": 1 }, { "id": 702311, "postDate": "2019-12-24T14:33:08.217Z", "content": "Many cheaters are lazy... That's why they are caught.
", "rawMarkdown": "Many cheaters are lazy... That's why they are caught.", "votes": 6 }, { "id": 702336, "postDate": "2019-12-24T15:11:44.727Z", "content": "Another clever idea by @gunesevitan to tag leaked sites is to check submissions that gave significant boost to a team's score and compare predictions against previous best to identify which sites give that boost to find likely leak.
", "rawMarkdown": "Another clever idea by @gunesevitan to tag leaked sites is to check submissions that gave significant boost to a team's score and compare predictions against previous best to identify which sites give that boost to find likely leak.", "votes": 2 }, { "id": 702412, "postDate": "2019-12-24T16:49:56.150Z", "content": "\n\n\nMany cheaters are lazyโฆ That's why they are caught.
\n
Yes. Because to avoid getting caught they would have to work hard which defeats the purpose of cheating in the 1st place. Hence lazy approach to the project.
", "rawMarkdown": "> Many cheaters are lazyโฆ That's why they are caught.\n\nYes. Because to avoid getting caught they would have to work hard which defeats the purpose of cheating in the 1st place. Hence lazy approach to the project.", "votes": 1 } ] }, { "id": 702027, "postDate": "2019-12-24T07:13:11.257Z", "content": "![\"\"](\"https://i.kym-cdn.com/photos/images/original/001/072/001/de0.gif\")
Hey All,
\n\nCross-posting Sohier's post here for visibility:
\n\nQuick update - we have a working draft containing ~40,000 rescores but need to check some edge cases to make sure we didn't miss valid submissions and to run additional tests to ensure that updating the scores kicks off all of the correct jobs downstream, like reinstating the ~1000+ teams that vanished from the LB. This will take additional time, but will not be done until after Christmas. We will post another update or the scores on the PST afternoon of the 26th. Thank you all for your patience!
\n\nTL;DR - We didn't have enough milk and cookies to feed Santa and his reindeer so they skipped over our offices, so we're celebrating Boxing Day instead this year.
", "rawMarkdown": "Hey All,\n\nCross-posting [Sohier's post](https://www.kaggle.com/c/ashrae-energy-prediction/discussion/122554#701813) here for visibility:\n\nQuick update - we have a working draft containing ~40,000 rescores but need to check some edge cases to make sure we didn't miss valid submissions and to run additional tests to ensure that updating the scores kicks off all of the correct jobs downstream, like reinstating the ~1000+ teams that vanished from the LB. This will take additional time, but will not be done until after Christmas. We will post another update or the scores on the PST afternoon of the 26th. Thank you all for your patience!\n\nTL;DR - We didn't have enough milk and cookies to feed Santa and his reindeer so they skipped over our offices, so we're celebrating Boxing Day instead this year. ", "votes": 15 }, { "id": 704235, "postDate": "2019-12-27T08:02:16.677Z", "content": "Under-promise and over-deliver
", "rawMarkdown": "Under-promise and over-deliver", "votes": 5, "replies": [ { "id": 704259, "postDate": "2019-12-27T08:22:33.240Z", "content": "Kaggle is over-promising and under-delivering.
", "rawMarkdown": "Kaggle is over-promising and under-delivering.", "votes": 5 }, { "id": 704376, "postDate": "2019-12-27T12:02:18.100Z", "content": "That's a red flag in project delivery.\nIt is extremely necessary to learn their lessons in this competition on how to improve their risk, issue and scope change management effectively and efficiently.
", "rawMarkdown": "That's a red flag in project delivery.\nIt is extremely necessary to learn their lessons in this competition on how to improve their risk, issue and scope change management effectively and efficiently." } ] }, { "id": 702584, "postDate": "2019-12-24T21:45:11.307Z", "content": "![\"๐](\"https://www.cs.vu.nl/~frankh/dilbert/strategy.jpg\")
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1788308%2Fd051a3cd6fdb990e98a29b74cd7b0aba%2Fgiphy%20(1\")
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1894505%2F6f6947cf882899307e7a6f8d83f114ee%2Fgiphy.gif?generation=1577459240344968&alt=media\")
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F3015190%2F065ac3f8f3d2a57d8ec48e0b96bc82ed%2Fdt_c110513.gif?generation=1577457663671782&alt=media\")
stalking the lb like\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F933480%2F0e2773fe2bc6d1e463dc241d13df013a%2Flb.jpg?generation=1577479713589019&alt=media\")
@addisonhoward, any update ? \n26th pst deadline has crossed now
", "rawMarkdown": "@addisonhoward, any update ? \n26th pst deadline has crossed now", "votes": 3 }, { "id": 702500, "postDate": "2019-12-24T19:00:06.903Z", "content": "@addisonhoward \nCould you explain, what is the reason for 40 000 rescores if, for example, our team selected 2 submissions?\nThus, no one should care about other submissions, especially wasting time, resources and damaging ecology of the planet.
\n\nI expect that there are 5%, 10% or a bit more which didn't take care and chose 2 submissions, but as I know, the only 2 best scores in public should be selected for them.
\n\nSo, simple calculations give me 3500 teams x 2 submissions = 7000 rescoring.
\n\nIs it so important to have some additional checks out of that? Or some teams have broken the rules and you try to identify that?
\n\nThanks for answering.
", "rawMarkdown": "@addisonhoward \nCould you explain, what is the reason for 40 000 rescores if, for example, our team selected 2 submissions?\nThus, no one should care about other submissions, especially wasting time, resources and damaging ecology of the planet. \n\nI expect that there are 5%, 10% or a bit more which didn't take care and chose 2 submissions, but as I know, the only 2 best scores in public should be selected for them. \n\nSo, simple calculations give me 3500 teams x 2 submissions = 7000 rescoring. \n\nIs it so important to have some additional checks out of that? Or some teams have broken the rules and you try to identify that? \n\nThanks for answering. ", "votes": 3, "replies": [ { "id": 702634, "postDate": "2019-12-25T00:15:23.630Z", "rawMarkdown": "", "votes": 1, "isDeleted": true }, { "id": 702635, "postDate": "2019-12-25T00:19:34.277Z", "content": "@smasoudn \n2 submission selection is a solid rule. Thus, two submisisons should be validated on the private dataset. And only those who didn't select 2 best submissions might need to rescore. I didn't expect that it requires 40 000 rescores but less.
", "rawMarkdown": "@smasoudn \n2 submission selection is a solid rule. Thus, two submisisons should be validated on the private dataset. And only those who didn't select 2 best submissions might need to rescore. I didn't expect that it requires 40 000 rescores but less.", "votes": 2 }, { "id": 703762, "postDate": "2019-12-26T15:48:21.507Z", "content": "EDITED: The public test set is not changing. Any unused rows in the private set are simply ignored, not made public.
", "rawMarkdown": "EDITED: The public test set is not changing. Any unused rows in the private set are simply ignored, not made public.", "votes": 1 }, { "id": 703790, "postDate": "2019-12-26T16:11:32.357Z", "content": "So that means the teams that chose their two submissions might have a disadvantage because those may not be the 'best' submissions on new public test set and they wouldn't be able to select their best submissions even if they wanted to.
", "rawMarkdown": "So that means the teams that chose their two submissions might have a disadvantage because those may not be the 'best' submissions on new public test set and they wouldn't be able to select their best submissions even if they wanted to.", "votes": 13 }, { "id": 703795, "postDate": "2019-12-26T16:27:38.250Z", "content": "I think it's definitely necessary to allow each team to reselect their submissions. Those who didn't manage to choose their submissions because of too long waiting time at the last moment will benefit from this too.
", "rawMarkdown": "I think it's definitely necessary to allow each team to reselect their submissions. Those who didn't manage to choose their submissions because of too long waiting time at the last moment will benefit from this too." }, { "id": 703796, "postDate": "2019-12-26T16:28:17.143Z", "content": "@addisonhoward I think it not fair for the teams that chose 2 submissions because that could not be the best submisions of them.
", "rawMarkdown": "@addisonhoward I think it not fair for the teams that chose 2 submissions because that could not be the best submisions of them.", "votes": 3 }, { "id": 703797, "postDate": "2019-12-26T16:33:18.653Z", "content": "re-select is definitely unfair. There might be someone who is done with this competition and on holiday. He did everything before deadline but he may miss this re-select period. The only logical solution is to select the best 2 submission according to the previous LB. This is what anyone who didn't select was aware of.
", "rawMarkdown": "re-select is definitely unfair. There might be someone who is done with this competition and on holiday. He did everything before deadline but he may miss this re-select period. The only logical solution is to select the best 2 submission according to the previous LB. This is what anyone who didn't select was aware of.", "votes": 6 }, { "id": 703802, "postDate": "2019-12-26T16:38:23.420Z", "content": "Agree with @divrikwicky, the most sensible and fair solution is to select best two submissions based on the original public LB for those who did not select beforehand.
", "rawMarkdown": "Agree with @divrikwicky, the most sensible and fair solution is to select best two submissions based on the original public LB for those who did not select beforehand.", "votes": 4 }, { "id": 703805, "postDate": "2019-12-26T16:40:42.167Z", "content": "\n\n\"So that means the teams that chose their two submissions might have a disadvantage because those may not be the 'best' submissions on new public test set and they wouldn't be able to select their best submissions even if they wanted to.\"\n- Agree. Good point!
\n\n@jielu0728 \n\"I think it's definitely necessary to allow each team to reselect their submissions. Those who didn't manage to choose their submissions because of too long waiting time at the last moment will benefit from this too.\"\n- It's not good, because someone already selected and it's done. And for others who did not there are old rules where according to the first public LB the 2 best will be selected.
\n\n@divrikwicky \n\"re-select is definitely unfair. There might be someone who is done with this competition and on holiday. He did everything before the deadline but he may miss this re-select period. The only logical solution is to select the best 2 submissions according to the previous LB. This is what anyone who didn't select was aware of.\"
\n\nThe competition will become a mess if the rules will be applied differently and give some advantage to one of the people (especially those who didn't make their job) with respect to others like us who did their job and selected 2 submissions.
\n\nAnyway, it should be based on first LB standings, even if data was changed it's not our point because the competition host didn't care about that before we optimized our selection and it was based on that data which was existed before all these rescorings.
", "rawMarkdown": "@rohanrao \n\n\"So that means the teams that chose their two submissions might have a disadvantage because those may not be the 'best' submissions on new public test set and they wouldn't be able to select their best submissions even if they wanted to.\"\n- Agree. Good point!\n\n@jielu0728 \n\"I think it's definitely necessary to allow each team to reselect their submissions. Those who didn't manage to choose their submissions because of too long waiting time at the last moment will benefit from this too.\"\n- It's not good, because someone already selected and it's done. And for others who did not there are old rules where according to the first public LB the 2 best will be selected. \n\n@divrikwicky \n\"re-select is definitely unfair. There might be someone who is done with this competition and on holiday. He did everything before the deadline but he may miss this re-select period. The only logical solution is to select the best 2 submissions according to the previous LB. This is what anyone who didn't select was aware of.\"\n\n- Agree with that, as I said to @jielu0728 \"for others who did not there are old rules where according to first public LB the 2 best will be selected. \"\n\nThe competition will become a mess if the rules will be applied differently and give some advantage to one of the people (especially those who didn't make their job) with respect to others like us who did their job and selected 2 submissions.\n\nAnyway, it should be based on first LB standings, even if data was changed it's not our point because the competition host didn't care about that before we optimized our selection and it was based on that data which was existed before all these rescorings. ", "votes": 4 }, { "id": 703810, "postDate": "2019-12-26T16:54:05.843Z", "content": "...and teams who didn't chose the two submissions may feel it as disadvantage too :-). I simply don't want my final submission to be chosen automagically based on something I never seen. If I understand it right, the new public LB dataset will be more polluted by leak data than the previous one. I was too lazy to hardcopy sites 4 & 15 as ground truth into my submit, but I used them in training. So, for me, re-scoring on new public LB will slightly prefer overfiting systems.
\nIn fact I don't care as much about it. What I want the most is this competition to end as soon as possible.
Autoselecting 2 best submissions based on the new LB is very unfair.\nI agree with @divrikwicky
", "rawMarkdown": "Autoselecting 2 best submissions based on the new LB is very unfair.\nI agree with @divrikwicky ", "votes": 4 }, { "id": 703849, "postDate": "2019-12-26T17:56:28.530Z", "content": "\n\n\nthe most sensible and fair solution is to select best two submissions based on the original public LB
\n
I bet that this is hard to implement. Kaggle probably don't have a simple way to redirect the choice of unselected submissions to the \"old\"' public LB instead of the current public LB.
", "rawMarkdown": "> the most sensible and fair solution is to select best two submissions based on the original public LB\n\nI bet that this is hard to implement. Kaggle probably don't have a simple way to redirect the choice of unselected submissions to the \"old\"' public LB instead of the current public LB. ", "votes": 2 }, { "id": 703852, "postDate": "2019-12-26T18:01:05.517Z", "rawMarkdown": "", "isDeleted": true }, { "id": 703853, "postDate": "2019-12-26T18:02:09.337Z", "content": "\n\n\nI bet that this is hard to implement. Kaggle probably don't have a simple way to redirect the choice of unselected submissions to the \"old\"' public LB instead of the current public LB.
\n
Yes, I'm pretty sure all this changing of datasets / scores is a new adhoc task for a Kaggle competition (maybe I'm wrong but I haven't seen on any other competition) and that's the only reason its worth giving the Kaggle admins the benefit of doubt for all the delays (with the intention of making the best out of this relatively poorly organized competition) to ensure the final results and models are as clean as possible.
", "rawMarkdown": ">I bet that this is hard to implement. Kaggle probably don't have a simple way to redirect the choice of unselected submissions to the \"old\"' public LB instead of the current public LB.\n\nYes, I'm pretty sure all this changing of datasets / scores is a new adhoc task for a Kaggle competition (maybe I'm wrong but I haven't seen on any other competition) and that's the only reason its worth giving the Kaggle admins the benefit of doubt for all the delays (with the intention of making the best out of this relatively poorly organized competition) to ensure the final results and models are as clean as possible.", "votes": 1 }, { "id": 703854, "postDate": "2019-12-26T18:02:23.120Z", "rawMarkdown": "", "isDeleted": true }, { "id": 703901, "postDate": "2019-12-26T19:23:27.740Z", "content": "The public test set isn't changing. Rows dropped from the private scoring will just be ignored.
", "rawMarkdown": "The public test set isn't changing. Rows dropped from the private scoring will just be ignored.", "votes": 10 }, { "id": 703929, "postDate": "2019-12-26T20:32:56.790Z", "content": "\n\n\nThe public test set isn't changing.
\n
This contradicts what Addison wrote previously.
\n\nThen public score will not change, and you don't need to rescore all submissions. You only need to rescore the selected ones or the top two public score ones. At most 2 subs per team does not add up to 40k rescore. ;)
", "rawMarkdown": "> The public test set isn't changing. \n\nThis contradicts what Addison wrote previously.\n\nThen public score will not change, and you don't need to rescore all submissions. You only need to rescore the selected ones or the top two public score ones. At most 2 subs per team does not add up to 40k rescore. ;)\n", "votes": 2 }, { "id": 703932, "postDate": "2019-12-26T20:40:54.910Z", "content": "Sohier's comment is the ground truth - I spoke too soon.
", "rawMarkdown": "Sohier's comment is the ground truth - I spoke too soon.", "votes": 4 }, { "id": 703934, "postDate": "2019-12-26T20:44:21.990Z", "content": "@cpmpml we came back with Kaggle team to what was discussed before ))) \n3500 teams = 7000 submissions.
\n\nI still don't understand where the 40 000 rescorings came from.
\n\nFor example, We have a slight less than 100 submissions and we selected 2 as by the rules. \nBut I see that 5 submissions from early-stage were rescored (which form our dummy posituioning outside some listings of 1877(~3500) teams). But I don't understand why it should be done. That means that they rescore all submissions that were made, this can be 40 000, and of course not good for ecology to waste electricity, time of people, GPU/CPU time of computers.
\n\nSomeone in previous competitions already gave an estimation of energy wasted/consumed due to non-well organized competitions. And it was enormous.
", "rawMarkdown": "@cpmpml we came back with Kaggle team to what was discussed before ))) \n3500 teams = 7000 submissions.\n\nI still don't understand where the 40 000 rescorings came from. \n\nFor example, We have a slight less than 100 submissions and we selected 2 as by the rules. \nBut I see that 5 submissions from early-stage were rescored (which form our dummy posituioning outside some listings of 1877(~3500) teams). But I don't understand why it should be done. That means that they rescore all submissions that were made, this can be 40 000, and of course not good for ecology to waste electricity, time of people, GPU/CPU time of computers. \n\nSomeone in previous competitions already gave an estimation of energy wasted/consumed due to non-well organized competitions. And it was enormous. ", "votes": 1 }, { "id": 703948, "postDate": "2019-12-26T21:12:39.450Z", "content": "Kaggle is rescoring all submissions it seems. I can see lots of my rescores returned an error and only one (at least for now) returned a number. \nNot sure what's going on, I thought applying a filter on a row number column was easy enough and some packages can read data and score very fast...
", "rawMarkdown": "Kaggle is rescoring all submissions it seems. I can see lots of my rescores returned an error and only one (at least for now) returned a number. \nNot sure what's going on, I thought applying a filter on a row number column was easy enough and some packages can read data and score very fast...", "votes": 1 } ] }, { "id": 704393, "postDate": "2019-12-27T12:28:55.343Z", "content": "In the soviet us competition scores you
", "rawMarkdown": "In the soviet us competition scores you", "votes": 4, "replies": [ { "id": 704438, "postDate": "2019-12-27T13:39:55.557Z", "content": "Google Translate should be banned ๐
", "rawMarkdown": "Google Translate should be banned ๐ ", "votes": 3 }, { "id": 704533, "postDate": "2019-12-27T15:35:39.860Z", "content": "Dude, we already got our medals
", "rawMarkdown": "Dude, we already got our medals", "votes": 1 } ] }, { "id": 704282, "postDate": "2019-12-27T09:10:52.397Z", "content": "Yesterday the LB changed a bit. It is weird, it was supposed to stop until they finish the run on the backup system. I wonder if everything is messed up and the person who can fix it is on holiday.
", "rawMarkdown": "Yesterday the LB changed a bit. It is weird, it was supposed to stop until they finish the run on the backup system. I wonder if everything is messed up and the person who can fix it is on holiday.", "votes": 4, "replies": [ { "id": 704441, "postDate": "2019-12-27T13:44:08.343Z", "content": "\n\n\nI wonder if everything is messed up and the person who can fix it is on holiday.
\n
@divrikwicky if so, it would not be the 1st time :-)
", "rawMarkdown": "> I wonder if everything is messed up and the person who can fix it is on holiday.\n\n@divrikwicky if so, it would not be the 1st time :-)", "votes": 1 }, { "id": 704449, "postDate": "2019-12-27T13:55:21.003Z", "content": "we have seen atleast one competition where the wait time was longer than this. Last Santander competition, we had to wait for 2 weeks before final pvt. LB was released. I won't be surprised if this one takes the same amount of time.
", "rawMarkdown": "we have seen atleast one competition where the wait time was longer than this. Last Santander competition, we had to wait for 2 weeks before final pvt. LB was released. I won't be surprised if this one takes the same amount of time.", "votes": 1 }, { "id": 704460, "postDate": "2019-12-27T14:09:48.933Z", "content": "At least at Santander we could see the preliminary standings, but now it's a madness to just sit and wait
", "rawMarkdown": "At least at Santander we could see the preliminary standings, but now it's a madness to just sit and wait", "votes": 1 }, { "id": 704465, "postDate": "2019-12-27T14:14:40.717Z", "content": "@stanislavblinov yes, that is why I had asked the organizers in their first LB Update post that if there would be a preliminary LB like Santander. At that time they had confirmed that there would be no preliminary LB...but I guess the issue with leaked data and the proposed way to deal with it has caused lot of issues. In Satander people were only looking for 'Magic' which wasn't a leak ๐ ...this one is way worse
", "rawMarkdown": "@stanislavblinov yes, that is why I had asked the organizers in their first LB Update post that if there would be a preliminary LB like Santander. At that time they had confirmed that there would be no preliminary LB...but I guess the issue with leaked data and the proposed way to deal with it has caused lot of issues. In Satander people were only looking for 'Magic' which wasn't a leak ๐ ...this one is way worse", "votes": 2 } ] }, { "id": 704245, "postDate": "2019-12-27T08:10:42.857Z", "content": "Is your calendar a solar calendar? Is it the lunar calendar? Or the lunisolar calendar? ๐
", "rawMarkdown": "Is your calendar a solar calendar? Is it the lunar calendar? Or the lunisolar calendar? ๐ ", "votes": 4 }, { "id": 702658, "postDate": "2019-12-25T01:16:36.893Z", "content": "I think that suspected users can be asked to submit code for inspection, especially in the gold zone
", "rawMarkdown": "I think that suspected users can be asked to submit code for inspection, especially in the gold zone", "votes": 2 }, { "id": 704256, "postDate": "2019-12-27T08:20:09.467Z", "content": "I just heard that many people in U.S. take a holiday from 12/25(Christmas Day) to 1/1(New Year's Day). They too? \nOMG๐
", "rawMarkdown": "I just heard that many people in U.S. take a holiday from 12/25(Christmas Day) to 1/1(New Year's Day). They too? \nOMG๐", "votes": 1 }, { "id": 704240, "postDate": "2019-12-27T08:07:37.367Z", "content": "> We will post another update or the scores on the PST afternoon of the 26th.
\n\n๐
", "rawMarkdown": "> We will post another update or the scores on the PST afternoon of the 26th.\n\n๐ ", "votes": 1, "replies": [ { "id": 704255, "postDate": "2019-12-27T08:19:37.677Z", "content": "Did anyone commit on a year ?
", "rawMarkdown": "Did anyone commit on a year ?", "votes": 3 }, { "id": 704277, "postDate": "2019-12-27T09:05:38.043Z", "content": "Maybe January 26th ? ๐
", "rawMarkdown": "Maybe January 26th ? ๐ ", "votes": 2 } ] }, { "id": 704181, "postDate": "2019-12-27T06:23:59.427Z", "content": "what about results ? can we expect this week ? or next year ? let us know
", "rawMarkdown": "what about results ? can we expect this week ? or next year ? let us know ", "votes": 1, "replies": [ { "id": 704344, "postDate": "2019-12-27T11:00:58.490Z", "content": "Next decade ๐
", "rawMarkdown": "Next decade ๐", "votes": 3 }, { "id": 704651, "postDate": "2019-12-27T19:28:19.323Z", "content": "hny btw lol
", "rawMarkdown": "hny btw lol", "votes": 2 } ] }, { "id": 701917, "postDate": "2019-12-24T03:41:43.603Z", "content": "Ready to help Kaggle kick off a milk n cookies operational optimization at anytime :-)
", "rawMarkdown": "Ready to help Kaggle kick off a milk n cookies operational optimization at anytime :-)", "votes": 1 }, { "id": 704487, "postDate": "2019-12-27T14:35:19.853Z", "content": "![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F848555%2F17fd43b4f68efee9cb7a07b38bfb79bd%2FBS3HxdTCEAAK4gN.png?generation=1577457315856744&alt=media\")
Actually, just now it crossed the 2019/12/26th ,it has become the 2019/12/27th,but the update or leapboard is not come,maybe something happened!\nCould you please come to tell us what happened@Addison Howard
", "rawMarkdown": "Actually, just now it crossed the 2019/12/26th ,it has become the 2019/12/27th,but the update or leapboard is not come,maybe something happened!\nCould you please come to tell us what happened[@Addison Howard](https://www.kaggle.com/addisonhoward)", "votes": 2, "replies": [ { "id": 704258, "postDate": "2019-12-27T08:21:30.070Z", "content": "Did any of your submission has any score in it?
\n\nMine is nothing which means it has stuck somewhere.
\n\nNow I got no more pc and phone to use.
", "rawMarkdown": "Did any of your submission has any score in it?\n\nMine is nothing which means it has stuck somewhere. \n\nNow I got no more pc and phone to use. ", "votes": 1 }, { "id": 704285, "postDate": "2019-12-27T09:15:05.677Z", "content": "@klloke The organizers said that they are running a new scoring program in the background to make the process faster. and scores are not being synchronized onto the LB, what you see is just the results calculated by a former program, which must have been stopped by them now
", "rawMarkdown": "@klloke The organizers said that they are running a new scoring program in the background to make the process faster. and scores are not being synchronized onto the LB, what you see is just the results calculated by a former program, which must have been stopped by them now", "votes": 1 } ] }, { "id": 704706, "postDate": "2019-12-27T21:49:39.117Z", "content": "![\"\"](\"http://phdcomics.com/comics/archive/phd020604s.gif\")
\n๐
Thanks a lot!!
", "rawMarkdown": "Thanks a lot!!" } ], "comments": [ { "id": 702505, "author_name": "Mukharbek Organokov", "author_url": "", "post_date": "2019-12-24T19:10:11.857000", "content": ".jpeg?generation=1577214609571546&alt=media)
Haha, good one!
", "votes": 2, "replies": [] } ] }, { "id": 702031, "author_name": "Ahmet Erdem", "author_url": "", "post_date": "2019-12-24T07:17:26.787000", "content": "I think he skipped because you weren't good kids. Santa hates leaky and unorganized competitions:) Thanks for the update anyway.
", "votes": 23, "replies": [] }, { "id": 704088, "author_name": "Manraj Singh", "author_url": "", "post_date": "2019-12-27T03:28:05.850000", "content": "\n\n\nWe will post another update or the scores on the PST afternoon of the 26th.
\n
Shouldn't there have been an update by now? It's already 7:30pm (PST)? @addisonhoward
", "votes": 18, "replies": [ { "id": 704120, "author_name": "Yoshi", "author_url": "", "post_date": "2019-12-27T04:41:40.613000", "content": "Probably he is on Christmas holiday๐
", "votes": 4, "replies": [] }, { "id": 704124, "author_name": "Yokoichi", "author_url": "", "post_date": "2019-12-27T04:46:49.957000", "content": "It โs a long long Christmas holiday:)
", "votes": 3, "replies": [] }, { "id": 704125, "author_name": "Linyan", "author_url": "", "post_date": "2019-12-27T04:47:50.303000", "content": "Actually,I also has this question,when can it be posted . afternoon? But it already night
", "votes": 3, "replies": [] }, { "id": 704126, "author_name": "Yust", "author_url": "", "post_date": "2019-12-27T04:50:34.043000", "content": "maybe, 2020/12/26?
", "votes": 4, "replies": [] }, { "id": 704128, "author_name": "Sugawarya", "author_url": "", "post_date": "2019-12-27T04:53:23.053000", "content": "his \"afternoon\" means \"till 23:59\" ?
", "votes": 2, "replies": [] }, { "id": 704148, "author_name": "Yiemon773", "author_url": "", "post_date": "2019-12-27T05:32:17.120000", "content": "My year-end is... \n\nfor hour in range(24):
\n press F5
\n
@yoichi7yamakawa this is the funniest thing I've seen today!
", "votes": 1, "replies": [] }, { "id": 704454, "author_name": "Addison Howard", "author_url": "", "post_date": "2019-12-27T14:02:10.807000", "content": "Unfortunately the update is that we're still working on it... We've finished a LOT of the rescores, but the rest are still underway. I promise you we are working hard on it, investigate edge cases, and doing what we can to get the results back to you both quickly and with quality.
\n\nThanks for the patience!
", "votes": 4, "replies": [] }, { "id": 704458, "author_name": "Stanislav Blinov", "author_url": "", "post_date": "2019-12-27T14:07:52.990000", "content": "@addisonhoward any guesses when it could be finished? Is it today/tomorrow or more likely next week?
", "votes": 1, "replies": [] }, { "id": 704461, "author_name": "Linyan", "author_url": "", "post_date": "2019-12-27T14:11:04.507000", "content": "same question,could you please tell us the exact time when it can be finished and when the outcome can be published. \nI hope this can be true
", "votes": 2, "replies": [] }, { "id": 704464, "author_name": "Addison Howard", "author_url": "", "post_date": "2019-12-27T14:14:24.887000", "content": "We're hopeful it will be today! But most of the timing is out of our control, so we're actively monitoring, troubleshooting, and trying to figure out what the hangups are.
\n\nWe'd like to have it wrapped up prior to the weekend just as much as you!
", "votes": 7, "replies": [] }, { "id": 704468, "author_name": "Ahmet Erdem", "author_url": "", "post_date": "2019-12-27T14:18:22.763000", "content": "Estimating the time shouldn't be so difficult. You know the time needed for each submission and how much resource you have. I really appreciate your effort but Kaggle can actually be more transparent. The problem could be solved in 3500x2x5/60/num_cores hours. If Kaggle has 500 cores, it is roughly 1 hour. I feel like there is something else going on.
", "votes": 1, "replies": [] }, { "id": 704747, "author_name": "Hiroyuki Namba", "author_url": "", "post_date": "2019-12-28T00:25:37.633000", "content": "@addisonhoward Thank you for information. I really appreciate your effort. Please take a long time for quality of LB including next year if needed.\nBut I'm sad if\n-Change after opening private LB due to some mistake in rescoring\n-Only rescoring, no check on cheaters\n-Repeating over-promising such as 24๏ฝ48 business hours and update 26th afternoon
", "votes": 2, "replies": [] } ] }, { "id": 704163, "author_name": "Vopani", "author_url": "", "post_date": "2019-12-27T05:56:45.773000", "content": "ASHRAE competitors be like:\n
I just bought a computer last week๐ญ
", "votes": 6, "replies": [] }, { "id": 704186, "author_name": "Yust", "author_url": "", "post_date": "2019-12-27T06:29:24.797000", "content": "if computer is broken, I use kaggle_kernel so its alright๐
", "votes": 4, "replies": [] }, { "id": 704196, "author_name": "Manraj Singh", "author_url": "", "post_date": "2019-12-27T06:40:50.967000", "content": "I'm replying from my phone now. PC-less
", "votes": 1, "replies": [] }, { "id": 704211, "author_name": "Stanislav Blinov", "author_url": "", "post_date": "2019-12-27T07:18:46.300000", "content": "
This competition has been a frustration for me for multiple reasons. \nI think leak cases could have been handled better, sooner. \nCheaters, probers could have been addressed sooner.\nAnd when we think at last the competition is over, waiting days, or probably weeks for results added insult to injury.\nMaybe simply prioritizing selected submits of teams if they have, or simply going late from early could have been a relief. Most of -serious- teams handpick their submits. We have only 2 rescores both of irrelevant and early tries.\nOn the other hand, I realize this is a demanding community and we will find a reason to whine about anyways,\nso good luck and happy new year to you all :)\n.
", "votes": 14, "replies": [ { "id": 703813, "author_name": "Jie Lu", "author_url": "", "post_date": "2019-12-26T17:01:27.497000", "content": "yeah we need private LB, not \"Leaderboard Reveal - Update 4\" ๐
", "votes": 6, "replies": [] } ] }, { "id": 703774, "author_name": "Caesar Lupum", "author_url": "", "post_date": "2019-12-26T15:57:34.647000", "content": "
I hope you are trying to catch those who identified a public data source and did not disclose it properly if it wasn't already disclosed. This is easy to find: look for perfect prediction...
\n\nI feel the agony of all the participants in the meantime...
", "votes": 12, "replies": [ { "id": 702188, "author_name": "Tim Yee", "author_url": "", "post_date": "2019-12-24T11:15:41.590000", "content": "I agree, but what happens if they trained on it instead? their scores would be much better, but would not result in perfect predictions. Or what happens if they blended their model predictions with perfect predictions? Reverse out the blend by dividing by ground truth labels and see if there is a consistent coefficient?
", "votes": 1, "replies": [] }, { "id": 702191, "author_name": "Ahmet Erdem", "author_url": "", "post_date": "2019-12-24T11:22:41.313000", "content": "They may avoid making perfect predictions. I would rank teams site by site and further investigate the teams which have much higher rank in some sites compared to other sites.
", "votes": 5, "replies": [] }, { "id": 702201, "author_name": "Stanislav Blinov", "author_url": "", "post_date": "2019-12-24T11:41:00.187000", "content": "Time to launch another competition - ASHRAE leak users detection. The winner gets his LB score in this one. In 3 months :D
", "votes": 6, "replies": [] }, { "id": 702275, "author_name": "eggachecat", "author_url": "", "post_date": "2019-12-24T13:37:32.637000", "content": "So there will be s private sharing on private sharing and a leakage on leakage
", "votes": 1, "replies": [] }, { "id": 702305, "author_name": "Sia", "author_url": "", "post_date": "2019-12-24T14:25:07.070000", "content": "Identifying severe difference of model performance on various site-ids is not that complicated. That's a great idea @cpmpml.
", "votes": 1, "replies": [] }, { "id": 702311, "author_name": "CPMP", "author_url": "", "post_date": "2019-12-24T14:33:08.217000", "content": "Many cheaters are lazy... That's why they are caught.
", "votes": 6, "replies": [] }, { "id": 702336, "author_name": "Vopani", "author_url": "", "post_date": "2019-12-24T15:11:44.727000", "content": "Another clever idea by @gunesevitan to tag leaked sites is to check submissions that gave significant boost to a team's score and compare predictions against previous best to identify which sites give that boost to find likely leak.
", "votes": 2, "replies": [] }, { "id": 702412, "author_name": "YaGana Sheriff-Hussaini", "author_url": "", "post_date": "2019-12-24T16:49:56.150000", "content": "\n\n\nMany cheaters are lazyโฆ That's why they are caught.
\n
Yes. Because to avoid getting caught they would have to work hard which defeats the purpose of cheating in the 1st place. Hence lazy approach to the project.
", "votes": 1, "replies": [] } ] }, { "id": 702027, "author_name": "Stanislav Blinov", "author_url": "", "post_date": "2019-12-24T07:13:11.257000", "content": "
Under-promise and over-deliver
", "votes": 5, "replies": [ { "id": 704259, "author_name": "Vopani", "author_url": "", "post_date": "2019-12-27T08:22:33.240000", "content": "Kaggle is over-promising and under-delivering.
", "votes": 5, "replies": [] }, { "id": 704376, "author_name": "FGPC", "author_url": "", "post_date": "2019-12-27T12:02:18.100000", "content": "That's a red flag in project delivery.\nIt is extremely necessary to learn their lessons in this competition on how to improve their risk, issue and scope change management effectively and efficiently.
", "votes": 0, "replies": [] } ] }, { "id": 702584, "author_name": "s.s.o", "author_url": "", "post_date": "2019-12-24T21:45:11.307000", "content": "
stalking the lb like\n
@addisonhoward, any update ? \n26th pst deadline has crossed now
", "votes": 3, "replies": [] }, { "id": 702500, "author_name": "Mukharbek Organokov", "author_url": "", "post_date": "2019-12-24T19:00:06.903000", "content": "@addisonhoward \nCould you explain, what is the reason for 40 000 rescores if, for example, our team selected 2 submissions?\nThus, no one should care about other submissions, especially wasting time, resources and damaging ecology of the planet.
\n\nI expect that there are 5%, 10% or a bit more which didn't take care and chose 2 submissions, but as I know, the only 2 best scores in public should be selected for them.
\n\nSo, simple calculations give me 3500 teams x 2 submissions = 7000 rescoring.
\n\nIs it so important to have some additional checks out of that? Or some teams have broken the rules and you try to identify that?
\n\nThanks for answering.
", "votes": 3, "replies": [ { "id": 702634, "author_name": "", "author_url": "", "post_date": "2019-12-25T00:15:23.630000", "content": "", "votes": 1, "replies": [] }, { "id": 702635, "author_name": "Mukharbek Organokov", "author_url": "", "post_date": "2019-12-25T00:19:34.277000", "content": "@smasoudn \n2 submission selection is a solid rule. Thus, two submisisons should be validated on the private dataset. And only those who didn't select 2 best submissions might need to rescore. I didn't expect that it requires 40 000 rescores but less.
", "votes": 2, "replies": [] }, { "id": 703762, "author_name": "Addison Howard", "author_url": "", "post_date": "2019-12-26T15:48:21.507000", "content": "EDITED: The public test set is not changing. Any unused rows in the private set are simply ignored, not made public.
", "votes": 1, "replies": [] }, { "id": 703790, "author_name": "Vopani", "author_url": "", "post_date": "2019-12-26T16:11:32.357000", "content": "So that means the teams that chose their two submissions might have a disadvantage because those may not be the 'best' submissions on new public test set and they wouldn't be able to select their best submissions even if they wanted to.
", "votes": 13, "replies": [] }, { "id": 703795, "author_name": "Jie Lu", "author_url": "", "post_date": "2019-12-26T16:27:38.250000", "content": "I think it's definitely necessary to allow each team to reselect their submissions. Those who didn't manage to choose their submissions because of too long waiting time at the last moment will benefit from this too.
", "votes": 0, "replies": [] }, { "id": 703796, "author_name": "L3viEvil", "author_url": "", "post_date": "2019-12-26T16:28:17.143000", "content": "@addisonhoward I think it not fair for the teams that chose 2 submissions because that could not be the best submisions of them.
", "votes": 3, "replies": [] }, { "id": 703797, "author_name": "Ahmet Erdem", "author_url": "", "post_date": "2019-12-26T16:33:18.653000", "content": "re-select is definitely unfair. There might be someone who is done with this competition and on holiday. He did everything before deadline but he may miss this re-select period. The only logical solution is to select the best 2 submission according to the previous LB. This is what anyone who didn't select was aware of.
", "votes": 6, "replies": [] }, { "id": 703802, "author_name": "Vopani", "author_url": "", "post_date": "2019-12-26T16:38:23.420000", "content": "Agree with @divrikwicky, the most sensible and fair solution is to select best two submissions based on the original public LB for those who did not select beforehand.
", "votes": 4, "replies": [] }, { "id": 703805, "author_name": "Mukharbek Organokov", "author_url": "", "post_date": "2019-12-26T16:40:42.167000", "content": "\n\n\"So that means the teams that chose their two submissions might have a disadvantage because those may not be the 'best' submissions on new public test set and they wouldn't be able to select their best submissions even if they wanted to.\"\n- Agree. Good point!
\n\n@jielu0728 \n\"I think it's definitely necessary to allow each team to reselect their submissions. Those who didn't manage to choose their submissions because of too long waiting time at the last moment will benefit from this too.\"\n- It's not good, because someone already selected and it's done. And for others who did not there are old rules where according to the first public LB the 2 best will be selected.
\n\n@divrikwicky \n\"re-select is definitely unfair. There might be someone who is done with this competition and on holiday. He did everything before the deadline but he may miss this re-select period. The only logical solution is to select the best 2 submissions according to the previous LB. This is what anyone who didn't select was aware of.\"
\n\nThe competition will become a mess if the rules will be applied differently and give some advantage to one of the people (especially those who didn't make their job) with respect to others like us who did their job and selected 2 submissions.
\n\nAnyway, it should be based on first LB standings, even if data was changed it's not our point because the competition host didn't care about that before we optimized our selection and it was based on that data which was existed before all these rescorings.
", "votes": 4, "replies": [] }, { "id": 703810, "author_name": "Josef Slavicek", "author_url": "", "post_date": "2019-12-26T16:54:05.843000", "content": "...and teams who didn't chose the two submissions may feel it as disadvantage too :-). I simply don't want my final submission to be chosen automagically based on something I never seen. If I understand it right, the new public LB dataset will be more polluted by leak data than the previous one. I was too lazy to hardcopy sites 4 & 15 as ground truth into my submit, but I used them in training. So, for me, re-scoring on new public LB will slightly prefer overfiting systems.
\nIn fact I don't care as much about it. What I want the most is this competition to end as soon as possible.
Autoselecting 2 best submissions based on the new LB is very unfair.\nI agree with @divrikwicky
", "votes": 4, "replies": [] }, { "id": 703849, "author_name": "CPMP", "author_url": "", "post_date": "2019-12-26T17:56:28.530000", "content": "\n\n\nthe most sensible and fair solution is to select best two submissions based on the original public LB
\n
I bet that this is hard to implement. Kaggle probably don't have a simple way to redirect the choice of unselected submissions to the \"old\"' public LB instead of the current public LB.
", "votes": 2, "replies": [] }, { "id": 703852, "author_name": "", "author_url": "", "post_date": "2019-12-26T18:01:05.517000", "content": "", "votes": 0, "replies": [] }, { "id": 703853, "author_name": "Vopani", "author_url": "", "post_date": "2019-12-26T18:02:09.337000", "content": "\n\n\nI bet that this is hard to implement. Kaggle probably don't have a simple way to redirect the choice of unselected submissions to the \"old\"' public LB instead of the current public LB.
\n
Yes, I'm pretty sure all this changing of datasets / scores is a new adhoc task for a Kaggle competition (maybe I'm wrong but I haven't seen on any other competition) and that's the only reason its worth giving the Kaggle admins the benefit of doubt for all the delays (with the intention of making the best out of this relatively poorly organized competition) to ensure the final results and models are as clean as possible.
", "votes": 1, "replies": [] }, { "id": 703854, "author_name": "", "author_url": "", "post_date": "2019-12-26T18:02:23.120000", "content": "", "votes": 0, "replies": [] }, { "id": 703901, "author_name": "Sohier Dane", "author_url": "", "post_date": "2019-12-26T19:23:27.740000", "content": "The public test set isn't changing. Rows dropped from the private scoring will just be ignored.
", "votes": 10, "replies": [] }, { "id": 703929, "author_name": "CPMP", "author_url": "", "post_date": "2019-12-26T20:32:56.790000", "content": "\n\n\nThe public test set isn't changing.
\n
This contradicts what Addison wrote previously.
\n\nThen public score will not change, and you don't need to rescore all submissions. You only need to rescore the selected ones or the top two public score ones. At most 2 subs per team does not add up to 40k rescore. ;)
", "votes": 2, "replies": [] }, { "id": 703932, "author_name": "Addison Howard", "author_url": "", "post_date": "2019-12-26T20:40:54.910000", "content": "Sohier's comment is the ground truth - I spoke too soon.
", "votes": 4, "replies": [] }, { "id": 703934, "author_name": "Mukharbek Organokov", "author_url": "", "post_date": "2019-12-26T20:44:21.990000", "content": "@cpmpml we came back with Kaggle team to what was discussed before ))) \n3500 teams = 7000 submissions.
\n\nI still don't understand where the 40 000 rescorings came from.
\n\nFor example, We have a slight less than 100 submissions and we selected 2 as by the rules. \nBut I see that 5 submissions from early-stage were rescored (which form our dummy posituioning outside some listings of 1877(~3500) teams). But I don't understand why it should be done. That means that they rescore all submissions that were made, this can be 40 000, and of course not good for ecology to waste electricity, time of people, GPU/CPU time of computers.
\n\nSomeone in previous competitions already gave an estimation of energy wasted/consumed due to non-well organized competitions. And it was enormous.
", "votes": 1, "replies": [] }, { "id": 703948, "author_name": "olivier", "author_url": "", "post_date": "2019-12-26T21:12:39.450000", "content": "Kaggle is rescoring all submissions it seems. I can see lots of my rescores returned an error and only one (at least for now) returned a number. \nNot sure what's going on, I thought applying a filter on a row number column was easy enough and some packages can read data and score very fast...
", "votes": 1, "replies": [] } ] }, { "id": 704393, "author_name": "Hegelianer", "author_url": "", "post_date": "2019-12-27T12:28:55.343000", "content": "In the soviet us competition scores you
", "votes": 4, "replies": [ { "id": 704438, "author_name": "Ahmet Erdem", "author_url": "", "post_date": "2019-12-27T13:39:55.557000", "content": "Google Translate should be banned ๐
", "votes": 3, "replies": [] }, { "id": 704533, "author_name": "Hegelianer", "author_url": "", "post_date": "2019-12-27T15:35:39.860000", "content": "Dude, we already got our medals
", "votes": 1, "replies": [] } ] }, { "id": 704282, "author_name": "Ahmet Erdem", "author_url": "", "post_date": "2019-12-27T09:10:52.397000", "content": "Yesterday the LB changed a bit. It is weird, it was supposed to stop until they finish the run on the backup system. I wonder if everything is messed up and the person who can fix it is on holiday.
", "votes": 4, "replies": [ { "id": 704441, "author_name": "YaGana Sheriff-Hussaini", "author_url": "", "post_date": "2019-12-27T13:44:08.343000", "content": "\n\n\nI wonder if everything is messed up and the person who can fix it is on holiday.
\n
@divrikwicky if so, it would not be the 1st time :-)
", "votes": 1, "replies": [] }, { "id": 704449, "author_name": "sandy1112", "author_url": "", "post_date": "2019-12-27T13:55:21.003000", "content": "we have seen atleast one competition where the wait time was longer than this. Last Santander competition, we had to wait for 2 weeks before final pvt. LB was released. I won't be surprised if this one takes the same amount of time.
", "votes": 1, "replies": [] }, { "id": 704460, "author_name": "Stanislav Blinov", "author_url": "", "post_date": "2019-12-27T14:09:48.933000", "content": "At least at Santander we could see the preliminary standings, but now it's a madness to just sit and wait
", "votes": 1, "replies": [] }, { "id": 704465, "author_name": "sandy1112", "author_url": "", "post_date": "2019-12-27T14:14:40.717000", "content": "@stanislavblinov yes, that is why I had asked the organizers in their first LB Update post that if there would be a preliminary LB like Santander. At that time they had confirmed that there would be no preliminary LB...but I guess the issue with leaked data and the proposed way to deal with it has caused lot of issues. In Satander people were only looking for 'Magic' which wasn't a leak ๐ ...this one is way worse
", "votes": 2, "replies": [] } ] }, { "id": 704245, "author_name": "Hirotaka Kawashima", "author_url": "", "post_date": "2019-12-27T08:10:42.857000", "content": "Is your calendar a solar calendar? Is it the lunar calendar? Or the lunisolar calendar? ๐
", "votes": 4, "replies": [] }, { "id": 702658, "author_name": "xckkcxxck", "author_url": "", "post_date": "2019-12-25T01:16:36.893000", "content": "I think that suspected users can be asked to submit code for inspection, especially in the gold zone
", "votes": 2, "replies": [] }, { "id": 704256, "author_name": "Tawara", "author_url": "", "post_date": "2019-12-27T08:20:09.467000", "content": "I just heard that many people in U.S. take a holiday from 12/25(Christmas Day) to 1/1(New Year's Day). They too? \nOMG๐
", "votes": 1, "replies": [] }, { "id": 704240, "author_name": "c-weed", "author_url": "", "post_date": "2019-12-27T08:07:37.367000", "content": "> We will post another update or the scores on the PST afternoon of the 26th.
\n\n๐
", "votes": 1, "replies": [ { "id": 704255, "author_name": "olivier", "author_url": "", "post_date": "2019-12-27T08:19:37.677000", "content": "Did anyone commit on a year ?
", "votes": 3, "replies": [] }, { "id": 704277, "author_name": "Jie Lu", "author_url": "", "post_date": "2019-12-27T09:05:38.043000", "content": "Maybe January 26th ? ๐
", "votes": 2, "replies": [] } ] }, { "id": 704181, "author_name": "ADITYA KUMAR (Negi)", "author_url": "", "post_date": "2019-12-27T06:23:59.427000", "content": "what about results ? can we expect this week ? or next year ? let us know
", "votes": 1, "replies": [ { "id": 704344, "author_name": "Vlad Kostoglodoff", "author_url": "", "post_date": "2019-12-27T11:00:58.490000", "content": "Next decade ๐
", "votes": 3, "replies": [] }, { "id": 704651, "author_name": "AlexeyKurochkin", "author_url": "", "post_date": "2019-12-27T19:28:19.323000", "content": "hny btw lol
", "votes": 2, "replies": [] } ] }, { "id": 701917, "author_name": "ุนุซู ุงู", "author_url": "", "post_date": "2019-12-24T03:41:43.603000", "content": "Ready to help Kaggle kick off a milk n cookies operational optimization at anytime :-)
", "votes": 1, "replies": [] }, { "id": 704487, "author_name": "Ouassim Adnane", "author_url": "", "post_date": "2019-12-27T14:35:19.853000", "content": "
Actually, just now it crossed the 2019/12/26th ,it has become the 2019/12/27th,but the update or leapboard is not come,maybe something happened!\nCould you please come to tell us what happened@Addison Howard
", "votes": 2, "replies": [ { "id": 704258, "author_name": "Mr Loke", "author_url": "", "post_date": "2019-12-27T08:21:30.070000", "content": "Did any of your submission has any score in it?
\n\nMine is nothing which means it has stuck somewhere.
\n\nNow I got no more pc and phone to use.
", "votes": 1, "replies": [] }, { "id": 704285, "author_name": "Jie Lu", "author_url": "", "post_date": "2019-12-27T09:15:05.677000", "content": "@klloke The organizers said that they are running a new scoring program in the background to make the process faster. and scores are not being synchronized onto the LB, what you see is just the results calculated by a former program, which must have been stopped by them now
", "votes": 1, "replies": [] } ] }, { "id": 704706, "author_name": "s.s.o", "author_url": "", "post_date": "2019-12-27T21:49:39.117000", "content": "\n๐
Thanks a lot!!
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "702505": ".jpeg?generation=1577214609571546&alt=media)\n", "702031": "I think he skipped because you weren't good kids. Santa hates leaky and unorganized competitions:) Thanks for the update anyway.", "704088": "> We will post another update or the scores on the PST afternoon of the 26th.\n\nShouldn't there have been an update by now? It's already 7:30pm (PST)? @addisonhoward ", "704163": "ASHRAE competitors be like:\n", "703812": "This competition has been a frustration for me for multiple reasons. \nI think leak cases could have been handled better, sooner. \nCheaters, probers could have been addressed sooner.\nAnd when we think at last the competition is over, waiting days, or probably weeks for results added insult to injury.\nMaybe simply prioritizing selected submits of teams if they have, or simply going late from early could have been a relief. Most of -serious- teams handpick their submits. We have only 2 rescores both of irrelevant and early tries.\nOn the other hand, I realize this is a demanding community and we will find a reason to whine about anyways,\nso good luck and happy new year to you all :)\n.", "703774": "", "702170": "I hope you are trying to catch those who identified a public data source and did not disclose it properly if it wasn't already disclosed. This is easy to find: look for perfect prediction...\n\nI feel the agony of all the participants in the meantime...", "702027": "", "701834": "Hey All,\n\nCross-posting [Sohier's post](https://www.kaggle.com/c/ashrae-energy-prediction/discussion/122554#701813) here for visibility:\n\nQuick update - we have a working draft containing ~40,000 rescores but need to check some edge cases to make sure we didn't miss valid submissions and to run additional tests to ensure that updating the scores kicks off all of the correct jobs downstream, like reinstating the ~1000+ teams that vanished from the LB. This will take additional time, but will not be done until after Christmas. We will post another update or the scores on the PST afternoon of the 26th. Thank you all for your patience!\n\nTL;DR - We didn't have enough milk and cookies to feed Santa and his reindeer so they skipped over our offices, so we're celebrating Boxing Day instead this year. ", "704235": "Under-promise and over-deliver", "702584": "", "704572": ".gif?generation=1577463814787012&alt=media)\n", "704514": "\n", "704500": "\n", "704331": "", "704161": "@addisonhoward, any update ? \n26th pst deadline has crossed now", "702500": "@addisonhoward \nCould you explain, what is the reason for 40 000 rescores if, for example, our team selected 2 submissions?\nThus, no one should care about other submissions, especially wasting time, resources and damaging ecology of the planet. \n\nI expect that there are 5%, 10% or a bit more which didn't take care and chose 2 submissions, but as I know, the only 2 best scores in public should be selected for them. \n\nSo, simple calculations give me 3500 teams x 2 submissions = 7000 rescoring. \n\nIs it so important to have some additional checks out of that? Or some teams have broken the rules and you try to identify that? \n\nThanks for answering. ", "704393": "In the soviet us competition scores you", "704282": "Yesterday the LB changed a bit. It is weird, it was supposed to stop until they finish the run on the backup system. I wonder if everything is messed up and the person who can fix it is on holiday.", "704245": "Is your calendar a solar calendar? Is it the lunar calendar? Or the lunisolar calendar? ๐ ", "702658": "I think that suspected users can be asked to submit code for inspection, especially in the gold zone", "704256": "I just heard that many people in U.S. take a holiday from 12/25(Christmas Day) to 1/1(New Year's Day). They too? \nOMG๐", "704240": "> We will post another update or the scores on the PST afternoon of the 26th.\n\n๐ ", "704181": "what about results ? can we expect this week ? or next year ? let us know ", "701917": "Ready to help Kaggle kick off a milk n cookies operational optimization at anytime :-)", "704487": "\n", "704243": "Actually, just now it crossed the 2019/12/26th ,it has become the 2019/12/27th,but the update or leapboard is not come,maybe something happened!\nCould you please come to tell us what happened[@Addison Howard](https://www.kaggle.com/addisonhoward)", "704706": "\n๐ ", "702497": "", "704860": "Thanks a lot!!" } }