I've just published the first set of score updates. It may take several minutes for all entries to show up on the leaderboard.
\nThank you all for your patience with this update! The next one is scheduled for March 2nd.
", "messages": [ { "id": 1694745, "postDate": "2022-02-17T17:22:53.740Z", "content": "I've just published the first set of score updates. It may take several minutes for all entries to show up on the leaderboard.
\nThank you all for your patience with this update! The next one is scheduled for March 2nd.
", "rawMarkdown": "I've just published the first set of score updates. It may take several minutes for all entries to show up on the leaderboard. \n\nThank you all for your patience with this update! The next one is scheduled for March 2nd.", "votes": 24 }, { "id": 1694973, "postDate": "2022-02-17T21:42:24.137Z", "content": "Please @sohier double check that both the suplementary train and the fresh data are ok an correctly joined (edit: sorting also) , this update looks very weird, so many errors, huge negative correlation scores… doesn't look like usual shakeout.
", "rawMarkdown": "Please @sohier double check that both the suplementary train and the fresh data are ok an correctly joined (edit: **sorting** also) , this update looks very weird, so many errors, huge negative correlation scores... doesn't look like usual shakeout.", "votes": 7, "replies": [ { "id": 1694995, "postDate": "2022-02-17T21:57:25.877Z", "content": "Or, what would be even better is if the supplemental training set could be updated again with the new data so that people could test their models. I feel like a lot of people put a lot of effort into their models and they at least deserve a little bit of a consolation to understand where their models went wrong.
", "rawMarkdown": "Or, what would be even better is if the supplemental training set could be updated again with the new data so that people could test their models. I feel like a lot of people put a lot of effort into their models and they at least deserve a little bit of a consolation to understand where their models went wrong.", "votes": 4 }, { "id": 1695712, "postDate": "2022-02-18T10:28:10.533Z", "content": "\n\nI feel like a lot of people put a lot of effort into their models
\n
I think that's the worst of all, but I really don't think there's anything wrong with their models, the problem isn't there, I hope they can fix it. If your submissions were successful in the first instance, they should work fine now. Even worse if they did it right in the first update a few days ago. Too many problems in this competition.
", "rawMarkdown": "> I feel like a lot of people put a lot of effort into their models \n\nI think that's the worst of all, but I really don't think there's anything wrong with their models, the problem isn't there, I hope they can fix it. If your submissions were successful in the first instance, they should work fine now. Even worse if they did it right in the first update a few days ago. Too many problems in this competition.", "votes": 2 } ] }, { "id": 1694932, "postDate": "2022-02-17T20:45:24.613Z", "content": "Well this is fun, my random number generator submission is (so far) in the top 37% 😂
", "rawMarkdown": "Well this is fun, my [random number generator submission](https://www.kaggle.com/marcogorelli/random-number-generator) is (so far) in the top 37% 😂", "votes": 8, "replies": [ { "id": 1694934, "postDate": "2022-02-17T20:47:21.367Z", "content": "My 5-fold models get 0.0005.
", "rawMarkdown": "My 5-fold models get 0.0005.", "votes": 2 }, { "id": 1695117, "postDate": "2022-02-18T00:56:02.467Z", "content": "I think a lot of big shake ups will happen in the coming updates. I only submitted a 2 fold model as my 3 fold one already ran 8 hours and 50 minutes 😅.
", "rawMarkdown": "I think a lot of big shake ups will happen in the coming updates. I only submitted a 2 fold model as my 3 fold one already ran 8 hours and 50 minutes 😅.", "votes": 1 }, { "id": 1695175, "postDate": "2022-02-18T02:01:47.687Z", "content": "For me, what matters most is that my model performs worse than a random guessing (lb 0.001). I don't get it.
", "rawMarkdown": "For me, what matters most is that my model performs worse than a random guessing (lb 0.001). I don't get it.", "votes": 1 } ] }, { "id": 1695266, "postDate": "2022-02-18T03:48:48.147Z", "content": "I have a proof that you guys @sohier may still be getting the wrong supplemental data. I have two submissions, one uses a reconstructed target to conduct periodic retraining and one doesn't. This reconstructed target previously fits well with train data, with a max absolute error of 1e-13. But since my model shouldn't start the retraining process for only two weeks, these two notebooks should obtain identical results, which is not the case.
", "rawMarkdown": "I have a proof that you guys @sohier may still be getting the wrong supplemental data. I have two submissions, one uses a reconstructed target to conduct periodic retraining and one doesn't. This reconstructed target previously fits well with train data, with a max absolute error of 1e-13. But since my model shouldn't start the retraining process for only two weeks, these two notebooks should obtain identical results, which is not the case.", "votes": 5, "replies": [ { "id": 1695645, "postDate": "2022-02-18T09:30:24.970Z", "content": "That's interesting. There is almost certainly something broken in the evaluation process and/or data. My both subs used reconstructed target, both score absurdly negative correlations. I hope they fix whatever it is the problem , some response by Kaggle would be also nice.
", "rawMarkdown": "That's interesting. There is almost certainly something broken in the evaluation process and/or data. My both subs used reconstructed target, both score absurdly negative correlations. I hope they fix whatever it is the problem , some response by Kaggle would be also nice." } ] }, { "id": 1694794, "postDate": "2022-02-17T17:57:34.263Z", "content": "Every single one of my submissions is now showing a submission error… Literally not a single submission is showing successful.. Even one that was a naive calculation just to test the submission.
\nThey were fine till now.. I even got the score update yesterday.. I think something has gone wrong again..
\n@sohier ? @cstein06 ? @kierangarvey ?
", "rawMarkdown": "Every single one of my submissions is now showing a submission error… Literally not a single submission is showing successful.. Even one that was a naive calculation just to test the submission.\n\nThey were fine till now.. I even got the score update yesterday.. I think something has gone wrong again..\n\n@sohier ? @cstein06 ? @kierangarvey ?", "votes": 6 }, { "id": 1694850, "postDate": "2022-02-17T18:52:23.297Z", "content": "I have a two negative scores. Any chance you flipped the sign of the scoring metric and I'm actually at the top of the leaderboard? JK
", "rawMarkdown": "I have a two negative scores. Any chance you flipped the sign of the scoring metric and I'm actually at the top of the leaderboard? JK", "votes": 4, "replies": [ { "id": 1694859, "postDate": "2022-02-17T19:05:55.733Z", "content": "I have two negative scores too. I had 5.7, 6.4 respectively on public lb. Now I have -0.0150 and -0.0116 respectively. @sohier check please that you give input data in right time order.
", "rawMarkdown": "I have two negative scores too. I had 5.7, 6.4 respectively on public lb. Now I have -0.0150 and -0.0116 respectively. @sohier check please that you give input data in right time order.", "votes": 1 }, { "id": 1694867, "postDate": "2022-02-17T19:23:17.060Z", "content": "I have that scores (-0.0150 and -0.0116 ) already 2 day. @sohier you said that were a lot of errors for notebooks and you rerun its. May be I have not errors due to a lot of try/except blocks. May be my notebooks must be rerun too?
", "rawMarkdown": "I have that scores (-0.0150 and -0.0116 ) already 2 day. @sohier you said that were a lot of errors for notebooks and you rerun its. May be I have not errors due to a lot of try/except blocks. May be my notebooks must be rerun too?" }, { "id": 1695009, "postDate": "2022-02-17T22:32:08.397Z", "content": "It might well be a sorting issue in the rerun data. Only explanation I can think of that affects so many of the subs (but not all).
", "rawMarkdown": "It might well be a sorting issue in the rerun data. Only explanation I can think of that affects so many of the subs (but not all). ", "votes": 1 }, { "id": 1695238, "postDate": "2022-02-18T03:12:54.577Z", "content": "our team is also a big negative correlation -0.0350, feel some diffference between public score and private score in the calculation.
", "rawMarkdown": "our team is also a big negative correlation -0.0350, feel some diffference between public score and private score in the calculation." }, { "id": 1695244, "postDate": "2022-02-18T03:15:34.070Z", "content": "@sohier can you help check the calculation.🙏🙏🙏
", "rawMarkdown": "@sohier can you help check the calculation.🙏🙏🙏" } ] }, { "id": 1694827, "postDate": "2022-02-17T18:27:58.287Z", "content": "@sohier both of my notebooks failled ;) any hope ?
", "rawMarkdown": "@sohier both of my notebooks failled ;) any hope ?", "votes": 4 }, { "id": 1695255, "postDate": "2022-02-18T03:26:09.780Z", "content": "It looks like many people are getting errors again… May a double check on supplemental data is necessary. Although my notebook finally ran normally😂But they don't seem to be getting a reasonable result. But we'll see in the next week.
", "rawMarkdown": "It looks like many people are getting errors again... May a double check on supplemental data is necessary. Although my notebook finally ran normally😂But they don't seem to be getting a reasonable result. But we'll see in the next week.", "votes": 1 }, { "id": 1694758, "postDate": "2022-02-17T17:29:44.370Z", "content": "Thanks @sohier for the update.
\nI can't understand why both of my notebook failled.
\nKnowing that I don't even have a model for one of them.
\nAnd they both run very fast.
\nI don't believe it's run time error.
Here's one of my notebook inference notebook
", "rawMarkdown": "Here's one of my notebook [inference notebook](https://www.kaggle.com/amedprof/linear-model)", "votes": 1 }, { "id": 1696284, "postDate": "2022-02-18T18:02:18.213Z", "content": "Are we getting any acknowledgment of these issues @sohier ?
", "rawMarkdown": "Are we getting any acknowledgment of these issues @sohier ? ", "votes": 1 } ] }, { "id": 1694877, "postDate": "2022-02-17T19:35:32.937Z", "content": "Could we get the logs or something for the kernel errors? All my submissions have failed while they were working before. I would like to understand what assumption I made that worked on the public data while not on the private one.
", "rawMarkdown": "Could we get the logs or something for the kernel errors? All my submissions have failed while they were working before. I would like to understand what assumption I made that worked on the public data while not on the private one.", "votes": 2 }, { "id": 1694805, "postDate": "2022-02-17T18:08:55.423Z", "content": "Only one submission is shown for our team. @sohier
", "rawMarkdown": "Only one submission is shown for our team. @sohier ", "votes": 2 }, { "id": 1695503, "postDate": "2022-02-18T07:25:18.493Z", "content": "I think there is something wrong - it seems obviously wrong.
\nSimulate random numbers and calculate correlation between them. You'll get lots of very small values, but every now and again you'd get a relatively large number, say a 0.3 or a -0.4. That is what you would expect.
\nYet here we see an extremely tight outcome (ignoring -1 values) for 1978 values that are between -0.035 to +0.0414.
\nThis does not seem right. Why so extremely tight?
", "rawMarkdown": "I think there is something wrong - it seems obviously wrong.\n\nSimulate random numbers and calculate correlation between them. You'll get lots of very small values, but every now and again you'd get a relatively large number, say a 0.3 or a -0.4. That is what you would expect.\n\nYet here we see an extremely tight outcome (ignoring -1 values) for 1978 values that are between -0.035 to +0.0414.\n\nThis does not seem right. Why so extremely tight?", "replies": [ { "id": 1695653, "postDate": "2022-02-18T09:33:02.580Z", "content": "presumably because lot's of people are forking public notebooks, so there's less variability than you might expect
", "rawMarkdown": "presumably because lot's of people are forking public notebooks, so there's less variability than you might expect" }, { "id": 1695673, "postDate": "2022-02-18T09:59:08.487Z", "content": "There will be less of course, but not nothing - a range of -0.04 to +0.04 is so little variation is seems all used the same model and seed
", "rawMarkdown": "There will be less of course, but not nothing - a range of -0.04 to +0.04 is so little variation is seems all used the same model and seed" }, { "id": 1695736, "postDate": "2022-02-18T10:38:05.663Z", "content": "Then again, maybe I've solved my own problem - given that there are a large number of observations, maybe such a small range is not that strange and to be expected.
\nAnyhow, I support the calls for an update to the supplementary data, then at least we can check our models ourselves as well.
Wow, only 0.0093 for now needed for bronze 😮
", "rawMarkdown": "Wow, only 0.0093 for now needed for bronze 😮", "votes": -1, "replies": [ { "id": 1694833, "postDate": "2022-02-17T18:30:48.590Z", "content": "It will surely change big during the competition :)
", "rawMarkdown": "It will surely change big during the competition :)" }, { "id": 1694948, "postDate": "2022-02-17T21:07:19.127Z", "content": "up or down?
", "rawMarkdown": "up or down?" } ] }, { "id": 1709647, "postDate": "2022-03-02T11:40:54.810Z", "content": "So, when will be the second update? Today?
", "rawMarkdown": "So, when will be the second update? Today?" }, { "id": 1695511, "postDate": "2022-02-18T07:32:22.140Z", "content": "Thanks for update.
\nIt's interesting to see updated score.
Hi, @sohier I found wired for my second submission result, which scored -1 temporarily. Comparing to my another submission (which is run by GPU) scored temporarily 0.0169, I added few more fine features calculated by my teammate for the input of DL model, and I aspected a better score for this update. While I submitted it, I miss clicked the saved version of notebook run by CPU on the date of submission deadline. I would like to ask will run by CPU differ a lot for the final score? Seeing my two submissions used the same environment, the time consumption for each iteration is less than the restriction, and it's submitted successfully.
", "rawMarkdown": "Hi, @sohier I found wired for my second submission result, which scored -1 temporarily. Comparing to my another submission (which is run by GPU) scored temporarily 0.0169, I added few more fine features calculated by my teammate for the input of DL model, and I aspected a better score for this update. While I submitted it, I miss clicked the saved version of notebook run by CPU on the date of submission deadline. I would like to ask will run by CPU differ a lot for the final score? Seeing my two submissions used the same environment, the time consumption for each iteration is less than the restriction, and it's submitted successfully." }, { "id": 1695007, "postDate": "2022-02-17T22:28:39.877Z", "content": "Aditionally, @sohier, I would have a look at the sorting of the rerun datasets vs original. It's the first explanation I can think of compatible with so many negative scores, but not all of them (different approaches in different notebooks). Otherwise, can not find explanation to near -0.04 negative correlation score that are happening here.
", "rawMarkdown": "Aditionally, @sohier, **I would have a look at the sorting of the rerun datasets vs original**. It's the first explanation I can think of compatible with so many negative scores, but not all of them (different approaches in different notebooks). Otherwise, can not find explanation to near -0.04 negative correlation score that are happening here." }, { "id": 1694797, "postDate": "2022-02-17T18:01:45.977Z", "content": "Cool, the subs is working and in the bronze zone 👍
", "rawMarkdown": "Cool, the subs is working and in the bronze zone 👍" }, { "id": 1694764, "postDate": "2022-02-17T17:34:03.557Z", "content": "Hi Sir, I wanted to make an inquiry. I only made 3 submissions and 5 appears, of which 3 have errors, and two negative values. Previously I had one that had been executed correctly and from what I see it would have been in the silver medal zone. Any reason for this behavior?
", "rawMarkdown": "Hi Sir, I wanted to make an inquiry. I only made 3 submissions and 5 appears, of which 3 have errors, and two negative values. Previously I had one that had been executed correctly and from what I see it would have been in the silver medal zone. Any reason for this behavior?", "replies": [ { "id": 1694828, "postDate": "2022-02-17T18:28:17.680Z", "content": "The two extra submission are the re-runs from the selected ones.
", "rawMarkdown": "The two extra submission are the re-runs from the selected ones.", "votes": 1 }, { "id": 1695727, "postDate": "2022-02-18T10:32:16.303Z", "content": "Thanks for the clarification Kinderf.
", "rawMarkdown": "Thanks for the clarification Kinderf. " } ] }, { "id": 1709559, "postDate": "2022-03-02T09:52:51.780Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 1694973, "author_name": "miguel perez", "author_url": "", "post_date": "2022-02-17T21:42:24.137000", "content": "Please @sohier double check that both the suplementary train and the fresh data are ok an correctly joined (edit: sorting also) , this update looks very weird, so many errors, huge negative correlation scores… doesn't look like usual shakeout.
", "votes": 7, "replies": [ { "id": 1694995, "author_name": "Kyle Peters", "author_url": "", "post_date": "2022-02-17T21:57:25.877000", "content": "Or, what would be even better is if the supplemental training set could be updated again with the new data so that people could test their models. I feel like a lot of people put a lot of effort into their models and they at least deserve a little bit of a consolation to understand where their models went wrong.
", "votes": 4, "replies": [] }, { "id": 1695712, "author_name": "Maximiliano Diaz Battan", "author_url": "", "post_date": "2022-02-18T10:28:10.533000", "content": "\n\nI feel like a lot of people put a lot of effort into their models
\n
I think that's the worst of all, but I really don't think there's anything wrong with their models, the problem isn't there, I hope they can fix it. If your submissions were successful in the first instance, they should work fine now. Even worse if they did it right in the first update a few days ago. Too many problems in this competition.
", "votes": 2, "replies": [] } ] }, { "id": 1694932, "author_name": "Marco Gorelli", "author_url": "", "post_date": "2022-02-17T20:45:24.613000", "content": "Well this is fun, my random number generator submission is (so far) in the top 37% 😂
", "votes": 8, "replies": [ { "id": 1694934, "author_name": "Yue Sun", "author_url": "", "post_date": "2022-02-17T20:47:21.367000", "content": "My 5-fold models get 0.0005.
", "votes": 2, "replies": [] }, { "id": 1695117, "author_name": "Fritz Cremer", "author_url": "", "post_date": "2022-02-18T00:56:02.467000", "content": "I think a lot of big shake ups will happen in the coming updates. I only submitted a 2 fold model as my 3 fold one already ran 8 hours and 50 minutes 😅.
", "votes": 1, "replies": [] }, { "id": 1695175, "author_name": "Yue Sun", "author_url": "", "post_date": "2022-02-18T02:01:47.687000", "content": "For me, what matters most is that my model performs worse than a random guessing (lb 0.001). I don't get it.
", "votes": 1, "replies": [] } ] }, { "id": 1695266, "author_name": "ACMHIT", "author_url": "", "post_date": "2022-02-18T03:48:48.147000", "content": "I have a proof that you guys @sohier may still be getting the wrong supplemental data. I have two submissions, one uses a reconstructed target to conduct periodic retraining and one doesn't. This reconstructed target previously fits well with train data, with a max absolute error of 1e-13. But since my model shouldn't start the retraining process for only two weeks, these two notebooks should obtain identical results, which is not the case.
", "votes": 5, "replies": [ { "id": 1695645, "author_name": "miguel perez", "author_url": "", "post_date": "2022-02-18T09:30:24.970000", "content": "That's interesting. There is almost certainly something broken in the evaluation process and/or data. My both subs used reconstructed target, both score absurdly negative correlations. I hope they fix whatever it is the problem , some response by Kaggle would be also nice.
", "votes": 0, "replies": [] } ] }, { "id": 1694794, "author_name": "JM", "author_url": "", "post_date": "2022-02-17T17:57:34.263000", "content": "Every single one of my submissions is now showing a submission error… Literally not a single submission is showing successful.. Even one that was a naive calculation just to test the submission.
\nThey were fine till now.. I even got the score update yesterday.. I think something has gone wrong again..
\n@sohier ? @cstein06 ? @kierangarvey ?
", "votes": 6, "replies": [] }, { "id": 1694850, "author_name": "Chris Miles", "author_url": "", "post_date": "2022-02-17T18:52:23.297000", "content": "I have a two negative scores. Any chance you flipped the sign of the scoring metric and I'm actually at the top of the leaderboard? JK
", "votes": 4, "replies": [ { "id": 1694859, "author_name": "Peter Chuikov", "author_url": "", "post_date": "2022-02-17T19:05:55.733000", "content": "I have two negative scores too. I had 5.7, 6.4 respectively on public lb. Now I have -0.0150 and -0.0116 respectively. @sohier check please that you give input data in right time order.
", "votes": 1, "replies": [] }, { "id": 1694867, "author_name": "Peter Chuikov", "author_url": "", "post_date": "2022-02-17T19:23:17.060000", "content": "I have that scores (-0.0150 and -0.0116 ) already 2 day. @sohier you said that were a lot of errors for notebooks and you rerun its. May be I have not errors due to a lot of try/except blocks. May be my notebooks must be rerun too?
", "votes": 0, "replies": [] }, { "id": 1695009, "author_name": "miguel perez", "author_url": "", "post_date": "2022-02-17T22:32:08.397000", "content": "It might well be a sorting issue in the rerun data. Only explanation I can think of that affects so many of the subs (but not all).
", "votes": 1, "replies": [] }, { "id": 1695238, "author_name": "Hayden234", "author_url": "", "post_date": "2022-02-18T03:12:54.577000", "content": "our team is also a big negative correlation -0.0350, feel some diffference between public score and private score in the calculation.
", "votes": 0, "replies": [] }, { "id": 1695244, "author_name": "Hayden234", "author_url": "", "post_date": "2022-02-18T03:15:34.070000", "content": "@sohier can you help check the calculation.🙏🙏🙏
", "votes": 0, "replies": [] } ] }, { "id": 1694827, "author_name": "Ghassen Alaya", "author_url": "", "post_date": "2022-02-17T18:27:58.287000", "content": "@sohier both of my notebooks failled ;) any hope ?
", "votes": 4, "replies": [] }, { "id": 1695255, "author_name": "ACMHIT", "author_url": "", "post_date": "2022-02-18T03:26:09.780000", "content": "It looks like many people are getting errors again… May a double check on supplemental data is necessary. Although my notebook finally ran normally😂But they don't seem to be getting a reasonable result. But we'll see in the next week.
", "votes": 1, "replies": [] }, { "id": 1694758, "author_name": "Amed", "author_url": "", "post_date": "2022-02-17T17:29:44.370000", "content": "Thanks @sohier for the update.
\nI can't understand why both of my notebook failled.
\nKnowing that I don't even have a model for one of them.
\nAnd they both run very fast.
\nI don't believe it's run time error.
Here's one of my notebook inference notebook
", "votes": 1, "replies": [] }, { "id": 1696284, "author_name": "JM", "author_url": "", "post_date": "2022-02-18T18:02:18.213000", "content": "Are we getting any acknowledgment of these issues @sohier ?
", "votes": 1, "replies": [] } ] }, { "id": 1694877, "author_name": "JQ", "author_url": "", "post_date": "2022-02-17T19:35:32.937000", "content": "Could we get the logs or something for the kernel errors? All my submissions have failed while they were working before. I would like to understand what assumption I made that worked on the public data while not on the private one.
", "votes": 2, "replies": [] }, { "id": 1694805, "author_name": "Yirun Zhang", "author_url": "", "post_date": "2022-02-17T18:08:55.423000", "content": "Only one submission is shown for our team. @sohier
", "votes": 2, "replies": [] }, { "id": 1695503, "author_name": "skaak", "author_url": "", "post_date": "2022-02-18T07:25:18.493000", "content": "I think there is something wrong - it seems obviously wrong.
\nSimulate random numbers and calculate correlation between them. You'll get lots of very small values, but every now and again you'd get a relatively large number, say a 0.3 or a -0.4. That is what you would expect.
\nYet here we see an extremely tight outcome (ignoring -1 values) for 1978 values that are between -0.035 to +0.0414.
\nThis does not seem right. Why so extremely tight?
", "votes": 0, "replies": [ { "id": 1695653, "author_name": "Marco Gorelli", "author_url": "", "post_date": "2022-02-18T09:33:02.580000", "content": "presumably because lot's of people are forking public notebooks, so there's less variability than you might expect
", "votes": 0, "replies": [] }, { "id": 1695673, "author_name": "skaak", "author_url": "", "post_date": "2022-02-18T09:59:08.487000", "content": "There will be less of course, but not nothing - a range of -0.04 to +0.04 is so little variation is seems all used the same model and seed
", "votes": 0, "replies": [] }, { "id": 1695736, "author_name": "skaak", "author_url": "", "post_date": "2022-02-18T10:38:05.663000", "content": "Then again, maybe I've solved my own problem - given that there are a large number of observations, maybe such a small range is not that strange and to be expected.
\nAnyhow, I support the calls for an update to the supplementary data, then at least we can check our models ourselves as well.
Wow, only 0.0093 for now needed for bronze 😮
", "votes": -1, "replies": [ { "id": 1694833, "author_name": "Kirderf", "author_url": "", "post_date": "2022-02-17T18:30:48.590000", "content": "It will surely change big during the competition :)
", "votes": 0, "replies": [] }, { "id": 1694948, "author_name": "Marco Gorelli", "author_url": "", "post_date": "2022-02-17T21:07:19.127000", "content": "up or down?
", "votes": 0, "replies": [] } ] }, { "id": 1709647, "author_name": "Hang Yin", "author_url": "", "post_date": "2022-03-02T11:40:54.810000", "content": "So, when will be the second update? Today?
", "votes": 0, "replies": [] }, { "id": 1695511, "author_name": "Hanphil Yoo", "author_url": "", "post_date": "2022-02-18T07:32:22.140000", "content": "Thanks for update.
\nIt's interesting to see updated score.
Hi, @sohier I found wired for my second submission result, which scored -1 temporarily. Comparing to my another submission (which is run by GPU) scored temporarily 0.0169, I added few more fine features calculated by my teammate for the input of DL model, and I aspected a better score for this update. While I submitted it, I miss clicked the saved version of notebook run by CPU on the date of submission deadline. I would like to ask will run by CPU differ a lot for the final score? Seeing my two submissions used the same environment, the time consumption for each iteration is less than the restriction, and it's submitted successfully.
", "votes": 0, "replies": [] }, { "id": 1695007, "author_name": "miguel perez", "author_url": "", "post_date": "2022-02-17T22:28:39.877000", "content": "Aditionally, @sohier, I would have a look at the sorting of the rerun datasets vs original. It's the first explanation I can think of compatible with so many negative scores, but not all of them (different approaches in different notebooks). Otherwise, can not find explanation to near -0.04 negative correlation score that are happening here.
", "votes": 0, "replies": [] }, { "id": 1694797, "author_name": "Kirderf", "author_url": "", "post_date": "2022-02-17T18:01:45.977000", "content": "Cool, the subs is working and in the bronze zone 👍
", "votes": 0, "replies": [] }, { "id": 1694764, "author_name": "Maximiliano Diaz Battan", "author_url": "", "post_date": "2022-02-17T17:34:03.557000", "content": "Hi Sir, I wanted to make an inquiry. I only made 3 submissions and 5 appears, of which 3 have errors, and two negative values. Previously I had one that had been executed correctly and from what I see it would have been in the silver medal zone. Any reason for this behavior?
", "votes": 0, "replies": [ { "id": 1694828, "author_name": "Kirderf", "author_url": "", "post_date": "2022-02-17T18:28:17.680000", "content": "The two extra submission are the re-runs from the selected ones.
", "votes": 1, "replies": [] }, { "id": 1695727, "author_name": "Maximiliano Diaz Battan", "author_url": "", "post_date": "2022-02-18T10:32:16.303000", "content": "Thanks for the clarification Kinderf.
", "votes": 0, "replies": [] } ] }, { "id": 1709559, "author_name": "", "author_url": "", "post_date": "2022-03-02T09:52:51.780000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1694745": "I've just published the first set of score updates. It may take several minutes for all entries to show up on the leaderboard. \n\nThank you all for your patience with this update! The next one is scheduled for March 2nd.", "1694973": "Please @sohier double check that both the suplementary train and the fresh data are ok an correctly joined (edit: **sorting** also) , this update looks very weird, so many errors, huge negative correlation scores... doesn't look like usual shakeout.", "1694932": "Well this is fun, my [random number generator submission](https://www.kaggle.com/marcogorelli/random-number-generator) is (so far) in the top 37% 😂", "1695266": "I have a proof that you guys @sohier may still be getting the wrong supplemental data. I have two submissions, one uses a reconstructed target to conduct periodic retraining and one doesn't. This reconstructed target previously fits well with train data, with a max absolute error of 1e-13. But since my model shouldn't start the retraining process for only two weeks, these two notebooks should obtain identical results, which is not the case.", "1694794": "Every single one of my submissions is now showing a submission error… Literally not a single submission is showing successful.. Even one that was a naive calculation just to test the submission.\n\nThey were fine till now.. I even got the score update yesterday.. I think something has gone wrong again..\n\n@sohier ? @cstein06 ? @kierangarvey ?", "1694850": "I have a two negative scores. Any chance you flipped the sign of the scoring metric and I'm actually at the top of the leaderboard? JK", "1694827": "@sohier both of my notebooks failled ;) any hope ?", "1695255": "It looks like many people are getting errors again... May a double check on supplemental data is necessary. Although my notebook finally ran normally😂But they don't seem to be getting a reasonable result. But we'll see in the next week.", "1694758": "Thanks @sohier for the update.\nI can't understand why both of my notebook failled.\nKnowing that I don't even have a model for one of them.\nAnd they both run very fast.\nI don't believe it's run time error.", "1694877": "Could we get the logs or something for the kernel errors? All my submissions have failed while they were working before. I would like to understand what assumption I made that worked on the public data while not on the private one.", "1694805": "Only one submission is shown for our team. @sohier ", "1695503": "I think there is something wrong - it seems obviously wrong.\n\nSimulate random numbers and calculate correlation between them. You'll get lots of very small values, but every now and again you'd get a relatively large number, say a 0.3 or a -0.4. That is what you would expect.\n\nYet here we see an extremely tight outcome (ignoring -1 values) for 1978 values that are between -0.035 to +0.0414.\n\nThis does not seem right. Why so extremely tight?", "1694815": "Wow, only 0.0093 for now needed for bronze 😮", "1709647": "So, when will be the second update? Today?", "1695511": "Thanks for update.\nIt's interesting to see updated score.", "1695040": "Hi, @sohier I found wired for my second submission result, which scored -1 temporarily. Comparing to my another submission (which is run by GPU) scored temporarily 0.0169, I added few more fine features calculated by my teammate for the input of DL model, and I aspected a better score for this update. While I submitted it, I miss clicked the saved version of notebook run by CPU on the date of submission deadline. I would like to ask will run by CPU differ a lot for the final score? Seeing my two submissions used the same environment, the time consumption for each iteration is less than the restriction, and it's submitted successfully.", "1695007": "Aditionally, @sohier, **I would have a look at the sorting of the rerun datasets vs original**. It's the first explanation I can think of compatible with so many negative scores, but not all of them (different approaches in different notebooks). Otherwise, can not find explanation to near -0.04 negative correlation score that are happening here.", "1694797": "Cool, the subs is working and in the bronze zone 👍", "1694764": "Hi Sir, I wanted to make an inquiry. I only made 3 submissions and 5 appears, of which 3 have errors, and two negative values. Previously I had one that had been executed correctly and from what I see it would have been in the silver medal zone. Any reason for this behavior?", "1709559": "" } }