We''ll be updating the scores roughly once every two weeks. We may need longer to get the very first update out the door, but my goal is to publish the first update on the 16th.
", "messages": [ { "id": 1673764, "postDate": "2022-02-03T00:37:08.133Z", "content": "We''ll be updating the scores roughly once every two weeks. We may need longer to get the very first update out the door, but my goal is to publish the first update on the 16th.
", "rawMarkdown": "We''ll be updating the scores roughly once every two weeks. We may need longer to get the very first update out the door, but my goal is to publish the first update on the 16th.", "votes": 56 }, { "id": 1692741, "postDate": "2022-02-16T07:59:52.463Z", "content": "I expect the shakeup for this competition to be like a rollercoaster.
", "rawMarkdown": "I expect the shakeup for this competition to be like a rollercoaster.", "votes": 5, "replies": [ { "id": 1694188, "postDate": "2022-02-17T08:17:28.097Z", "content": "It seems that for some people, this competition will become a free lottery with prizes of $125,000.
", "rawMarkdown": "It seems that for some people, this competition will become a free lottery with prizes of $125,000.", "votes": 4 }, { "id": 1694255, "postDate": "2022-02-17T09:49:05.957Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1695345, "postDate": "2022-02-18T05:00:55.517Z", "content": "Yes, I was on top of the rollercoaster till yesterday and now at bottom, too numb to react 😦☹️😨🤒
", "rawMarkdown": "Yes, I was on top of the rollercoaster till yesterday and now at bottom, too numb to react 😦☹️😨🤒" }, { "id": 1695564, "postDate": "2022-02-18T08:15:55.703Z", "content": "Not sure about that, I don't think I've ever read a winning solution write up in which it looked like the person just got lucky, and I don't think this competition will be different in this regard
", "rawMarkdown": "Not sure about that, I don't think I've ever read a winning solution write up in which it looked like the person just got lucky, and I don't think this competition will be different in this regard", "votes": 3 }, { "id": 1696687, "postDate": "2022-02-19T03:10:20.733Z", "content": "Just epic sauce 😄 hold tight, this will be a bumpy ride !
", "rawMarkdown": "Just epic sauce 😄 hold tight, this will be a bumpy ride !" } ] }, { "id": 1695534, "postDate": "2022-02-18T07:48:46.613Z", "content": "Unfortunately, for me this shakeup was a totally crash: both my selected submissions reruns returned errors.
", "rawMarkdown": "Unfortunately, for me this shakeup was a totally crash: both my selected submissions reruns returned errors.", "votes": 6, "replies": [ { "id": 1696291, "postDate": "2022-02-18T18:08:04.373Z", "content": "Maybe you used some external data or something like that?
\nOtherwise, it must be pretty frustrating, especially because they don't explain what exactly went wrong. My understanding was that all successful submissions made in the competition stage had to be successful in the evaluation stage.
\nThe message \"This version of the API is not optimized…\" kind of suggests that the submission process will be much better / faster in the evaluation stage. I might be wrong tough.
@denisserbin I got the same error in both of my notebooks and I didn't even use the supp data to train or evaluate my models.I only used the started training data. So I don't think supp data is the issue.
", "rawMarkdown": "@denisserbin I got the same error in both of my notebooks and I didn't even use the supp data to train or evaluate my models.I only used the started training data. So I don't think supp data is the issue. " }, { "id": 1697012, "postDate": "2022-02-19T09:56:41.477Z", "content": "No, no external data. One of them runs in less than 1h30 on GPU, without internet access and without external additional data.
", "rawMarkdown": "No, no external data. One of them runs in less than 1h30 on GPU, without internet access and without external additional data.", "votes": 1 } ] }, { "id": 1695129, "postDate": "2022-02-18T01:13:57.910Z", "content": "Well, at least the updated scores look like what you should expect in real life.
", "rawMarkdown": "Well, at least the updated scores look like what you should expect in real life.", "votes": 4 }, { "id": 1722280, "postDate": "2022-03-14T11:48:28.717Z", "content": "When is third update coming?
", "rawMarkdown": "When is third update coming?", "votes": 4 }, { "id": 1674399, "postDate": "2022-02-03T13:06:19.423Z", "content": "Brace yourselves
\nthe shakeup is coming!
", "rawMarkdown": "Brace yourselves\n\nthe shakeup is coming!", "votes": 3, "replies": [ { "id": 1675407, "postDate": "2022-02-04T08:51:42.307Z", "content": "Crypto market is waiting a shakeup as well :-).
", "rawMarkdown": "Crypto market is waiting a shakeup as well :-).", "votes": 4 } ] }, { "id": 1693117, "postDate": "2022-02-16T12:52:10.720Z", "content": "Is the update reflecting on leaderboard because i can't see score in the leaderboard being updated?
\nWhat just happened here idk. Today only I saw kaggle competition section and my rank got up from 12 to 9. Is it good or bad?
It hasn't been reflected on the lb, but there was a change to the public lb. This will not be important though for the private lb evaluation phase.
", "rawMarkdown": "It hasn't been reflected on the lb, but there was a change to the public lb. This will not be important though for the private lb evaluation phase.", "votes": 1 }, { "id": 1694036, "postDate": "2022-02-17T05:51:46.620Z", "content": "Ok. Thanks for the information.
", "rawMarkdown": "Ok. Thanks for the information. " } ] }, { "id": 1693027, "postDate": "2022-02-16T11:57:51.480Z", "content": "Has anyone seen any score update after 1st round of reruns?
", "rawMarkdown": "Has anyone seen any score update after 1st round of reruns?", "votes": 1, "replies": [ { "id": 1693084, "postDate": "2022-02-16T12:25:55.753Z", "content": "I have updated score for 1 submission and 1 with -1 score, don't know why it happened
", "rawMarkdown": "I have updated score for 1 submission and 1 with -1 score, don't know why it happened\n", "votes": 1 }, { "id": 1693130, "postDate": "2022-02-16T13:08:36.693Z", "content": "I am confused, because I went up in the ranks but it is still that submission just for fun which is not included in my selected submissions list.
", "rawMarkdown": "I am confused, because I went up in the ranks but it is still that submission just for fun which is not included in my selected submissions list.", "votes": 1 } ] }, { "id": 1692114, "postDate": "2022-02-15T20:39:21.433Z", "content": "What does it mean if your submission succeed , but the score is -1.0? I have two exactly the same notebooks, one of them just have supplemental train. The one with supplemental shows no error, but -1.0 score.
", "rawMarkdown": "What does it mean if your submission succeed , but the score is -1.0? I have two exactly the same notebooks, one of them just have supplemental train. The one with supplemental shows no error, but -1.0 score.", "votes": 1, "replies": [ { "id": 1692127, "postDate": "2022-02-15T21:02:35.650Z", "content": "I think predicting value 0 for every entry will give score -1.
", "rawMarkdown": "I think predicting value 0 for every entry will give score -1.", "votes": 2, "isDeleted": true }, { "id": 1692585, "postDate": "2022-02-16T06:08:56.013Z", "content": "It should not predict all zeros, because we have exactly the same notebooks with different data. May be we need to wait until the run finishes..
", "rawMarkdown": "It should not predict all zeros, because we have exactly the same notebooks with different data. May be we need to wait until the run finishes.." }, { "id": 1692589, "postDate": "2022-02-16T06:10:37.037Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1692593, "postDate": "2022-02-16T06:13:01.453Z", "content": "Yes, I understand it, but it's strange since they are the same notebooks
", "rawMarkdown": "Yes, I understand it, but it's strange since they are the same notebooks\n" }, { "id": 1693184, "postDate": "2022-02-16T13:54:45.427Z", "content": "Being the perfect opposite of the true values would be practically impossible - if you just switch your predictions, then you have a perfect forecast 😄
\nI think it just means it's an invalid score - perhaps something went wrong when joining with the supplemental data?
", "rawMarkdown": "Being the perfect opposite of the true values would be practically impossible - if you just switch your predictions, then you have a perfect forecast :smile:\n\nI think it just means it's an invalid score - perhaps something went wrong when joining with the supplemental data?", "votes": 1 }, { "id": 1693191, "postDate": "2022-02-16T14:00:55.910Z", "content": "I have made test runs before and everything was well. I have not used memory for full limit and had some free memory for additional data, I think it was enough for last 2 weeks of submission period. Anyway the notebook have run without any error and is marked as Succeed.
", "rawMarkdown": "I have made test runs before and everything was well. I have not used memory for full limit and had some free memory for additional data, I think it was enough for last 2 weeks of submission period. Anyway the notebook have run without any error and is marked as Succeed." } ] }, { "id": 1675302, "postDate": "2022-02-04T07:15:16.980Z", "content": "On 16th will be the private score already visible?
", "rawMarkdown": "On 16th will be the private score already visible?", "votes": 1 }, { "id": 1691672, "postDate": "2022-02-15T14:42:45.887Z", "content": "Looked into the \"Notebook Threw Error\" found this explanation from the Kaggle staff. Looks like in previous competitions where this happened that the situation didn't fix itself and that no score was achieved with this error. :(
\nDustinKaggle Staff • 7 months ago • Options • Report • Reply
\n1
\n@pritommojumder Usually this means you're getting an error on the re-run of the submission that doesn't occur during the original run on the public dataset.
\nUsually this happens because of some assumptions that are being made about the private data which don't hold, but do for the public data.
", "rawMarkdown": "Looked into the \"Notebook Threw Error\" found this explanation from the Kaggle staff. Looks like in previous competitions where this happened that the situation didn't fix itself and that no score was achieved with this error. :( \n\nDustinKaggle Staff • 7 months ago • Options • Report • Reply\n\n1\n\n@pritommojumder Usually this means you're getting an error on the re-run of the submission that doesn't occur during the original run on the public dataset.\n\nUsually this happens because of some assumptions that are being made about the private data which don't hold, but do for the public data.", "votes": 2, "replies": [ { "id": 1691699, "postDate": "2022-02-15T15:00:18.020Z", "content": "Yes but in this case the only changes from the public run is that they added 1/3 of supplemental_train in the same format/file/location. My guess it will work in the real re-run as the private leaderboard isn't updated yet.
\nWith 2/3 of the 300Mb supplemental_train.csv everything went great, just adding the other 1/3 shouldn't be a problem for my models.
Agree, would not have thought it would have been an issue on myside either. Lets see what tomorrow brings.
", "rawMarkdown": "Agree, would not have thought it would have been an issue on myside either. Lets see what tomorrow brings.", "votes": 1 } ] }, { "id": 1690551, "postDate": "2022-02-15T02:05:13.037Z", "content": "@sohier Is the update already in progress? I have different scores for the models I have submitted.
", "rawMarkdown": "@sohier Is the update already in progress? I have different scores for the models I have submitted.", "votes": 2 }, { "id": 1709421, "postDate": "2022-03-02T07:19:47.437Z", "content": "When will the second update come.
", "rawMarkdown": "When will the second update come." }, { "id": 1707476, "postDate": "2022-02-28T13:53:11.900Z", "content": "So, when will be the second update? :)
", "rawMarkdown": "So, when will be the second update? :)", "replies": [ { "id": 1707507, "postDate": "2022-02-28T14:20:48.150Z", "content": "2nd March I think
", "rawMarkdown": "2nd March I think" } ] }, { "id": 1694251, "postDate": "2022-02-17T09:43:12.170Z", "content": "One thing that I do not understand is why 1.000 is still the best score (followed by many 0.999), if submissions are rerun? I suppose that most of us did not select submissions with 0.999 & 1.000 for private eval.
", "rawMarkdown": "One thing that I do not understand is why 1.000 is still the best score (followed by many 0.999), if submissions are rerun? I suppose that most of us did not select submissions with 0.999 & 1.000 for private eval." }, { "id": 1693929, "postDate": "2022-02-17T04:11:03.683Z", "content": "My two submissions are now pure \"ERROR\" but not \"Note book throw an Exception\" or \"Score submission error\"😂It seems the hosts have rerun them again.
", "rawMarkdown": "My two submissions are now pure \"ERROR\" but not \"Note book throw an Exception\" or \"Score submission error\"😂It seems the hosts have rerun them again.", "replies": [ { "id": 1696695, "postDate": "2022-02-19T03:18:47.890Z", "content": "I wonder if the submission with error will be replace with another one… on the LB I have one without ERROR that I have not selected as submission…
", "rawMarkdown": "I wonder if the submission with error will be replace with another one... on the LB I have one without ERROR that I have not selected as submission..." } ] }, { "id": 1693501, "postDate": "2022-02-16T17:45:00.573Z", "content": "I got a \"Notebook exceeded allowed compute\" error in my submission. Does anyone know what this could mean in this competition? Is it a memory issue?
", "rawMarkdown": "I got a \"Notebook exceeded allowed compute\" error in my submission. Does anyone know what this could mean in this competition? Is it a memory issue?" }, { "id": 1693197, "postDate": "2022-02-16T14:05:46.437Z", "content": "LB update do not reflect the new scores because it is still ordered by the old scores. What happened is some were moved to the end of the LB with -1 score either because of notebooks errors or another reason. This explains the move up in LB for the others.
", "rawMarkdown": "LB update do not reflect the new scores because it is still ordered by the old scores. What happened is some were moved to the end of the LB with -1 score either because of notebooks errors or another reason. This explains the move up in LB for the others." }, { "id": 1691351, "postDate": "2022-02-15T11:16:02.837Z", "content": "I got error in my two submissions. Has anyone used custom files in input directory and got a different error score?
", "rawMarkdown": "I got error in my two submissions. Has anyone used custom files in input directory and got a different error score?" }, { "id": 1691322, "postDate": "2022-02-15T10:50:21.483Z", "content": "Both my note books failed as well!
\nHave there been any general problems with the assessment test or should I be concerned? ;-)
", "rawMarkdown": "Both my note books failed as well!\n\nHave there been any general problems with the assessment test or should I be concerned? ;-)\n", "replies": [ { "id": 1693507, "postDate": "2022-02-16T17:47:07.800Z", "content": "Were you close to the time or memory limit?
", "rawMarkdown": "Were you close to the time or memory limit?" } ] }, { "id": 1690862, "postDate": "2022-02-15T06:14:35.143Z", "content": "could you also provide the data, you calculated the scores (up to the point) ?
", "rawMarkdown": "could you also provide the data, you calculated the scores (up to the point) ?" }, { "id": 1690785, "postDate": "2022-02-15T05:23:43.810Z", "content": "good luck to everyone
", "rawMarkdown": "good luck to everyone" }, { "id": 1690570, "postDate": "2022-02-15T02:26:59.737Z", "content": "Suddenly from 490th to 153th, is the first update have been published?
", "rawMarkdown": "Suddenly from 490th to 153th, is the first update have been published?", "replies": [ { "id": 1690585, "postDate": "2022-02-15T02:41:21.343Z", "content": "I don't think so. The 153rd place is shown based on your .9999 submission. Maybe, your spot changed because many people had .9999 submissions and they reran notebooks or did something and it moved your score up relative to all the other .9999 submissions. Some of the change can also be attributed to some people being removed from the competition which Kaggle does for cheating purposes at the end of any competition.
", "rawMarkdown": "I don't think so. The 153rd place is shown based on your .9999 submission. Maybe, your spot changed because many people had .9999 submissions and they reran notebooks or did something and it moved your score up relative to all the other .9999 submissions. Some of the change can also be attributed to some people being removed from the competition which Kaggle does for cheating purposes at the end of any competition. ", "votes": 1 }, { "id": 1690604, "postDate": "2022-02-15T02:56:51.837Z", "content": "Maybe you're right, I didn't choose .9999 submission 😂, I think the model has been updated, and my other selected scores have changed
", "rawMarkdown": "Maybe you're right, I didn't choose .9999 submission 😂, I think the model has been updated, and my other selected scores have changed" }, { "id": 1690624, "postDate": "2022-02-15T03:17:04.890Z", "content": "Did your selected score improve or do worse after the update?
", "rawMarkdown": "Did your selected score improve or do worse after the update?" }, { "id": 1690681, "postDate": "2022-02-15T03:59:00.633Z", "content": "Hi, what scores do you get? My two submissions get -0.0010 while originally the scores were over 0.02.
", "rawMarkdown": "Hi, what scores do you get? My two submissions get -0.0010 while originally the scores were over 0.02.", "votes": 2 }, { "id": 1690702, "postDate": "2022-02-15T04:14:11.493Z", "content": "Me too. One got -0.0024 (0.018 originally)
", "rawMarkdown": "Me too. One got -0.0024 (0.018 originally)" }, { "id": 1690718, "postDate": "2022-02-15T04:26:47.740Z", "content": "I got 0.005 (0.025 originally)
", "rawMarkdown": "I got 0.005 (0.025 originally)" }, { "id": 1690763, "postDate": "2022-02-15T05:02:34.137Z", "rawMarkdown": "", "isDeleted": true } ] }, { "id": 1690553, "postDate": "2022-02-15T02:10:47.840Z", "content": "We got submission error, is it normal?
", "rawMarkdown": "We got submission error, is it normal?", "replies": [ { "id": 1690564, "postDate": "2022-02-15T02:19:15.793Z", "content": "I got worse scores than I was expecting if that helps. He did say \"We may need longer to get the very first update out the door\" so there might be some errors on their side.
", "rawMarkdown": "I got worse scores than I was expecting if that helps. He did say \"We may need longer to get the very first update out the door\" so there might be some errors on their side." }, { "id": 1690569, "postDate": "2022-02-15T02:26:47.307Z", "content": "thanks. I have no idea what's going on 😂
", "rawMarkdown": "thanks. I have no idea what's going on 😂" }, { "id": 1690609, "postDate": "2022-02-15T03:05:54.687Z", "content": "I got submission scoring error as well…Also \"Notebook throw an Exception\"
", "rawMarkdown": "I got submission scoring error as well...Also \"Notebook throw an Exception\"" }, { "id": 1690716, "postDate": "2022-02-15T04:26:12.493Z", "content": "Both my note books failed as well, same error!
", "rawMarkdown": "Both my note books failed as well, same error!" }, { "id": 1690811, "postDate": "2022-02-15T05:36:39.967Z", "content": "my guess is, there was a problem in the evaluation implementation and the notebooks received errors, and the scores are for the notebooks where a last-ditch try-except manevour is implemented,, we will see…
", "rawMarkdown": "my guess is, there was a problem in the evaluation implementation and the notebooks received errors, and the scores are for the notebooks where a last-ditch try-except manevour is implemented,, we will see..." }, { "id": 1690855, "postDate": "2022-02-15T06:12:28.257Z", "content": "maybe not because then, for try-except notebooks, the score would be the same
", "rawMarkdown": "maybe not because then, for try-except notebooks, the score would be the same" }, { "id": 1690980, "postDate": "2022-02-15T07:15:17.270Z", "content": "I suspect that all of the positions on the leader board which are still extremely unrealistic and had not been removed had the same error. I also suspect there was/is a problem in the evaluation, but as you say, we will see…..
", "rawMarkdown": "I suspect that all of the positions on the leader board which are still extremely unrealistic and had not been removed had the same error. I also suspect there was/is a problem in the evaluation, but as you say, we will see....." }, { "id": 1691045, "postDate": "2022-02-15T07:48:44.327Z", "content": "Same here Notebook Threw Exception on both subs, so maybe they just try some test. Some extra days of data shouldn't create any problem for my subs.
", "rawMarkdown": "Same here Notebook Threw Exception on both subs, so maybe they just try some test. Some extra days of data shouldn't create any problem for my subs." }, { "id": 1691047, "postDate": "2022-02-15T07:50:56.650Z", "content": "One of my submissions (heavy) read the supplemental_train.csv and threw exception. While the other (light) didn't read the supplemental_train.csv and got a score of 0.009
\nIt seems to be a timeout error or a data loading/preprocessing error.
", "rawMarkdown": "One of my submissions (heavy) read the supplemental_train.csv and threw exception. While the other (light) didn't read the supplemental_train.csv and got a score of 0.009\n\nIt seems to be a timeout error or a data loading/preprocessing error.", "replies": [ { "id": 1691061, "postDate": "2022-02-15T08:03:34.957Z", "content": "\n\nOne of my submissions (heavy) read the supplemental_train.csv and threw exception. While the other (light) didn't read the supplemental_train.csv and got a score of 0.009
\nIt seems to be a timeout error or a data loading/preprocessing error.
\n
Each time when the supplemental train data was updated, my code \"pd.read_csv(../input/g-research-crypto-forecasting/supplemental_train.csv)\" would fail saying that \"cannot find the file\". I had to remove GR data and reloaded and then it would work each time.
", "rawMarkdown": "> One of my submissions (heavy) read the supplemental_train.csv and threw exception. While the other (light) didn't read the supplemental_train.csv and got a score of 0.009\n> \n> It seems to be a timeout error or a data loading/preprocessing error.\n\nEach time when the supplemental train data was updated, my code \"pd.read_csv(../input/g-research-crypto-forecasting/supplemental_train.csv)\" would fail saying that \"cannot find the file\". I had to remove GR data and reloaded and then it would work each time.", "votes": 3 } ] }, { "id": 1691055, "postDate": "2022-02-15T07:54:37.567Z", "content": "I guess its possible with additional data that the notebook runs out of memory; I did try cater for this but might have been on the cusp!!
", "rawMarkdown": "I guess its possible with additional data that the notebook runs out of memory; I did try cater for this but might have been on the cusp!!", "votes": 1, "replies": [ { "id": 1691060, "postDate": "2022-02-15T08:02:55.987Z", "rawMarkdown": "", "isDeleted": true } ] }, { "id": 1691122, "postDate": "2022-02-15T08:39:11.523Z", "content": "It’s something else than memory issue, they will just update with the rest of the 1/3 of the file. And I can’t see that the file is updated with all data at this time.
\n\n\n• supplemental_train.csv - After the submission period is over this file's data will be replaced with cryptoasset prices from the entire submission period. The current copy has been updated from the original and covers roughly 2/3 of the submission period.
\n
I have also done a memory test with double the supplemental_train data, which is more than the final data and it runs without memory problem, not even close.
", "rawMarkdown": "It’s something else than memory issue, they will just update with the rest of the 1/3 of the file. And I can’t see that the file is updated with all data at this time. \n\n> •\tsupplemental_train.csv - After the submission period is over this file's data will be replaced with cryptoasset prices from the entire submission period. The current copy has been updated from the original and covers roughly 2/3 of the submission period.\n\nI have also done a memory test with double the supplemental_train data, which is more than the final data and it runs without memory problem, not even close.", "votes": 1 }, { "id": 1691200, "postDate": "2022-02-15T09:40:04.933Z", "content": "my final thoughts on 'low scores' issue: they probably tested just the first few days of evaluation period, therefore the low/unstable scores.
", "rawMarkdown": "my final thoughts on 'low scores' issue: they probably tested just the first few days of evaluation period, therefore the low/unstable scores. ", "votes": 2 }, { "id": 1692765, "postDate": "2022-02-16T08:11:25.887Z", "content": "Memory shouldn't be the problem. I have 10G memory left when previously testing my code.
", "rawMarkdown": "Memory shouldn't be the problem. I have 10G memory left when previously testing my code." }, { "id": 1692766, "postDate": "2022-02-16T08:12:59.353Z", "content": "But it might be the problem of supplemental_train.csv. I noticed this in the training phase but I thought the hosts would fix it.
", "rawMarkdown": "But it might be the problem of supplemental_train.csv. I noticed this in the training phase but I thought the hosts would fix it." }, { "id": 1692774, "postDate": "2022-02-16T08:16:05.137Z", "content": "Also wouldn't have thought so but it seems to me that 100s of notebooks have failed.
", "rawMarkdown": "Also wouldn't have thought so but it seems to me that 100s of notebooks have failed." } ] }, { "id": 1674236, "postDate": "2022-02-03T10:41:29.810Z", "content": "Are we likely to have submissions run much slower due to all being scored at once? -> timeout errors ?
", "rawMarkdown": "Are we likely to have submissions run much slower due to all being scored at once? -> timeout errors ?", "replies": [ { "id": 1684745, "postDate": "2022-02-10T17:19:49.867Z", "content": "That shouldn't be an issue.
", "rawMarkdown": "That shouldn't be an issue.", "votes": 2 } ] }, { "id": 1674187, "postDate": "2022-02-03T09:40:42.030Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1674184, "postDate": "2022-02-03T09:39:09.640Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 1692741, "author_name": "Gabriel Preda", "author_url": "", "post_date": "2022-02-16T07:59:52.463000", "content": "I expect the shakeup for this competition to be like a rollercoaster.
", "votes": 5, "replies": [ { "id": 1694188, "author_name": "", "author_url": "", "post_date": "2022-02-17T08:17:28.097000", "content": "It seems that for some people, this competition will become a free lottery with prizes of $125,000.
", "votes": 4, "replies": [] }, { "id": 1694255, "author_name": "", "author_url": "", "post_date": "2022-02-17T09:49:05.957000", "content": "", "votes": 0, "replies": [] }, { "id": 1695345, "author_name": "Old Monk", "author_url": "", "post_date": "2022-02-18T05:00:55.517000", "content": "Yes, I was on top of the rollercoaster till yesterday and now at bottom, too numb to react 😦☹️😨🤒
", "votes": 0, "replies": [] }, { "id": 1695564, "author_name": "Marco Gorelli", "author_url": "", "post_date": "2022-02-18T08:15:55.703000", "content": "Not sure about that, I don't think I've ever read a winning solution write up in which it looked like the person just got lucky, and I don't think this competition will be different in this regard
", "votes": 3, "replies": [] }, { "id": 1696687, "author_name": "Alin Cijov", "author_url": "", "post_date": "2022-02-19T03:10:20.733000", "content": "Just epic sauce 😄 hold tight, this will be a bumpy ride !
", "votes": 0, "replies": [] } ] }, { "id": 1695534, "author_name": "Gabriel Preda", "author_url": "", "post_date": "2022-02-18T07:48:46.613000", "content": "Unfortunately, for me this shakeup was a totally crash: both my selected submissions reruns returned errors.
", "votes": 6, "replies": [ { "id": 1696291, "author_name": "Denis Serbin", "author_url": "", "post_date": "2022-02-18T18:08:04.373000", "content": "Maybe you used some external data or something like that?
\nOtherwise, it must be pretty frustrating, especially because they don't explain what exactly went wrong. My understanding was that all successful submissions made in the competition stage had to be successful in the evaluation stage.
\nThe message \"This version of the API is not optimized…\" kind of suggests that the submission process will be much better / faster in the evaluation stage. I might be wrong tough.
@denisserbin I got the same error in both of my notebooks and I didn't even use the supp data to train or evaluate my models.I only used the started training data. So I don't think supp data is the issue.
", "votes": 0, "replies": [] }, { "id": 1697012, "author_name": "Gabriel Preda", "author_url": "", "post_date": "2022-02-19T09:56:41.477000", "content": "No, no external data. One of them runs in less than 1h30 on GPU, without internet access and without external additional data.
", "votes": 1, "replies": [] } ] }, { "id": 1695129, "author_name": "Denis Serbin", "author_url": "", "post_date": "2022-02-18T01:13:57.910000", "content": "Well, at least the updated scores look like what you should expect in real life.
", "votes": 4, "replies": [] }, { "id": 1722280, "author_name": "neoistheone", "author_url": "", "post_date": "2022-03-14T11:48:28.717000", "content": "When is third update coming?
", "votes": 4, "replies": [] }, { "id": 1674399, "author_name": "Marco Gorelli", "author_url": "", "post_date": "2022-02-03T13:06:19.423000", "content": "Brace yourselves
\nthe shakeup is coming!
", "votes": 3, "replies": [ { "id": 1675407, "author_name": "Gabriel Preda", "author_url": "", "post_date": "2022-02-04T08:51:42.307000", "content": "Crypto market is waiting a shakeup as well :-).
", "votes": 4, "replies": [] } ] }, { "id": 1693117, "author_name": "Aditya Sharma", "author_url": "", "post_date": "2022-02-16T12:52:10.720000", "content": "Is the update reflecting on leaderboard because i can't see score in the leaderboard being updated?
\nWhat just happened here idk. Today only I saw kaggle competition section and my rank got up from 12 to 9. Is it good or bad?
It hasn't been reflected on the lb, but there was a change to the public lb. This will not be important though for the private lb evaluation phase.
", "votes": 1, "replies": [] }, { "id": 1694036, "author_name": "Aditya Sharma", "author_url": "", "post_date": "2022-02-17T05:51:46.620000", "content": "Ok. Thanks for the information.
", "votes": 0, "replies": [] } ] }, { "id": 1693027, "author_name": "Gabriel Preda", "author_url": "", "post_date": "2022-02-16T11:57:51.480000", "content": "Has anyone seen any score update after 1st round of reruns?
", "votes": 1, "replies": [ { "id": 1693084, "author_name": "Alan Kabisov", "author_url": "", "post_date": "2022-02-16T12:25:55.753000", "content": "I have updated score for 1 submission and 1 with -1 score, don't know why it happened
", "votes": 1, "replies": [] }, { "id": 1693130, "author_name": "Gabriel Preda", "author_url": "", "post_date": "2022-02-16T13:08:36.693000", "content": "I am confused, because I went up in the ranks but it is still that submission just for fun which is not included in my selected submissions list.
", "votes": 1, "replies": [] } ] }, { "id": 1692114, "author_name": "Alan Kabisov", "author_url": "", "post_date": "2022-02-15T20:39:21.433000", "content": "What does it mean if your submission succeed , but the score is -1.0? I have two exactly the same notebooks, one of them just have supplemental train. The one with supplemental shows no error, but -1.0 score.
", "votes": 1, "replies": [ { "id": 1692127, "author_name": "", "author_url": "", "post_date": "2022-02-15T21:02:35.650000", "content": "I think predicting value 0 for every entry will give score -1.
", "votes": 2, "replies": [] }, { "id": 1692585, "author_name": "Alan Kabisov", "author_url": "", "post_date": "2022-02-16T06:08:56.013000", "content": "It should not predict all zeros, because we have exactly the same notebooks with different data. May be we need to wait until the run finishes..
", "votes": 0, "replies": [] }, { "id": 1692589, "author_name": "", "author_url": "", "post_date": "2022-02-16T06:10:37.037000", "content": "", "votes": 0, "replies": [] }, { "id": 1692593, "author_name": "Alan Kabisov", "author_url": "", "post_date": "2022-02-16T06:13:01.453000", "content": "Yes, I understand it, but it's strange since they are the same notebooks
", "votes": 0, "replies": [] }, { "id": 1693184, "author_name": "Marco Gorelli", "author_url": "", "post_date": "2022-02-16T13:54:45.427000", "content": "Being the perfect opposite of the true values would be practically impossible - if you just switch your predictions, then you have a perfect forecast 😄
\nI think it just means it's an invalid score - perhaps something went wrong when joining with the supplemental data?
", "votes": 1, "replies": [] }, { "id": 1693191, "author_name": "Alan Kabisov", "author_url": "", "post_date": "2022-02-16T14:00:55.910000", "content": "I have made test runs before and everything was well. I have not used memory for full limit and had some free memory for additional data, I think it was enough for last 2 weeks of submission period. Anyway the notebook have run without any error and is marked as Succeed.
", "votes": 0, "replies": [] } ] }, { "id": 1675302, "author_name": "Gabriel Preda", "author_url": "", "post_date": "2022-02-04T07:15:16.980000", "content": "On 16th will be the private score already visible?
", "votes": 1, "replies": [] }, { "id": 1691672, "author_name": "JonnydosSantos", "author_url": "", "post_date": "2022-02-15T14:42:45.887000", "content": "Looked into the \"Notebook Threw Error\" found this explanation from the Kaggle staff. Looks like in previous competitions where this happened that the situation didn't fix itself and that no score was achieved with this error. :(
\nDustinKaggle Staff • 7 months ago • Options • Report • Reply
\n1
\n@pritommojumder Usually this means you're getting an error on the re-run of the submission that doesn't occur during the original run on the public dataset.
\nUsually this happens because of some assumptions that are being made about the private data which don't hold, but do for the public data.
", "votes": 2, "replies": [ { "id": 1691699, "author_name": "Kirderf", "author_url": "", "post_date": "2022-02-15T15:00:18.020000", "content": "Yes but in this case the only changes from the public run is that they added 1/3 of supplemental_train in the same format/file/location. My guess it will work in the real re-run as the private leaderboard isn't updated yet.
\nWith 2/3 of the 300Mb supplemental_train.csv everything went great, just adding the other 1/3 shouldn't be a problem for my models.
Agree, would not have thought it would have been an issue on myside either. Lets see what tomorrow brings.
", "votes": 1, "replies": [] } ] }, { "id": 1690551, "author_name": "Kyle Peters", "author_url": "", "post_date": "2022-02-15T02:05:13.037000", "content": "@sohier Is the update already in progress? I have different scores for the models I have submitted.
", "votes": 2, "replies": [] }, { "id": 1709421, "author_name": "Aditya Sharma", "author_url": "", "post_date": "2022-03-02T07:19:47.437000", "content": "When will the second update come.
", "votes": 0, "replies": [] }, { "id": 1707476, "author_name": "bturan19", "author_url": "", "post_date": "2022-02-28T13:53:11.900000", "content": "So, when will be the second update? :)
", "votes": 0, "replies": [ { "id": 1707507, "author_name": "JonnydosSantos", "author_url": "", "post_date": "2022-02-28T14:20:48.150000", "content": "2nd March I think
", "votes": 0, "replies": [] } ] }, { "id": 1694251, "author_name": "Gabriel Preda", "author_url": "", "post_date": "2022-02-17T09:43:12.170000", "content": "One thing that I do not understand is why 1.000 is still the best score (followed by many 0.999), if submissions are rerun? I suppose that most of us did not select submissions with 0.999 & 1.000 for private eval.
", "votes": 0, "replies": [] }, { "id": 1693929, "author_name": "ACMHIT", "author_url": "", "post_date": "2022-02-17T04:11:03.683000", "content": "My two submissions are now pure \"ERROR\" but not \"Note book throw an Exception\" or \"Score submission error\"😂It seems the hosts have rerun them again.
", "votes": 0, "replies": [ { "id": 1696695, "author_name": "Alin Cijov", "author_url": "", "post_date": "2022-02-19T03:18:47.890000", "content": "I wonder if the submission with error will be replace with another one… on the LB I have one without ERROR that I have not selected as submission…
", "votes": 0, "replies": [] } ] }, { "id": 1693501, "author_name": "ricc7777", "author_url": "", "post_date": "2022-02-16T17:45:00.573000", "content": "I got a \"Notebook exceeded allowed compute\" error in my submission. Does anyone know what this could mean in this competition? Is it a memory issue?
", "votes": 0, "replies": [] }, { "id": 1693197, "author_name": "Marius Badea", "author_url": "", "post_date": "2022-02-16T14:05:46.437000", "content": "LB update do not reflect the new scores because it is still ordered by the old scores. What happened is some were moved to the end of the LB with -1 score either because of notebooks errors or another reason. This explains the move up in LB for the others.
", "votes": 0, "replies": [] }, { "id": 1691351, "author_name": "ricardoCSAG", "author_url": "", "post_date": "2022-02-15T11:16:02.837000", "content": "I got error in my two submissions. Has anyone used custom files in input directory and got a different error score?
", "votes": 0, "replies": [] }, { "id": 1691322, "author_name": "ricardoCSAG", "author_url": "", "post_date": "2022-02-15T10:50:21.483000", "content": "Both my note books failed as well!
\nHave there been any general problems with the assessment test or should I be concerned? ;-)
", "votes": 0, "replies": [ { "id": 1693507, "author_name": "Kyle Peters", "author_url": "", "post_date": "2022-02-16T17:47:07.800000", "content": "Were you close to the time or memory limit?
", "votes": 0, "replies": [] } ] }, { "id": 1690862, "author_name": "GUNER", "author_url": "", "post_date": "2022-02-15T06:14:35.143000", "content": "could you also provide the data, you calculated the scores (up to the point) ?
", "votes": 0, "replies": [] }, { "id": 1690785, "author_name": "GUNER", "author_url": "", "post_date": "2022-02-15T05:23:43.810000", "content": "good luck to everyone
", "votes": 0, "replies": [] }, { "id": 1690570, "author_name": "fxalll", "author_url": "", "post_date": "2022-02-15T02:26:59.737000", "content": "Suddenly from 490th to 153th, is the first update have been published?
", "votes": 0, "replies": [ { "id": 1690585, "author_name": "Kyle Peters", "author_url": "", "post_date": "2022-02-15T02:41:21.343000", "content": "I don't think so. The 153rd place is shown based on your .9999 submission. Maybe, your spot changed because many people had .9999 submissions and they reran notebooks or did something and it moved your score up relative to all the other .9999 submissions. Some of the change can also be attributed to some people being removed from the competition which Kaggle does for cheating purposes at the end of any competition.
", "votes": 1, "replies": [] }, { "id": 1690604, "author_name": "fxalll", "author_url": "", "post_date": "2022-02-15T02:56:51.837000", "content": "Maybe you're right, I didn't choose .9999 submission 😂, I think the model has been updated, and my other selected scores have changed
", "votes": 0, "replies": [] }, { "id": 1690624, "author_name": "Kyle Peters", "author_url": "", "post_date": "2022-02-15T03:17:04.890000", "content": "Did your selected score improve or do worse after the update?
", "votes": 0, "replies": [] }, { "id": 1690681, "author_name": "Yue Sun", "author_url": "", "post_date": "2022-02-15T03:59:00.633000", "content": "Hi, what scores do you get? My two submissions get -0.0010 while originally the scores were over 0.02.
", "votes": 2, "replies": [] }, { "id": 1690702, "author_name": "fxalll", "author_url": "", "post_date": "2022-02-15T04:14:11.493000", "content": "Me too. One got -0.0024 (0.018 originally)
", "votes": 0, "replies": [] }, { "id": 1690718, "author_name": "Bachtuoc", "author_url": "", "post_date": "2022-02-15T04:26:47.740000", "content": "I got 0.005 (0.025 originally)
", "votes": 0, "replies": [] }, { "id": 1690763, "author_name": "", "author_url": "", "post_date": "2022-02-15T05:02:34.137000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 1690553, "author_name": "yuanzhe zhou", "author_url": "", "post_date": "2022-02-15T02:10:47.840000", "content": "We got submission error, is it normal?
", "votes": 0, "replies": [ { "id": 1690564, "author_name": "Kyle Peters", "author_url": "", "post_date": "2022-02-15T02:19:15.793000", "content": "I got worse scores than I was expecting if that helps. He did say \"We may need longer to get the very first update out the door\" so there might be some errors on their side.
", "votes": 0, "replies": [] }, { "id": 1690569, "author_name": "yuanzhe zhou", "author_url": "", "post_date": "2022-02-15T02:26:47.307000", "content": "thanks. I have no idea what's going on 😂
", "votes": 0, "replies": [] }, { "id": 1690609, "author_name": "ACMHIT", "author_url": "", "post_date": "2022-02-15T03:05:54.687000", "content": "I got submission scoring error as well…Also \"Notebook throw an Exception\"
", "votes": 0, "replies": [] }, { "id": 1690716, "author_name": "JonnydosSantos", "author_url": "", "post_date": "2022-02-15T04:26:12.493000", "content": "Both my note books failed as well, same error!
", "votes": 0, "replies": [] }, { "id": 1690811, "author_name": "GUNER", "author_url": "", "post_date": "2022-02-15T05:36:39.967000", "content": "my guess is, there was a problem in the evaluation implementation and the notebooks received errors, and the scores are for the notebooks where a last-ditch try-except manevour is implemented,, we will see…
", "votes": 0, "replies": [] }, { "id": 1690855, "author_name": "GUNER", "author_url": "", "post_date": "2022-02-15T06:12:28.257000", "content": "maybe not because then, for try-except notebooks, the score would be the same
", "votes": 0, "replies": [] }, { "id": 1690980, "author_name": "JonnydosSantos", "author_url": "", "post_date": "2022-02-15T07:15:17.270000", "content": "I suspect that all of the positions on the leader board which are still extremely unrealistic and had not been removed had the same error. I also suspect there was/is a problem in the evaluation, but as you say, we will see…..
", "votes": 0, "replies": [] }, { "id": 1691045, "author_name": "Kirderf", "author_url": "", "post_date": "2022-02-15T07:48:44.327000", "content": "Same here Notebook Threw Exception on both subs, so maybe they just try some test. Some extra days of data shouldn't create any problem for my subs.
", "votes": 0, "replies": [] }, { "id": 1691047, "author_name": " 9hz", "author_url": "", "post_date": "2022-02-15T07:50:56.650000", "content": "One of my submissions (heavy) read the supplemental_train.csv and threw exception. While the other (light) didn't read the supplemental_train.csv and got a score of 0.009
\nIt seems to be a timeout error or a data loading/preprocessing error.
", "votes": 0, "replies": [ { "id": 1691061, "author_name": "Yue Sun", "author_url": "", "post_date": "2022-02-15T08:03:34.957000", "content": "\n\nOne of my submissions (heavy) read the supplemental_train.csv and threw exception. While the other (light) didn't read the supplemental_train.csv and got a score of 0.009
\nIt seems to be a timeout error or a data loading/preprocessing error.
\n
Each time when the supplemental train data was updated, my code \"pd.read_csv(../input/g-research-crypto-forecasting/supplemental_train.csv)\" would fail saying that \"cannot find the file\". I had to remove GR data and reloaded and then it would work each time.
", "votes": 3, "replies": [] } ] }, { "id": 1691055, "author_name": "JonnydosSantos", "author_url": "", "post_date": "2022-02-15T07:54:37.567000", "content": "I guess its possible with additional data that the notebook runs out of memory; I did try cater for this but might have been on the cusp!!
", "votes": 1, "replies": [ { "id": 1691060, "author_name": "", "author_url": "", "post_date": "2022-02-15T08:02:55.987000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 1691122, "author_name": "Kirderf", "author_url": "", "post_date": "2022-02-15T08:39:11.523000", "content": "It’s something else than memory issue, they will just update with the rest of the 1/3 of the file. And I can’t see that the file is updated with all data at this time.
\n\n\n• supplemental_train.csv - After the submission period is over this file's data will be replaced with cryptoasset prices from the entire submission period. The current copy has been updated from the original and covers roughly 2/3 of the submission period.
\n
I have also done a memory test with double the supplemental_train data, which is more than the final data and it runs without memory problem, not even close.
", "votes": 1, "replies": [] }, { "id": 1691200, "author_name": "GUNER", "author_url": "", "post_date": "2022-02-15T09:40:04.933000", "content": "my final thoughts on 'low scores' issue: they probably tested just the first few days of evaluation period, therefore the low/unstable scores.
", "votes": 2, "replies": [] }, { "id": 1692765, "author_name": "ACMHIT", "author_url": "", "post_date": "2022-02-16T08:11:25.887000", "content": "Memory shouldn't be the problem. I have 10G memory left when previously testing my code.
", "votes": 0, "replies": [] }, { "id": 1692766, "author_name": "ACMHIT", "author_url": "", "post_date": "2022-02-16T08:12:59.353000", "content": "But it might be the problem of supplemental_train.csv. I noticed this in the training phase but I thought the hosts would fix it.
", "votes": 0, "replies": [] }, { "id": 1692774, "author_name": "JonnydosSantos", "author_url": "", "post_date": "2022-02-16T08:16:05.137000", "content": "Also wouldn't have thought so but it seems to me that 100s of notebooks have failed.
", "votes": 0, "replies": [] } ] }, { "id": 1674236, "author_name": "JM", "author_url": "", "post_date": "2022-02-03T10:41:29.810000", "content": "Are we likely to have submissions run much slower due to all being scored at once? -> timeout errors ?
", "votes": 0, "replies": [ { "id": 1684745, "author_name": "Sohier Dane", "author_url": "", "post_date": "2022-02-10T17:19:49.867000", "content": "That shouldn't be an issue.
", "votes": 2, "replies": [] } ] }, { "id": 1674187, "author_name": "", "author_url": "", "post_date": "2022-02-03T09:40:42.030000", "content": "", "votes": 0, "replies": [] }, { "id": 1674184, "author_name": "", "author_url": "", "post_date": "2022-02-03T09:39:09.640000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1673764": "We''ll be updating the scores roughly once every two weeks. We may need longer to get the very first update out the door, but my goal is to publish the first update on the 16th.", "1692741": "I expect the shakeup for this competition to be like a rollercoaster.", "1695534": "Unfortunately, for me this shakeup was a totally crash: both my selected submissions reruns returned errors.", "1695129": "Well, at least the updated scores look like what you should expect in real life.", "1722280": "When is third update coming?", "1674399": "Brace yourselves\n\nthe shakeup is coming!", "1693117": "Is the update reflecting on leaderboard because i can't see score in the leaderboard being updated?\nWhat just happened here idk. Today only I saw kaggle competition section and my rank got up from 12 to 9. Is it good or bad? ", "1693027": "Has anyone seen any score update after 1st round of reruns?", "1692114": "What does it mean if your submission succeed , but the score is -1.0? I have two exactly the same notebooks, one of them just have supplemental train. The one with supplemental shows no error, but -1.0 score.", "1675302": "On 16th will be the private score already visible?", "1691672": "Looked into the \"Notebook Threw Error\" found this explanation from the Kaggle staff. Looks like in previous competitions where this happened that the situation didn't fix itself and that no score was achieved with this error. :( \n\nDustinKaggle Staff • 7 months ago • Options • Report • Reply\n\n1\n\n@pritommojumder Usually this means you're getting an error on the re-run of the submission that doesn't occur during the original run on the public dataset.\n\nUsually this happens because of some assumptions that are being made about the private data which don't hold, but do for the public data.", "1690551": "@sohier Is the update already in progress? I have different scores for the models I have submitted.", "1709421": "When will the second update come.", "1707476": "So, when will be the second update? :)", "1694251": "One thing that I do not understand is why 1.000 is still the best score (followed by many 0.999), if submissions are rerun? I suppose that most of us did not select submissions with 0.999 & 1.000 for private eval.", "1693929": "My two submissions are now pure \"ERROR\" but not \"Note book throw an Exception\" or \"Score submission error\"😂It seems the hosts have rerun them again.", "1693501": "I got a \"Notebook exceeded allowed compute\" error in my submission. Does anyone know what this could mean in this competition? Is it a memory issue?", "1693197": "LB update do not reflect the new scores because it is still ordered by the old scores. What happened is some were moved to the end of the LB with -1 score either because of notebooks errors or another reason. This explains the move up in LB for the others.", "1691351": "I got error in my two submissions. Has anyone used custom files in input directory and got a different error score?", "1691322": "Both my note books failed as well!\n\nHave there been any general problems with the assessment test or should I be concerned? ;-)\n", "1690862": "could you also provide the data, you calculated the scores (up to the point) ?", "1690785": "good luck to everyone", "1690570": "Suddenly from 490th to 153th, is the first update have been published?", "1690553": "We got submission error, is it normal?", "1674236": "Are we likely to have submissions run much slower due to all being scored at once? -> timeout errors ?", "1674187": "", "1674184": "" } }