After too many submissions to debug why my notebooks are receiving submission scoring error, these are my findings:
\nAll the previous statements are based on actual submissions
\nexample:
for (test_df, sample_prediction_df) in iter_test:\n if (test_df.timestamp.values%60).max()>0:\n raise\n env.predict(sample_prediction_df)\nThis one is based on someone's else comment in the forum, I didn't test it, they said their submission fails if they use test_df.timestamp[0] but it works when they use test_df.timestamp.iat[0], which indicates that test_df index does not necessary starts by 0
This one I didn't test but just reminding you that the train data contains few inf values in VWAP
-As for notebook timeouts error, I tested dummy submissions with time.sleep()
\nfor (test_df, sample_prediction_df) in iter_test:\n time.sleep(0.25) \n env.predict(sample_prediction_df)\n^ above timed out
\ntime.sleep(300) \nfor (test_df, sample_prediction_df) in iter_test:\n\n time.sleep(0.23) \n env.predict(sample_prediction_df)\n^ timed out
\ntime.sleep(300) \nfor (test_df, sample_prediction_df) in iter_test:\n time.sleep(0.22) \n env.predict(sample_prediction_df)\n^ this passed
\nThe hidden test set is expected to be 3 months of data (129600 iterations) and notebook runtime limit is 9 hours (32400 seconds) which means you shouldn't exceed 0.25 seconds per iteration (this is rough estimates)
\nAccording to competition timeline
\n\"Updates will take place roughly every two weeks\"
\nif this means that our submitted notebook will be ran on new data every two weeks, each run has 9 hours runtime, then I expect the hidden test set to be two weeks of data rather than 3 months, but unfortunately this is not the case
After too many submissions to debug why my notebooks are receiving submission scoring error, these are my findings:
\nAll the previous statements are based on actual submissions
\nexample:
for (test_df, sample_prediction_df) in iter_test:\n if (test_df.timestamp.values%60).max()>0:\n raise\n env.predict(sample_prediction_df)\nThis one is based on someone's else comment in the forum, I didn't test it, they said their submission fails if they use test_df.timestamp[0] but it works when they use test_df.timestamp.iat[0], which indicates that test_df index does not necessary starts by 0
This one I didn't test but just reminding you that the train data contains few inf values in VWAP
-As for notebook timeouts error, I tested dummy submissions with time.sleep()
\nfor (test_df, sample_prediction_df) in iter_test:\n time.sleep(0.25) \n env.predict(sample_prediction_df)\n^ above timed out
\ntime.sleep(300) \nfor (test_df, sample_prediction_df) in iter_test:\n\n time.sleep(0.23) \n env.predict(sample_prediction_df)\n^ timed out
\ntime.sleep(300) \nfor (test_df, sample_prediction_df) in iter_test:\n time.sleep(0.22) \n env.predict(sample_prediction_df)\n^ this passed
\nThe hidden test set is expected to be 3 months of data (129600 iterations) and notebook runtime limit is 9 hours (32400 seconds) which means you shouldn't exceed 0.25 seconds per iteration (this is rough estimates)
\nAccording to competition timeline
\n\"Updates will take place roughly every two weeks\"
\nif this means that our submitted notebook will be ran on new data every two weeks, each run has 9 hours runtime, then I expect the hidden test set to be two weeks of data rather than 3 months, but unfortunately this is not the case
Adding one more reason causing submission scoring error which is that there are definitely missing cryptos in the test set i.e. it is not always delivering 14 cryptos so your submission needs to account for this. If not, there likely be an exception that causes the scoring error. Hope this helps those reading this excellent thread!
", "rawMarkdown": "Adding one more reason causing submission scoring error which is that there are definitely missing cryptos in the test set i.e. it is not always delivering 14 cryptos so your submission needs to account for this. If not, there likely be an exception that causes the scoring error. Hope this helps those reading this excellent thread!", "votes": 3 }, { "id": 1637473, "postDate": "2022-01-03T23:22:06.693Z", "content": "Hi @shadymokh - interesting that you've drawn attention to the \"updates will take place roughly every two weeks\" statement - I'd been assuming that we should plan for the 3 months of data to be dealt within the 9 hour runtime, so was a bit worried I'd missed something.
\nTwo possiblilities that cohere with the 9 hours of runtime, 3 months of data, and fortnightly updates:
\nMy guess is option 2. Trying to implement 1. would be a lot of unnecessary hassle.
\nA fun twist with option 2 is that anyone who is just over the runtime constraint for the full 3 months of data will only find out right at the last update. ๐ฆ.
", "rawMarkdown": "Hi @shadymokh - interesting that you've drawn attention to the *\"updates will take place roughly every two weeks\"* statement - I'd been assuming that we should plan for the 3 months of data to be dealt within the 9 hour runtime, so was a bit worried I'd missed something.\n\nTwo possiblilities that cohere with the 9 hours of runtime, 3 months of data, and fortnightly updates:\n1. We only need to plan for 2 weeks worth of test data (~ 2 * 7 * 24 * 60 = 20160) at each fortnightly update, for which we have 9 hours runtime. \n2. We really do need to plan for 3 months worth of test data for the 9hrs runtime. At each update, all of the test data accumulated up until that point is used in the test set. After 2 weeks, 2 weeks worth of test data is used; after 4 weeks, 4 weeks worth, etc... \n\nMy guess is option 2. Trying to implement 1. would be a lot of unnecessary hassle.\n\nA fun twist with option 2 is that anyone who is *just* over the runtime constraint for the full 3 months of data will only find out right at the last update. ๐ฆ.", "votes": 3, "replies": [ { "id": 1641195, "postDate": "2022-01-07T07:14:27.563Z", "content": "I have a strong suspicion that option 2 is the most likely.
", "rawMarkdown": "I have a strong suspicion that option 2 is the most likely.", "votes": 1 } ] }, { "id": 1628733, "postDate": "2021-12-25T07:32:31.560Z", "content": "Hi, great findings. Here is my estimation:
\n1. 300 + 0.22x < 32400\n2. 300 + 0.23x > 32400\nthe number of loops x in the hidden test set: 140739.13 < x < 145909.09
\nEach loop represent one minute information. So the time length y of hidden test set is 97.74 days < y < 101.36 days.
can you explain why between these.thank you
", "rawMarkdown": "can you explain why between these.thank you", "votes": 2 }, { "id": 1637483, "postDate": "2022-01-03T23:33:42.983Z", "content": "Hi,
\nThe number of loops is x. 9 hours = 32400 seconds.
time.sleep(300) \nfor (test_df, sample_prediction_df) in iter_test:\n time.sleep(0.22) \n env.predict(sample_prediction_df)\n^ this passed
\nTherefore, 300 + 0.22x < 32400.
time.sleep(300) \nfor (test_df, sample_prediction_df) in iter_test:\n\n time.sleep(0.23) \n env.predict(sample_prediction_df)\n^ timed out
\nTherefore, 300 + 0.23x > 32400.
\nThen, we solve the inequalities and get 40739.13 < x < 145909.09. Each loop represent one minute time window. So we get 40739.13 * 1 min < y < 145909.09 * 1 min => 97.74 days < y < 101.36 days.
๐Thank you, I got it.
", "rawMarkdown": "๐Thank you, I got it.", "votes": 2 }, { "id": 1637955, "postDate": "2022-01-04T10:28:16.230Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1638743, "postDate": "2022-01-05T01:34:05.297Z", "content": "Thank you. When I test submission, I always failed, but given no details about fail. Only give me the status: Notebook Exceeded Allowed Compute or Submission Scoring Error. I get the test_df's timestamp and then put this timestamp's Target from train.csv into pred_df. Can you tell me more about it. I paticipate in this competition before a few days.๐
", "rawMarkdown": "Thank you. When I test submission, I always failed, but given no details about fail. Only give me the status: ***Notebook Exceeded Allowed Compute or Submission Scoring Error***. I get the test_df's timestamp and then put this timestamp's Target from train.csv into pred_df. Can you tell me more about it. I paticipate in this competition before a few days.๐", "votes": 1 }, { "id": 1639744, "postDate": "2022-01-05T21:44:47.570Z", "content": "@hayden234 have you checked out this post from kaggle? I'd suggest reading that and looking at some of the public notebooks that do submit without error, and see what's different with your notebook to compare.
\nYou could also try using the local API emulator that I have produced - if your code fails there then you should at least get a local debuggable error that might suggest what's going wrong with your code.
", "rawMarkdown": "@hayden234 have you checked out [this post](https://www.kaggle.com/code-competition-debugging) from kaggle? I'd suggest reading that and looking at some of the public notebooks that do submit without error, and see what's different with your notebook to compare. \n\nYou could also try using the [local API emulator](https://www.kaggle.com/jagofc/local-api-emulator?scriptVersionId=82773758&cellId=29) that I have produced - if your code fails there then you should at least get a local debuggable error that might suggest what's going wrong with your code.", "votes": 4 }, { "id": 1640193, "postDate": "2022-01-06T09:17:43.863Z", "content": "@hayden234 If you are looking for the exact test_df's timestamp in train.csv, maybe this is the issue, try looking for the closest timestamp instead, I confirmed that there are timestamps in test that are not in train, this could explain why max leaderboard score is 0.99 and not 1.
\nin my post:
Timestamps in the hidden set are are not always 0 seconds unlike the training data, this one gave me the hardest time in the competition till I discovered it.\nThis notebook is doing what you want to achieve
\nhttps://www.kaggle.com/julian3833/g-research-using-the-overlap-fully-lb-0-99
\nthey added try/except
@jagofc, you are very nice. local API emulator is useful for me. Errors & Debugging Tips also help me to understand where I have errors.๐
", "rawMarkdown": "@jagofc, you are very nice. [local API emulator](https://www.kaggle.com/jagofc/local-api-emulator?scriptVersionId=82773758&cellId=29) is useful for me. Errors & Debugging Tips also help me to understand where I have errors.๐", "votes": 2 }, { "id": 1640441, "postDate": "2022-01-06T13:42:03.183Z", "content": "@Shady Mokh, Thank you for sharing this notebook for me. I'm going to debug๐
", "rawMarkdown": "@Shady Mokh, Thank you for sharing this notebook for me. I'm going to debug๐", "votes": 1 }, { "id": 1640716, "postDate": "2022-01-06T18:23:06.137Z", "content": "@shadymokh - I think I might have missed something here. Could you clarify what you mean in your comment
\n\n\nTimestamps in the hidden set are are not always 0 seconds unlike the training data
\n
I do see that the timestamp sets between public LB test and train.csv are not exactly the same, but have you pinned down the difference more precisely than that?
", "rawMarkdown": "@shadymokh - I think I might have missed something here. Could you clarify what you mean in your comment\n> Timestamps in the hidden set are are not always 0 seconds unlike the training data\n\nI do see that the timestamp sets between public LB test and train.csv are not exactly the same, but have you pinned down the difference more precisely than that?", "votes": 1 }, { "id": 1640753, "postDate": "2022-01-06T19:22:32.053Z", "content": "@jagofc
\nall timestamps in train set are divisible by 60, (timestamp % 60 =0 ) which means that all seconds are zeroes
for instance:
\nThursday, January 6, 2022 7:08:42 PM --> 1641496122
\nThe above time is 7 hrs, 42 mins and 42 seconds (seconds is not 0, its 42) hence, the corresponding timestamp 1641496122 is not divisible by 60
while this one Thursday, January 6, 2022 7:09:00 PM --> 1641496140
\n7 hrs, 8 mins and 0 seconds (seconds is 0) so this timestamp 1641496140 is divisible by 60
so you can find that all seconds in training data timestamps are 0 seconds (let me know if i am not clear enough)
\nThe reason i say this is not the case in test data is because this submission fails
\nfor (test_df, sample_prediction_df) in iter_test:\n if (test_df.timestamp.values%60).max()>0:\n raise\n env.predict(sample_prediction_df)\nperhaps you don't need to worry about all this, but my submission was failing because my code assumes that all seconds are zeroes which is not the case for everyone else's codes/solutions
", "rawMarkdown": "@jagofc \nall timestamps in train set are divisible by 60, (timestamp % 60 =0 ) which means that all seconds are zeroes\n\nfor instance:\nThursday, January 6, 2022 7:08:**42** PM --> 1641496122\nThe above time is 7 hrs, 42 mins and **42 seconds** (seconds is not 0, its 42) hence, the corresponding timestamp 1641496122 is not divisible by 60\n\nwhile this one Thursday, January 6, 2022 7:09:**00** PM --> 1641496140\n7 hrs, 8 mins and **0 seconds** (seconds is 0) so this timestamp 1641496140 is divisible by 60\n\nso you can find that all seconds in training data timestamps are 0 seconds (let me know if i am not clear enough)\n\nThe reason i say this is not the case in test data is because this submission fails\n```\n\nfor (test_df, sample_prediction_df) in iter_test:\n if (test_df.timestamp.values%60).max()>0:\n raise\n env.predict(sample_prediction_df)\n\n\n```\nperhaps you don't need to worry about all this, but my submission was failing because my code assumes that all seconds are zeroes which is not the case for everyone else's codes/solutions\n", "votes": 3 }, { "id": 1640887, "postDate": "2022-01-06T23:27:29.953Z", "content": "Thanks for explaining in detail Shady, I had also been expecting the timestamps to be divisible by 60 and I am quite surprised by your finding. Here is something little to add to it.
\nThis code passes, so all timestamps in each df_data iteration are the same:
\nfor (df_data, df_pred) in iter_test:\n if len(df_data.timestamp.unique()) == 1:\n df_pred.loc[:,'Target'] = 0.01 * np.random.randn(len(df_pred), 1)\n else:\n raise RuntimeError()\n env.predict(df_pred)\nA further experiment:
\nThe following strategy is successful with LB = 0.9999.
\ndef predict_from_train(df_data, df_pred):\n ts = df_data.iat[0,0]\n df = pd.merge(df_data[['Asset_ID', 'row_id']],\n train_df[train_df.timestamp==ts][['Asset_ID', 'Target']],\n on='Asset_ID')\n df = df.drop(columns=['Asset_ID'])\n df_pred = (pd.merge(df_pred[['row_id']], df,\n on='row_id', how='left')\n .fillna(0.))\n return df_pred\nenv = gresearch_crypto.make_env()\niter_test = env.iter_test()\ntrain_timestamps = train_df.timestamp.unique()\nstrategy = 0\n\nfor (df_data, df_pred) in iter_test:\n\n if strategy == 0:\n ts = df_data.iat[0,0]\n if ts in train_timestamps:\n df_pred = predict_from_train(df_data, df_pred)\n else:\n df_pred.loc[:,'Target'] = 0\n strategy = 1\n else:\n df_pred.loc[:,'Target'] = 0\n\n env.predict(df_pred)\nTogether with the result of the unique timestamp code, this means any non-train set timestamps must be very near the end. Otherwise the strategy would switch from 0 to 1 earlier and the zero predictions would bring the LB score lower than 0.9999. Together with @shadymokh 's (mod 60) code this means that any non-zero (mod 60) timestamp must also be very near the end.
", "rawMarkdown": "A further experiment: \n\nThe following strategy is successful with LB = 0.9999.\n\n```\ndef predict_from_train(df_data, df_pred):\n ts = df_data.iat[0,0]\n df = pd.merge(df_data[['Asset_ID', 'row_id']],\n train_df[train_df.timestamp==ts][['Asset_ID', 'Target']],\n on='Asset_ID')\n df = df.drop(columns=['Asset_ID'])\n df_pred = (pd.merge(df_pred[['row_id']], df,\n on='row_id', how='left')\n .fillna(0.))\n return df_pred\n```\n\n```\nenv = gresearch_crypto.make_env()\niter_test = env.iter_test()\ntrain_timestamps = train_df.timestamp.unique()\nstrategy = 0\n\nfor (df_data, df_pred) in iter_test:\n \n if strategy == 0:\n ts = df_data.iat[0,0]\n if ts in train_timestamps:\n df_pred = predict_from_train(df_data, df_pred)\n else:\n df_pred.loc[:,'Target'] = 0\n strategy = 1\n else:\n df_pred.loc[:,'Target'] = 0\n \n env.predict(df_pred)\n\n```\n\nTogether with the result of the unique timestamp code, this means any non-train set timestamps must be very near the end. Otherwise the strategy would switch from 0 to 1 earlier and the zero predictions would bring the LB score lower than 0.9999. Together with @shadymokh 's (mod 60) code this means that any non-zero (mod 60) timestamp must also be very near the end. ", "votes": 3 }, { "id": 1642297, "postDate": "2022-01-08T08:44:00.870Z", "content": "Just in case anyone is still reading :D
\ntrain_timestamps = train_df.timestamp.unique()\n\nfor (df_data, df_pred) in iter_test:\n\n ts = df_data.iat[0,0]\n if ts > train_timestamps.max() + T:\n raise RuntimeError()\n\n env.predict(df_pred)\nWith T = 0 this fails. With T = 3600 it passes.
This means that there are a limited number of additional timestamps delivered by the API for the public LB beyond the end of the train.csv.
\nAt this point I think there are diminishing returns to pursuing further: for all we know the additional timestamps might just be contrived test cases to stop us writing code that relies on particular assumptions (like @shadymokh 's != 0 % 60 finding); or to frustrate us by not being able to perfectly reproduce the score of the API.
", "rawMarkdown": "Just in case anyone is still reading :D\n\n```\ntrain_timestamps = train_df.timestamp.unique()\n\nfor (df_data, df_pred) in iter_test:\n \n ts = df_data.iat[0,0]\n if ts > train_timestamps.max() + T:\n raise RuntimeError()\n \n env.predict(df_pred)\n```\n\nWith `T = 0` this fails. With `T = 3600` it passes.\n\nThis means that there are a limited number of additional timestamps delivered by the API for the public LB beyond the end of the train.csv. \n\nAt this point I think there are diminishing returns to pursuing further: for all we know the additional timestamps might just be contrived test cases to stop us writing code that relies on particular assumptions (like @shadymokh 's != 0 % 60 finding); or to frustrate us by not being able to perfectly reproduce the score of the API.", "votes": 3 } ] } ], "comments": [ { "id": 1642198, "author_name": "Lu Chung I", "author_url": "", "post_date": "2022-01-08T06:28:06.383000", "content": "Adding one more reason causing submission scoring error which is that there are definitely missing cryptos in the test set i.e. it is not always delivering 14 cryptos so your submission needs to account for this. If not, there likely be an exception that causes the scoring error. Hope this helps those reading this excellent thread!
", "votes": 3, "replies": [] }, { "id": 1637473, "author_name": "jagofc", "author_url": "", "post_date": "2022-01-03T23:22:06.693000", "content": "Hi @shadymokh - interesting that you've drawn attention to the \"updates will take place roughly every two weeks\" statement - I'd been assuming that we should plan for the 3 months of data to be dealt within the 9 hour runtime, so was a bit worried I'd missed something.
\nTwo possiblilities that cohere with the 9 hours of runtime, 3 months of data, and fortnightly updates:
\nMy guess is option 2. Trying to implement 1. would be a lot of unnecessary hassle.
\nA fun twist with option 2 is that anyone who is just over the runtime constraint for the full 3 months of data will only find out right at the last update. ๐ฆ.
", "votes": 3, "replies": [ { "id": 1641195, "author_name": "JonnydosSantos", "author_url": "", "post_date": "2022-01-07T07:14:27.563000", "content": "I have a strong suspicion that option 2 is the most likely.
", "votes": 1, "replies": [] } ] }, { "id": 1628733, "author_name": "Yue Sun", "author_url": "", "post_date": "2021-12-25T07:32:31.560000", "content": "Hi, great findings. Here is my estimation:
\n1. 300 + 0.22x < 32400\n2. 300 + 0.23x > 32400\nthe number of loops x in the hidden test set: 140739.13 < x < 145909.09
\nEach loop represent one minute information. So the time length y of hidden test set is 97.74 days < y < 101.36 days.
can you explain why between these.thank you
", "votes": 2, "replies": [] }, { "id": 1637483, "author_name": "Yue Sun", "author_url": "", "post_date": "2022-01-03T23:33:42.983000", "content": "Hi,
\nThe number of loops is x. 9 hours = 32400 seconds.
time.sleep(300) \nfor (test_df, sample_prediction_df) in iter_test:\n time.sleep(0.22) \n env.predict(sample_prediction_df)\n^ this passed
\nTherefore, 300 + 0.22x < 32400.
time.sleep(300) \nfor (test_df, sample_prediction_df) in iter_test:\n\n time.sleep(0.23) \n env.predict(sample_prediction_df)\n^ timed out
\nTherefore, 300 + 0.23x > 32400.
\nThen, we solve the inequalities and get 40739.13 < x < 145909.09. Each loop represent one minute time window. So we get 40739.13 * 1 min < y < 145909.09 * 1 min => 97.74 days < y < 101.36 days.
๐Thank you, I got it.
", "votes": 2, "replies": [] }, { "id": 1637955, "author_name": "", "author_url": "", "post_date": "2022-01-04T10:28:16.230000", "content": "", "votes": 0, "replies": [] }, { "id": 1638743, "author_name": "Hayden234", "author_url": "", "post_date": "2022-01-05T01:34:05.297000", "content": "Thank you. When I test submission, I always failed, but given no details about fail. Only give me the status: Notebook Exceeded Allowed Compute or Submission Scoring Error. I get the test_df's timestamp and then put this timestamp's Target from train.csv into pred_df. Can you tell me more about it. I paticipate in this competition before a few days.๐
", "votes": 1, "replies": [] }, { "id": 1639744, "author_name": "jagofc", "author_url": "", "post_date": "2022-01-05T21:44:47.570000", "content": "@hayden234 have you checked out this post from kaggle? I'd suggest reading that and looking at some of the public notebooks that do submit without error, and see what's different with your notebook to compare.
\nYou could also try using the local API emulator that I have produced - if your code fails there then you should at least get a local debuggable error that might suggest what's going wrong with your code.
", "votes": 4, "replies": [] }, { "id": 1640193, "author_name": "Shady Mokh", "author_url": "", "post_date": "2022-01-06T09:17:43.863000", "content": "@hayden234 If you are looking for the exact test_df's timestamp in train.csv, maybe this is the issue, try looking for the closest timestamp instead, I confirmed that there are timestamps in test that are not in train, this could explain why max leaderboard score is 0.99 and not 1.
\nin my post:
Timestamps in the hidden set are are not always 0 seconds unlike the training data, this one gave me the hardest time in the competition till I discovered it.\nThis notebook is doing what you want to achieve
\nhttps://www.kaggle.com/julian3833/g-research-using-the-overlap-fully-lb-0-99
\nthey added try/except
@jagofc, you are very nice. local API emulator is useful for me. Errors & Debugging Tips also help me to understand where I have errors.๐
", "votes": 2, "replies": [] }, { "id": 1640441, "author_name": "Hayden234", "author_url": "", "post_date": "2022-01-06T13:42:03.183000", "content": "@Shady Mokh, Thank you for sharing this notebook for me. I'm going to debug๐
", "votes": 1, "replies": [] }, { "id": 1640716, "author_name": "jagofc", "author_url": "", "post_date": "2022-01-06T18:23:06.137000", "content": "@shadymokh - I think I might have missed something here. Could you clarify what you mean in your comment
\n\n\nTimestamps in the hidden set are are not always 0 seconds unlike the training data
\n
I do see that the timestamp sets between public LB test and train.csv are not exactly the same, but have you pinned down the difference more precisely than that?
", "votes": 1, "replies": [] }, { "id": 1640753, "author_name": "Shady Mokh", "author_url": "", "post_date": "2022-01-06T19:22:32.053000", "content": "@jagofc
\nall timestamps in train set are divisible by 60, (timestamp % 60 =0 ) which means that all seconds are zeroes
for instance:
\nThursday, January 6, 2022 7:08:42 PM --> 1641496122
\nThe above time is 7 hrs, 42 mins and 42 seconds (seconds is not 0, its 42) hence, the corresponding timestamp 1641496122 is not divisible by 60
while this one Thursday, January 6, 2022 7:09:00 PM --> 1641496140
\n7 hrs, 8 mins and 0 seconds (seconds is 0) so this timestamp 1641496140 is divisible by 60
so you can find that all seconds in training data timestamps are 0 seconds (let me know if i am not clear enough)
\nThe reason i say this is not the case in test data is because this submission fails
\nfor (test_df, sample_prediction_df) in iter_test:\n if (test_df.timestamp.values%60).max()>0:\n raise\n env.predict(sample_prediction_df)\nperhaps you don't need to worry about all this, but my submission was failing because my code assumes that all seconds are zeroes which is not the case for everyone else's codes/solutions
", "votes": 3, "replies": [] }, { "id": 1640887, "author_name": "jagofc", "author_url": "", "post_date": "2022-01-06T23:27:29.953000", "content": "Thanks for explaining in detail Shady, I had also been expecting the timestamps to be divisible by 60 and I am quite surprised by your finding. Here is something little to add to it.
\nThis code passes, so all timestamps in each df_data iteration are the same:
\nfor (df_data, df_pred) in iter_test:\n if len(df_data.timestamp.unique()) == 1:\n df_pred.loc[:,'Target'] = 0.01 * np.random.randn(len(df_pred), 1)\n else:\n raise RuntimeError()\n env.predict(df_pred)\nA further experiment:
\nThe following strategy is successful with LB = 0.9999.
\ndef predict_from_train(df_data, df_pred):\n ts = df_data.iat[0,0]\n df = pd.merge(df_data[['Asset_ID', 'row_id']],\n train_df[train_df.timestamp==ts][['Asset_ID', 'Target']],\n on='Asset_ID')\n df = df.drop(columns=['Asset_ID'])\n df_pred = (pd.merge(df_pred[['row_id']], df,\n on='row_id', how='left')\n .fillna(0.))\n return df_pred\nenv = gresearch_crypto.make_env()\niter_test = env.iter_test()\ntrain_timestamps = train_df.timestamp.unique()\nstrategy = 0\n\nfor (df_data, df_pred) in iter_test:\n\n if strategy == 0:\n ts = df_data.iat[0,0]\n if ts in train_timestamps:\n df_pred = predict_from_train(df_data, df_pred)\n else:\n df_pred.loc[:,'Target'] = 0\n strategy = 1\n else:\n df_pred.loc[:,'Target'] = 0\n\n env.predict(df_pred)\nTogether with the result of the unique timestamp code, this means any non-train set timestamps must be very near the end. Otherwise the strategy would switch from 0 to 1 earlier and the zero predictions would bring the LB score lower than 0.9999. Together with @shadymokh 's (mod 60) code this means that any non-zero (mod 60) timestamp must also be very near the end.
", "votes": 3, "replies": [] }, { "id": 1642297, "author_name": "jagofc", "author_url": "", "post_date": "2022-01-08T08:44:00.870000", "content": "Just in case anyone is still reading :D
\ntrain_timestamps = train_df.timestamp.unique()\n\nfor (df_data, df_pred) in iter_test:\n\n ts = df_data.iat[0,0]\n if ts > train_timestamps.max() + T:\n raise RuntimeError()\n\n env.predict(df_pred)\nWith T = 0 this fails. With T = 3600 it passes.
This means that there are a limited number of additional timestamps delivered by the API for the public LB beyond the end of the train.csv.
\nAt this point I think there are diminishing returns to pursuing further: for all we know the additional timestamps might just be contrived test cases to stop us writing code that relies on particular assumptions (like @shadymokh 's != 0 % 60 finding); or to frustrate us by not being able to perfectly reproduce the score of the API.
", "votes": 3, "replies": [] } ] } ], "raw_markdown_by_id": { "1628269": "After too many submissions to debug why my notebooks are receiving submission scoring error, these are my findings:\n\n- Submission scoring error could simply be a notebook exception, not necessary issues with the submission files which is inconsistent with [Kaggle code debugging docs](https://www.kaggle.com/code-competition-debugging) (not sure if this is known to everyone, I am new here)\n- There are time gaps in the hidden test set.\n- Timestamps in the hidden set are are not always 0 seconds unlike the training data, this one gave me the hardest time in the competition till I discovered it.\n\nAll the previous statements are based on actual submissions\nexample:\n```\nfor (test_df, sample_prediction_df) in iter_test:\n if (test_df.timestamp.values%60).max()>0:\n raise\n env.predict(sample_prediction_df)\n\n```\n\n- This one is based on someone's else comment in the forum, I didn't test it, they said their submission fails if they use test_df.timestamp[0] but it works when they use test_df.timestamp.iat[0], which indicates that test_df index does not necessary starts by 0\n\n- This one I didn't test but just reminding you that the train data contains few inf values in VWAP\n\n-As for notebook timeouts error, I tested dummy submissions with time.sleep() \n```\nfor (test_df, sample_prediction_df) in iter_test:\n time.sleep(0.25) \n env.predict(sample_prediction_df)\n```\n^ above timed out\n\n```\ntime.sleep(300) \nfor (test_df, sample_prediction_df) in iter_test:\n\n time.sleep(0.23) \n env.predict(sample_prediction_df)\n```\n^ timed out\n```\ntime.sleep(300) \nfor (test_df, sample_prediction_df) in iter_test:\n time.sleep(0.22) \n env.predict(sample_prediction_df)\n```\n^ this passed\n\nThe hidden test set is expected to be 3 months of data (129600 iterations) and notebook runtime limit is 9 hours (32400 seconds) which means you shouldn't exceed 0.25 seconds per iteration (this is rough estimates)\n\nAccording to [competition timeline](https://www.kaggle.com/c/g-research-crypto-forecasting/overview/timeline) \n\"Updates will take place roughly every two weeks\" \nif this means that our submitted notebook will be ran on new data every two weeks, each run has 9 hours runtime, then I expect the hidden test set to be two weeks of data rather than 3 months, but unfortunately this is not the case\n", "1642198": "Adding one more reason causing submission scoring error which is that there are definitely missing cryptos in the test set i.e. it is not always delivering 14 cryptos so your submission needs to account for this. If not, there likely be an exception that causes the scoring error. Hope this helps those reading this excellent thread!", "1637473": "Hi @shadymokh - interesting that you've drawn attention to the *\"updates will take place roughly every two weeks\"* statement - I'd been assuming that we should plan for the 3 months of data to be dealt within the 9 hour runtime, so was a bit worried I'd missed something.\n\nTwo possiblilities that cohere with the 9 hours of runtime, 3 months of data, and fortnightly updates:\n1. We only need to plan for 2 weeks worth of test data (~ 2 * 7 * 24 * 60 = 20160) at each fortnightly update, for which we have 9 hours runtime. \n2. We really do need to plan for 3 months worth of test data for the 9hrs runtime. At each update, all of the test data accumulated up until that point is used in the test set. After 2 weeks, 2 weeks worth of test data is used; after 4 weeks, 4 weeks worth, etc... \n\nMy guess is option 2. Trying to implement 1. would be a lot of unnecessary hassle.\n\nA fun twist with option 2 is that anyone who is *just* over the runtime constraint for the full 3 months of data will only find out right at the last update. ๐ฆ.", "1628733": "Hi, great findings. Here is my estimation:\n```\n1. 300 + 0.22x < 32400\n2. 300 + 0.23x > 32400\n```\nthe number of loops x in the hidden test set: 140739.13 < x < 145909.09\nEach loop represent one minute information. So the time length y of hidden test set is 97.74 days < y < 101.36 days." } }