Hi. For my model I need to do some more or less complicated calculations during the iteration steps of the environment. Using the test version during a save and commit is all fine, while I constantly got \"submission scoring errors\" when submitting to competition. There was no obvious reason at all and I debugged for a couple of days (since having only 5 attempts per day). I eventually arrived at this, which irritates me quite a bit:
\nThis code runs perfectly fine, creating a public score of -1 as expected:
\nimport gresearch_crypto\n\nenv = gresearch_crypto.make_env()\niter_test = env.iter_test()\n\n\nfor i, (df_test, df_pred) in enumerate(iter_test):\n if i == 0:\n display(df_test)\n display(df_pred)\n\n df_pred[\"Target\"] == 0\n env.predict(df_pred)\nHowever, the following code runs fine in save commit and all but consistently creates \"Submission scoring Errors\" when submitting to competition:
\nimport gresearch_crypto\n\nenv = gresearch_crypto.make_env()\niter_test = env.iter_test()\n\n\nfor i, (df_test, df_pred) in enumerate(iter_test):\n if i == 0:\n display(df_test)\n display(df_pred)\n\n ### adding this line creates the \"submission scoring error\"\n ts = df_test[\"timestamp\"][0]\n ###\n\n df_pred[\"Target\"] == 0\n env.predict(df_pred)\nIf I wrap the line ts = df_test[\"timestamp\"][0] inside a try … except … block it again runs fine yielding a public score of -1. So it looks to me like the statement ts = df_test[\"timestamp\"][0] causes and error in the hidden test environment. I will try to work around this now, but I thought maybe someone knows what's going on here. If this is the case, the error label \"Submission Scoring Error\" is a rather misleading one here.
Thanks!
", "messages": [ { "id": "1601547", "postDate": "12/01/2021 11:54:59", "content": "Hi. For my model I need to do some more or less complicated calculations during the iteration steps of the environment. Using the test version during a save and commit is all fine, while I constantly got \"submission scoring errors\" when submitting to competition. There was no obvious reason at all and I debugged for a couple of days (since having only 5 attempts per day). I eventually arrived at this, which irritates me quite a bit:
\nThis code runs perfectly fine, creating a public score of -1 as expected:
\nimport gresearch_crypto\n\nenv = gresearch_crypto.make_env()\niter_test = env.iter_test()\n\n\nfor i, (df_test, df_pred) in enumerate(iter_test):\n if i == 0:\n display(df_test)\n display(df_pred)\n\n df_pred[\"Target\"] == 0\n env.predict(df_pred)\nHowever, the following code runs fine in save commit and all but consistently creates \"Submission scoring Errors\" when submitting to competition:
\nimport gresearch_crypto\n\nenv = gresearch_crypto.make_env()\niter_test = env.iter_test()\n\n\nfor i, (df_test, df_pred) in enumerate(iter_test):\n if i == 0:\n display(df_test)\n display(df_pred)\n\n ### adding this line creates the \"submission scoring error\"\n ts = df_test[\"timestamp\"][0]\n ###\n\n df_pred[\"Target\"] == 0\n env.predict(df_pred)\nIf I wrap the line ts = df_test[\"timestamp\"][0] inside a try … except … block it again runs fine yielding a public score of -1. So it looks to me like the statement ts = df_test[\"timestamp\"][0] causes and error in the hidden test environment. I will try to work around this now, but I thought maybe someone knows what's going on here. If this is the case, the error label \"Submission Scoring Error\" is a rather misleading one here.
Thanks!
", "rawMarkdown": "Hi. For my model I need to do some more or less complicated calculations during the iteration steps of the environment. Using the test version during a save and commit is all fine, while I constantly got \"submission scoring errors\" when submitting to competition. There was no obvious reason at all and I debugged for a couple of days (since having only 5 attempts per day). I eventually arrived at this, which irritates me quite a bit:\n\nThis code runs perfectly fine, creating a public score of -1 as expected:\n\n```\nimport gresearch_crypto\n\nenv = gresearch_crypto.make_env()\niter_test = env.iter_test()\n \n \nfor i, (df_test, df_pred) in enumerate(iter_test):\n if i == 0:\n display(df_test)\n display(df_pred)\n \n df_pred[\"Target\"] == 0\n env.predict(df_pred)\n```\n\nHowever, the following code runs fine in save commit and all but consistently creates \"Submission scoring Errors\" when submitting to competition:\n\n```\nimport gresearch_crypto\n\nenv = gresearch_crypto.make_env()\niter_test = env.iter_test()\n \n \nfor i, (df_test, df_pred) in enumerate(iter_test):\n if i == 0:\n display(df_test)\n display(df_pred)\n \n ### adding this line creates the \"submission scoring error\"\n ts = df_test[\"timestamp\"][0]\n ###\n \n df_pred[\"Target\"] == 0\n env.predict(df_pred)\n```\n\nIf I wrap the line `ts = df_test[\"timestamp\"][0]` inside a try ... except ... block it again runs fine yielding a public score of -1. So it looks to me like the statement `ts = df_test[\"timestamp\"][0]` causes and error in the hidden test environment. I will try to work around this now, but I thought maybe someone knows what's going on here. If this is the case, the error label \"Submission Scoring Error\" is a rather misleading one here.\n\nThanks!", "votes": null }, { "id": "1601574", "postDate": "12/01/2021 12:50:00", "content": "Your code seems to assume that the index of the first entry of the ’timestamp’ series is 0. This is probably not the case for the submission. (Since they provide the data in chunks, it might be that the first chunk has indices [0, 1, …, n - 1] and the second [n, n + 1, …] and so on.)
\nYou could try to replace df_test[”timestamp”][0] with df_test[”timestamp”].iloc[0] and see if that makes any difference.
Thank you so much! Indeed your suggestion works. 💯
\nIn the end it's me to blame for not being aware of the indexing implications. But I do think that using the same conventions for the data batches in the hidden testing env than in the dummy environment (indexes starting at 0 at every batch there) would have been be a good idea by the organizers :)
I got a similar issue , I removed every print or display from the submission code area and everything worked properly, just try it
", "rawMarkdown": "I got a similar issue , I removed every print or display from the submission code area and everything worked properly, just try it", "votes": null }, { "id": "1601721", "postDate": "12/01/2021 14:43:11", "content": "Thanks. I tried that, was nothing like that in my case. First answer here solved it for me.
", "rawMarkdown": "Thanks. I tried that, was nothing like that in my case. First answer here solved it for me.", "votes": null } ], "comments": [ { "id": 1601574, "author_name": "christoffer", "author_url": "", "post_date": "12/01/2021 12:50:00", "content": "Your code seems to assume that the index of the first entry of the ’timestamp’ series is 0. This is probably not the case for the submission. (Since they provide the data in chunks, it might be that the first chunk has indices [0, 1, …, n - 1] and the second [n, n + 1, …] and so on.)
\nYou could try to replace df_test[”timestamp”][0] with df_test[”timestamp”].iloc[0] and see if that makes any difference.
Thank you so much! Indeed your suggestion works. 💯
\nIn the end it's me to blame for not being aware of the indexing implications. But I do think that using the same conventions for the data batches in the hidden testing env than in the dummy environment (indexes starting at 0 at every batch there) would have been be a good idea by the organizers :)
I got a similar issue , I removed every print or display from the submission code area and everything worked properly, just try it
", "votes": null, "replies": [ { "id": 1601721, "author_name": "wspinkaggle", "author_url": "", "post_date": "12/01/2021 14:43:11", "content": "Thanks. I tried that, was nothing like that in my case. First answer here solved it for me.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1601547": "Hi. For my model I need to do some more or less complicated calculations during the iteration steps of the environment. Using the test version during a save and commit is all fine, while I constantly got \"submission scoring errors\" when submitting to competition. There was no obvious reason at all and I debugged for a couple of days (since having only 5 attempts per day). I eventually arrived at this, which irritates me quite a bit:\n\nThis code runs perfectly fine, creating a public score of -1 as expected:\n\n```\nimport gresearch_crypto\n\nenv = gresearch_crypto.make_env()\niter_test = env.iter_test()\n \n \nfor i, (df_test, df_pred) in enumerate(iter_test):\n if i == 0:\n display(df_test)\n display(df_pred)\n \n df_pred[\"Target\"] == 0\n env.predict(df_pred)\n```\n\nHowever, the following code runs fine in save commit and all but consistently creates \"Submission scoring Errors\" when submitting to competition:\n\n```\nimport gresearch_crypto\n\nenv = gresearch_crypto.make_env()\niter_test = env.iter_test()\n \n \nfor i, (df_test, df_pred) in enumerate(iter_test):\n if i == 0:\n display(df_test)\n display(df_pred)\n \n ### adding this line creates the \"submission scoring error\"\n ts = df_test[\"timestamp\"][0]\n ###\n \n df_pred[\"Target\"] == 0\n env.predict(df_pred)\n```\n\nIf I wrap the line `ts = df_test[\"timestamp\"][0]` inside a try ... except ... block it again runs fine yielding a public score of -1. So it looks to me like the statement `ts = df_test[\"timestamp\"][0]` causes and error in the hidden test environment. I will try to work around this now, but I thought maybe someone knows what's going on here. If this is the case, the error label \"Submission Scoring Error\" is a rather misleading one here.\n\nThanks!", "1601574": "Your code seems to assume that the index of the first entry of the ’timestamp’ series is 0. This is probably not the case for the submission. (Since they provide the data in chunks, it might be that the first chunk has indices [0, 1, …, n - 1] and the second [n, n + 1, …] and so on.)\n\nYou could try to replace `df_test[”timestamp”][0]` with `df_test[”timestamp”].iloc[0]` and see if that makes any difference.", "1601668": "Thank you so much! Indeed your suggestion works. 💯\nIn the end it's me to blame for not being aware of the indexing implications. But I do think that using the same conventions for the data batches in the hidden testing env than in the dummy environment (indexes starting at 0 at every batch there) would have been be a good idea by the organizers :)", "1601709": "I got a similar issue , I removed every print or display from the submission code area and everything worked properly, just try it", "1601721": "Thanks. I tried that, was nothing like that in my case. First answer here solved it for me." }, "source": "meta" }