Does anyone have tips for diagnosing what may be going wrong when making an official submission but receiving an error when it is scored?
\nThe notebook and iteration loop runs fine when running manually on Kaggle, but when making an official \"submission\" it takes much longer than expected (~30-60 minutes) before resulting in an error. The logs don't show any obvious errors, and instead say that the notebook only takes less than 10 minutes to finish executing. The last log message is \"Writing 374820 bytes to results.html\" which I assume is the results from the .predict() method during the iteration loop?
\nPotentially related to this, and something that would be good to clarify, is the importance of the \"group_num\" column that is in the \"example_sample_submission.csv\" file provided with the competition. The API walkthrough provided by the competition hosts do not make reference to this column nor does it appear in the sample_prediction_df object given by the test iterator. Is adding, or not adding, this variable to a set of predictions potentially causing an error? Unfortunately both including and excluding this variable result in a submission error for my notebook.
\nFinally, even though the example submission csv is provided, we don't actually write out any results to a csv file for scoring, correct? We only store the predictions in the iteration loop using the .predict() method and it is supposed to work from there?
\nThanks in advanced for any help that could be provided.
", "messages": [ { "id": "1660860", "postDate": "01/23/2022 03:10:45", "content": "Does anyone have tips for diagnosing what may be going wrong when making an official submission but receiving an error when it is scored?
\nThe notebook and iteration loop runs fine when running manually on Kaggle, but when making an official \"submission\" it takes much longer than expected (~30-60 minutes) before resulting in an error. The logs don't show any obvious errors, and instead say that the notebook only takes less than 10 minutes to finish executing. The last log message is \"Writing 374820 bytes to results.html\" which I assume is the results from the .predict() method during the iteration loop?
\nPotentially related to this, and something that would be good to clarify, is the importance of the \"group_num\" column that is in the \"example_sample_submission.csv\" file provided with the competition. The API walkthrough provided by the competition hosts do not make reference to this column nor does it appear in the sample_prediction_df object given by the test iterator. Is adding, or not adding, this variable to a set of predictions potentially causing an error? Unfortunately both including and excluding this variable result in a submission error for my notebook.
\nFinally, even though the example submission csv is provided, we don't actually write out any results to a csv file for scoring, correct? We only store the predictions in the iteration loop using the .predict() method and it is supposed to work from there?
\nThanks in advanced for any help that could be provided.
", "rawMarkdown": "Does anyone have tips for diagnosing what may be going wrong when making an official submission but receiving an error when it is scored? \n\nThe notebook and iteration loop runs fine when running manually on Kaggle, but when making an official \"submission\" it takes much longer than expected (~30-60 minutes) before resulting in an error. The logs don't show any obvious errors, and instead say that the notebook only takes less than 10 minutes to finish executing. The last log message is \"Writing 374820 bytes to __results__.html\" which I assume is the results from the .predict() method during the iteration loop? \n\nPotentially related to this, and something that would be good to clarify, is the importance of the \"group_num\" column that is in the \"example_sample_submission.csv\" file provided with the competition. The API [walkthrough](https://www.kaggle.com/sohier/detailed-api-introduction) provided by the competition hosts do not make reference to this column nor does it appear in the sample_prediction_df object given by the test iterator. Is adding, or not adding, this variable to a set of predictions potentially causing an error? Unfortunately both including and excluding this variable result in a submission error for my notebook.\n\nFinally, even though the example submission csv is provided, we don't actually write out any results to a csv file for scoring, correct? We only store the predictions in the iteration loop using the .predict() method and it is supposed to work from there?\n\nThanks in advanced for any help that could be provided.", "votes": null }, { "id": "1661789", "postDate": "01/23/2022 17:41:12", "content": "I had a similar problem with the submission bombing out after 30mins or so, I realized that not all of the group numbers have all coins, I solved this by merging onto the original test file received for each group. I wouldn't worry to much about group num, think that this follows the iteration number. The CSV is written automatically. Good luck.
", "rawMarkdown": "I had a similar problem with the submission bombing out after 30mins or so, I realized that not all of the group numbers have all coins, I solved this by merging onto the original test file received for each group. I wouldn't worry to much about group num, think that this follows the iteration number. The CSV is written automatically. Good luck.", "votes": null }, { "id": "1663660", "postDate": "01/25/2022 10:09:08", "content": "Hi, I had a submission scoring err since few days ago, I've tried a lot, still the same problem. I would like to ask if you have any idea what might caused that, or if you would like to have a look on my notebook to give me some indications?
", "rawMarkdown": "Hi, I had a submission scoring err since few days ago, I've tried a lot, still the same problem. I would like to ask if you have any idea what might caused that, or if you would like to have a look on my notebook to give me some indications?", "votes": null }, { "id": "1664750", "postDate": "01/26/2022 08:48:03", "content": "Happy to have a look to see if I can help; however I would check how long your API is taking using your test submission when committing using:
\nfrom datetime import datetime
\nstart_time = datetime.now()
Submission Process
\ntime_elapsed = datetime.now() - start_time
\nprint('Time elapsed (hh:mm:ss.ms) {}'.format(time_elapsed))
\nand make sure that the four test iterations are less than about 0.8 of a second otherwise you are going to time out.
", "rawMarkdown": "Happy to have a look to see if I can help; however I would check how long your API is taking using your test submission when committing using:\n\nfrom datetime import datetime\nstart_time = datetime.now()\n\nSubmission Process\n\ntime_elapsed = datetime.now() - start_time\n\nprint('Time elapsed (hh:mm:ss.ms) {}'.format(time_elapsed))\n\nand make sure that the four test iterations are less than about 0.8 of a second otherwise you are going to time out.", "votes": null }, { "id": "1665413", "postDate": "01/26/2022 20:06:56", "content": "Hello, thank you, that's very kind of you. I will send you a message within link of notebook and result of run.
", "rawMarkdown": "Hello, thank you, that's very kind of you. I will send you a message within link of notebook and result of run.", "votes": null }, { "id": "1666519", "postDate": "01/27/2022 20:19:36", "content": "Hey Zhao, I had a look at your code, thereI can say the following:
\nGood luck and cut that run time down!
", "rawMarkdown": "Hey Zhao, I had a look at your code, thereI can say the following:\n- Your submission routine is taking far too long, it needs to be less than 0.2s per iteration with yours almost at 8s per iteration. I strongly suggest that you convert to numpy arrays for the submission part. I would start by converting this history = history.iloc[-(max_lookback*14+100):] to an array.\n- It looks like the iterative process on df_test.iterrows ensures that the correct row id is always returned to the env API so I dont think that this is an issues as I suggested above. \n\nGood luck and cut that run time down!", "votes": null }, { "id": "1666919", "postDate": "01/28/2022 07:07:10", "content": "Hey Mr JonnydosSantos, Thank you.
\nI also realised that it took too long for each iteration, I will start from your suggestion :)
I had a similar problem with the submission bombing out after 30mins or so, I realized that not all of the group numbers have all coins, I solved this by merging onto the original test file received for each group. I wouldn't worry to much about group num, think that this follows the iteration number. The CSV is written automatically. Good luck.
", "votes": null, "replies": [ { "id": 1663660, "author_name": "zhaoyuanyuanzyy", "author_url": "", "post_date": "01/25/2022 10:09:08", "content": "Hi, I had a submission scoring err since few days ago, I've tried a lot, still the same problem. I would like to ask if you have any idea what might caused that, or if you would like to have a look on my notebook to give me some indications?
", "votes": null, "replies": [] }, { "id": 1664750, "author_name": "jonnydossantos", "author_url": "", "post_date": "01/26/2022 08:48:03", "content": "Happy to have a look to see if I can help; however I would check how long your API is taking using your test submission when committing using:
\nfrom datetime import datetime
\nstart_time = datetime.now()
Submission Process
\ntime_elapsed = datetime.now() - start_time
\nprint('Time elapsed (hh:mm:ss.ms) {}'.format(time_elapsed))
\nand make sure that the four test iterations are less than about 0.8 of a second otherwise you are going to time out.
", "votes": null, "replies": [] }, { "id": 1665413, "author_name": "zhaoyuanyuanzyy", "author_url": "", "post_date": "01/26/2022 20:06:56", "content": "Hello, thank you, that's very kind of you. I will send you a message within link of notebook and result of run.
", "votes": null, "replies": [] }, { "id": 1666519, "author_name": "jonnydossantos", "author_url": "", "post_date": "01/27/2022 20:19:36", "content": "Hey Zhao, I had a look at your code, thereI can say the following:
\nGood luck and cut that run time down!
", "votes": null, "replies": [] }, { "id": 1666919, "author_name": "zhaoyuanyuanzyy", "author_url": "", "post_date": "01/28/2022 07:07:10", "content": "Hey Mr JonnydosSantos, Thank you.
\nI also realised that it took too long for each iteration, I will start from your suggestion :)