Hello all. A novice such as myself is wondering, how long does a scoring take on kaggle after a submission? <= 30 mins or > 30 mins? Should I be worried if it is taking too long to score? Is it anything to do with the user or not? Thanks.
", "messages": [ { "id": 2444564, "postDate": "2023-09-18T10:40:11.207Z", "content": "Consider saving your model and reload it for submission rather than training again. One of the major bottleneck I observed is conversion of timestamp column to timestamp object.
", "rawMarkdown": "Consider saving your model and reload it for submission rather than training again. One of the major bottleneck I observed is conversion of timestamp column to timestamp object.", "votes": 7, "replies": [ { "id": 2444864, "postDate": "2023-09-18T14:18:33.913Z", "content": "Agree on this. For a comparison, converting timestamps of 3 series IDs with least steps take ~10s, and ~30 series take ~100s in my case.
", "rawMarkdown": "Agree on this. For a comparison, converting timestamps of 3 series IDs with least steps take ~10s, and ~30 series take ~100s in my case.", "votes": 3 }, { "id": 2445712, "postDate": "2023-09-19T03:35:06.293Z", "content": "Oops. Reusing the model would have saved my day.
\nI'll try that next time.
\nThanks @anilreddyvv !
Been trying that. Still failed even with a reloaded trained model. Don't know what's going on anymore to be honest.
", "rawMarkdown": "Been trying that. Still failed even with a reloaded trained model. Don't know what's going on anymore to be honest. ", "replies": [ { "id": 2447571, "postDate": "2023-09-20T06:47:30.070Z", "content": "Try to predict on the entire train data of competition (including feature extraction) using your saved model and observe how much time is taking, do this experimentation in kaggle notebook only.
", "rawMarkdown": "Try to predict on the entire train data of competition (including feature extraction) using your saved model and observe how much time is taking, do this experimentation in kaggle notebook only.", "votes": 2 }, { "id": 2447584, "postDate": "2023-09-20T06:51:56.647Z", "content": "On the kaggle notebook it doesn't take long since it is pretrained. Less than 3 or 5 mins (I think). The problem comes when submitting for scoring.
", "rawMarkdown": "On the kaggle notebook it doesn't take long since it is pretrained. Less than 3 or 5 mins (I think). The problem comes when submitting for scoring." }, { "id": 2447593, "postDate": "2023-09-20T06:55:15.770Z", "content": "Whoops. Sorry. Do you mean I should use the entire training set as a test set to gauge the runtime? Am I getting you right?
", "rawMarkdown": "Whoops. Sorry. Do you mean I should use the entire training set as a test set to gauge the runtime? Am I getting you right?" }, { "id": 2447601, "postDate": "2023-09-20T06:56:54.873Z", "rawMarkdown": "", "isDeleted": true }, { "id": 2447607, "postDate": "2023-09-20T07:00:06.387Z", "content": "Yes, because using saved model or even training models like lightgbm, my submission is getting scored in less than 2 hrs. If you have complex model or more number of features it might take comparatively more time.
", "rawMarkdown": "Yes, because using saved model or even training models like lightgbm, my submission is getting scored in less than 2 hrs. If you have complex model or more number of features it might take comparatively more time." }, { "id": 2447631, "postDate": "2023-09-20T07:10:16.177Z", "content": "This is the time I got with the pretrained model when predicting on the entire training dataset on my end:
\nAfter loading pretrained model
\nCPU times: user 3 µs, sys: 0 ns, total: 3 µs
\nWall time: 5.25 µs
After prediction using pretrained model
\nCPU times: user 2 µs, sys: 0 ns, total: 2 µs
\nWall time: 5.96 µs
Seems a bit strange. Or is this correct?
", "rawMarkdown": "This is the time I got with the pretrained model when predicting on the entire training dataset on my end:\n\nAfter loading pretrained model\nCPU times: user 3 µs, sys: 0 ns, total: 3 µs\nWall time: 5.25 µs\n\nAfter prediction using pretrained model\nCPU times: user 2 µs, sys: 0 ns, total: 2 µs\nWall time: 5.96 µs\n\nSeems a bit strange. Or is this correct?" }, { "id": 2447634, "postDate": "2023-09-20T07:11:42.607Z", "content": "Meanwhile, still awaiting scoring @23 mins and counting…
", "rawMarkdown": "Meanwhile, still awaiting scoring @23 mins and counting..." }, { "id": 2447646, "postDate": "2023-09-20T07:16:40.953Z", "content": "I understand that but why wait till after 9 hours and then throwing an exception. Lost 4 submissions just like that.
\nEven on the GPU starter notebook in another competition ran out of memory just rerunning it without doing anything. Is it from my end or what? Other people run the same or similar code and it gets scored.
", "rawMarkdown": "I understand that but why wait till after 9 hours and then throwing an exception. Lost 4 submissions just like that.\n\nEven on the GPU starter notebook in another competition ran out of memory just rerunning it without doing anything. Is it from my end or what? Other people run the same or similar code and it gets scored." }, { "id": 2447651, "postDate": "2023-09-20T07:21:03.327Z", "content": "Out of memory or timeout messages I can understand but this exception throwing messages are ambiguous because the code runs fine offline and online even with larger offline datasets run elsewhere even if it takes hours.
", "rawMarkdown": "Out of memory or timeout messages I can understand but this exception throwing messages are ambiguous because the code runs fine offline and online even with larger offline datasets run elsewhere even if it takes hours." }, { "id": 2447654, "postDate": "2023-09-20T07:22:59.060Z", "content": "did you measure you feature engineering time as well?
", "rawMarkdown": "did you measure you feature engineering time as well?" }, { "id": 2447656, "postDate": "2023-09-20T07:23:19.930Z", "content": "Another weird thing is that the same code submitted twice or more, the newer submitted one fails or goes through while the earlier submissions keep on until a very long time and then fail as well.
", "rawMarkdown": "Another weird thing is that the same code submitted twice or more, the newer submitted one fails or goes through while the earlier submissions keep on until a very long time and then fail as well." }, { "id": 2447662, "postDate": "2023-09-20T07:29:43.283Z", "content": "The problem when using GPU is you get less RAM, and if you are doing multiple pandas operation like timestamp conversion on entire test set, it is going to run out of memory. You should probably consider loading maybe one series like
\ntest = pd.read_parquet(TEST_FILE, filters=[('series_id', '=', series)]).
\nand using
\ndel test
\ngc.collect() and load next batch
\nYou can refer here
Is it scoring error or any other error?
Here are the times:
\nCPU times: user 4 µs, sys: 0 ns, total: 4 µs
\nWall time: 6.68 µs
\nCPU times: user 4 µs, sys: 1 µs, total: 5 µs
\nWall time: 6.68 µs
\nCPU times: user 3 µs, sys: 0 ns, total: 3 µs
\nWall time: 6.2 µs
\nCPU times: user 3 µs, sys: 0 ns, total: 3 µs
\nWall time: 6.2 µs
\nCPU times: user 5 µs, sys: 1 µs, total: 6 µs
\nWall time: 7.39 µs
\nCPU times: user 3 µs, sys: 0 ns, total: 3 µs
\nWall time: 6.2 µs
I think you are doing something wrong because all the logs shows timing as microseconds
", "rawMarkdown": "I think you are doing something wrong because all the logs shows timing as microseconds", "votes": 1 }, { "id": 2447670, "postDate": "2023-09-20T07:36:16.630Z", "content": "thrown exception is the message I keep getting after long periods of scoring. On the Active events tab, it always says notebook \"successful\" but competition scoring after hours, says \"failed\" and opening it shows thrown exception errors.
", "rawMarkdown": "thrown exception is the message I keep getting after long periods of scoring. On the Active events tab, it always says notebook \"successful\" but competition scoring after hours, says \"failed\" and opening it shows thrown exception errors." }, { "id": 2447676, "postDate": "2023-09-20T07:39:07.937Z", "content": "I am using the edited lgbm notebook of renatoreggian
\nTrained and saved model and relaunched model to save time in prediction. Another weird thing is that my earlier saved gradient boosting model even when uploaded unto kaggle folders is no where to be found and when I upload it again it says that a file with the same name exists. Even when trying to upload to a new created folder, it still prevents me from uploading the file again.
", "rawMarkdown": "Trained and saved model and relaunched model to save time in prediction. Another weird thing is that my earlier saved gradient boosting model even when uploaded unto kaggle folders is no where to be found and when I upload it again it says that a file with the same name exists. Even when trying to upload to a new created folder, it still prevents me from uploading the file again. " }, { "id": 2447684, "postDate": "2023-09-20T07:45:53.120Z", "content": "Hi, I have been using this your exact solution and it runs okay on my end. Only when its time for scoring is where there is problem.
", "rawMarkdown": "Hi, I have been using this your exact solution and it runs okay on my end. Only when its time for scoring is where there is problem." }, { "id": 2447691, "postDate": "2023-09-20T07:53:23.587Z", "content": "I upload it again it says that a file with the same name exists, You can create a fresh dataset if you have issues uploading to existing dataset. The notebook you posted looks like ran without any issues, debug the changes you made on top of that notebook.
Thank you so much @anilreddyvv for this idea about save and reload model. Best regards!!
", "rawMarkdown": "Thank you so much @anilreddyvv for this idea about save and reload model. Best regards!!" } ] }, { "id": 2444181, "postDate": "2023-09-18T07:48:20.963Z", "content": "Hi @aidlre001
\nIt really depends on your workflow.
\nHow many percent of the datasets are you using on training? The lightweight dataset from dr @carlmcbrideellis could be the fastest to work on since it only contains ~30 series from total ~200 from the whole training set.
\nWhat algorithm did you use? Based on the public notebook, XGBoost or sklearn GradientBoostingClassifier could take hours, while my work using LGBM is done in under 1h.
\nHow many features did you use? The more features, the longer time your solution will run.
\nOther aspects like the accelerators or the libraries you used to load data might impact your running time as well.
", "rawMarkdown": "Hi @aidlre001 \n\nIt really depends on your workflow.\n\nHow many percent of the datasets are you using on training? The lightweight dataset from dr @carlmcbrideellis could be the fastest to work on since it only contains ~30 series from total ~200 from the whole training set.\n\nWhat algorithm did you use? Based on the public notebook, XGBoost or sklearn GradientBoostingClassifier could take hours, while my work using LGBM is done in under 1h.\n\nHow many features did you use? The more features, the longer time your solution will run.\n\nOther aspects like the accelerators or the libraries you used to load data might impact your running time as well.", "votes": 4, "replies": [ { "id": 2444259, "postDate": "2023-09-18T08:24:38.323Z", "content": "Hi Rimba,
\nI used the entire dataset but processed in parquet file batches and using the earlier three features from Carl McBride Ellis's notebook. The weird thing is that it runs fast in my notebook session and when I submitted it says successful but the scoring stage is what takes a long time. It basically stays at
\n\"Scoring… \"
Yes probably it is because of the size of the hidden test set. The hidden test set contains roughly similar number of series ID as in the training set (~200), so I think if you want to project the scoring time, you can try using the full ~200 series IDs in the training set first.
", "rawMarkdown": "Yes probably it is because of the size of the hidden test set. The hidden test set contains roughly similar number of series ID as in the training set (~200), so I think if you want to project the scoring time, you can try using the full ~200 series IDs in the training set first." }, { "id": 2444305, "postDate": "2023-09-18T08:44:15.950Z", "content": "But the thing is that my earlier submissions using the lightweight dataset for training using the same classifier was relatively faster in the submission scoring stages. So if they are testing my submission with the same hidden dataset of same size, why the runtime disparity?
", "rawMarkdown": "But the thing is that my earlier submissions using the lightweight dataset for training using the same classifier was relatively faster in the submission scoring stages. So if they are testing my submission with the same hidden dataset of same size, why the runtime disparity?" }, { "id": 2444319, "postDate": "2023-09-18T08:47:16.290Z", "content": "Using the entire training set without batch processing crashes both online and offline, thus the reason for breaking it down to parquet file batches. Oh well, will try with the multi feature training set and see. Thanks for all your help and taking the time. I hope I can get something that would be useful for others here too.
", "rawMarkdown": "Using the entire training set without batch processing crashes both online and offline, thus the reason for breaking it down to parquet file batches. Oh well, will try with the multi feature training set and see. Thanks for all your help and taking the time. I hope I can get something that would be useful for others here too." }, { "id": 2447905, "postDate": "2023-09-20T09:58:38.327Z", "content": "I think because the lightweight datasets contains much less than 200 series, and the hidden one is 200-ish, hence the disparity. You can try your own suggestion as well. Good luck!
", "rawMarkdown": "I think because the lightweight datasets contains much less than 200 series, and the hidden one is 200-ish, hence the disparity. You can try your own suggestion as well. Good luck!", "votes": 1 } ] } ] }, { "id": 2443566, "postDate": "2023-09-17T19:12:10.950Z", "content": "Hello all. A novice such as myself is wondering, how long does a scoring take on kaggle after a submission? <= 30 mins or > 30 mins? Should I be worried if it is taking too long to score? Is it anything to do with the user or not? Thanks.
", "rawMarkdown": "Hello all. A novice such as myself is wondering, how long does a scoring take on kaggle after a submission? <= 30 mins or > 30 mins? Should I be worried if it is taking too long to score? Is it anything to do with the user or not? Thanks.", "votes": 1 }, { "id": 2447733, "postDate": "2023-09-20T08:16:28.043Z", "content": "@Anil Reddy and @Rimba Erlangga,
\nThank you for your helpful responses on here. It seems my complaining worked. The code has finally been scored for the first time. However, its saying it is not an improvement over my previous scores (which never scored). I wish I did more so I could have something to share but the time wasted on this scoring issue has me drained so I will go back offline to continue working and hopefully have something good to share. Thanks again to all involved commenters.
", "rawMarkdown": "@Anil Reddy and @Rimba Erlangga,\n\nThank you for your helpful responses on here. It seems my complaining worked. The code has finally been scored for the first time. However, its saying it is not an improvement over my previous scores (which never scored). I wish I did more so I could have something to share but the time wasted on this scoring issue has me drained so I will go back offline to continue working and hopefully have something good to share. Thanks again to all involved commenters.", "votes": 2, "replies": [ { "id": 2447743, "postDate": "2023-09-20T08:24:35.360Z", "content": "Good luck @aidlre001
", "rawMarkdown": "Good luck @aidlre001 ", "votes": 1, "replies": [ { "id": 2448994, "postDate": "2023-09-20T23:28:58.520Z", "content": "Thanks. Good luck too.
", "rawMarkdown": "Thanks. Good luck too." } ] }, { "id": 2447907, "postDate": "2023-09-20T09:59:41.447Z", "content": "Great to hear @aidlre001 ! Looking forward for your maneuver.
", "rawMarkdown": "Great to hear @aidlre001 ! Looking forward for your maneuver.", "votes": 1, "replies": [ { "id": 2448993, "postDate": "2023-09-20T23:28:33.087Z", "content": "Thanks guys. Good luck to all of us.
", "rawMarkdown": "Thanks guys. Good luck to all of us.", "votes": 1 } ] } ] }, { "id": 2443815, "postDate": "2023-09-18T02:46:31.853Z", "content": "Not more than 9 hours according to the competition rules. In my case, it looks like it will be close to 9 hours due to the data size and limited computing power.
", "rawMarkdown": "Not more than 9 hours according to the competition rules. In my case, it looks like it will be close to 9 hours due to the data size and limited computing power.", "votes": 2, "replies": [ { "id": 2444264, "postDate": "2023-09-18T08:27:16.560Z", "content": "It runs fast on my own end until the scoring part of the submission. That's the weird thing. Thought I had solved the memory bottleneck by breaking down the files into batches. I don't know if it will be useful to folks but I will have to redo it with the 15+ features and see how it goes.
", "rawMarkdown": "It runs fast on my own end until the scoring part of the submission. That's the weird thing. Thought I had solved the memory bottleneck by breaking down the files into batches. I don't know if it will be useful to folks but I will have to redo it with the 15+ features and see how it goes." } ] }, { "id": 2447547, "postDate": "2023-09-20T06:24:43.097Z", "content": "Got a \"Notebook threw an exception\" after over 9 hours waiting for scoring. Now that's sure to motivate anyone to keep trying.
", "rawMarkdown": "Got a \"Notebook threw an exception\" after over 9 hours waiting for scoring. Now that's sure to motivate anyone to keep trying.", "replies": [ { "id": 2447587, "postDate": "2023-09-20T06:52:33.197Z", "content": "Open your submitted notebook and check all the cell outputs, you might find the code where the issue is happening. If you still can not find the issue, consider making notebook public, maybe someone can help you debug it.
", "rawMarkdown": "Open your submitted notebook and check all the cell outputs, you might find the code where the issue is happening. If you still can not find the issue, consider making notebook public, maybe someone can help you debug it.", "votes": 1, "replies": [ { "id": 2447605, "postDate": "2023-09-20T06:58:12.140Z", "content": "Thanks @Anit Reddy,
\nI will do that. Thanks for taking the time. My face has been bashed in from banging my head on the wall repeatedly for some time now.
", "rawMarkdown": "Thanks @Anit Reddy,\n\nI will do that. Thanks for taking the time. My face has been bashed in from banging my head on the wall repeatedly for some time now." } ] } ] }, { "id": 2443681, "postDate": "2023-09-17T21:14:40.393Z", "content": "It depends on your pipeline. However, if you check the public notebooks using GBDT models, the training on lightweight dataset takes 3-4 hours not taking feature engineering and inference time into consideration.
", "rawMarkdown": "It depends on your pipeline. However, if you check the public notebooks using GBDT models, the training on lightweight dataset takes 3-4 hours not taking feature engineering and inference time into consideration.", "replies": [ { "id": 2444267, "postDate": "2023-09-18T08:29:00.997Z", "content": "Yikes! I an in a place with unstable power so I can only come online when power and network is available. I work mostly offline on this. I hope that the notebook scoring can keep running even with a power and network outage on my end?
", "rawMarkdown": "Yikes! I an in a place with unstable power so I can only come online when power and network is available. I work mostly offline on this. I hope that the notebook scoring can keep running even with a power and network outage on my end?" } ] } ], "comments": [ { "id": 2444564, "author_name": "Anil Reddy", "author_url": "", "post_date": "2023-09-18T10:40:11.207000", "content": "Consider saving your model and reload it for submission rather than training again. One of the major bottleneck I observed is conversion of timestamp column to timestamp object.
", "votes": 7, "replies": [ { "id": 2444864, "author_name": "Rimba Erlangga", "author_url": "", "post_date": "2023-09-18T14:18:33.913000", "content": "Agree on this. For a comparison, converting timestamps of 3 series IDs with least steps take ~10s, and ~30 series take ~100s in my case.
", "votes": 3, "replies": [] }, { "id": 2445712, "author_name": "Ryota_Py", "author_url": "", "post_date": "2023-09-19T03:35:06.293000", "content": "Oops. Reusing the model would have saved my day.
\nI'll try that next time.
\nThanks @anilreddyvv !
Been trying that. Still failed even with a reloaded trained model. Don't know what's going on anymore to be honest.
", "votes": 0, "replies": [ { "id": 2447571, "author_name": "Anil Reddy", "author_url": "", "post_date": "2023-09-20T06:47:30.070000", "content": "Try to predict on the entire train data of competition (including feature extraction) using your saved model and observe how much time is taking, do this experimentation in kaggle notebook only.
", "votes": 2, "replies": [] }, { "id": 2447584, "author_name": "AIDLRE001", "author_url": "", "post_date": "2023-09-20T06:51:56.647000", "content": "On the kaggle notebook it doesn't take long since it is pretrained. Less than 3 or 5 mins (I think). The problem comes when submitting for scoring.
", "votes": 0, "replies": [] }, { "id": 2447593, "author_name": "AIDLRE001", "author_url": "", "post_date": "2023-09-20T06:55:15.770000", "content": "Whoops. Sorry. Do you mean I should use the entire training set as a test set to gauge the runtime? Am I getting you right?
", "votes": 0, "replies": [] }, { "id": 2447601, "author_name": "", "author_url": "", "post_date": "2023-09-20T06:56:54.873000", "content": "", "votes": 0, "replies": [] }, { "id": 2447607, "author_name": "Anil Reddy", "author_url": "", "post_date": "2023-09-20T07:00:06.387000", "content": "Yes, because using saved model or even training models like lightgbm, my submission is getting scored in less than 2 hrs. If you have complex model or more number of features it might take comparatively more time.
", "votes": 0, "replies": [] }, { "id": 2447631, "author_name": "AIDLRE001", "author_url": "", "post_date": "2023-09-20T07:10:16.177000", "content": "This is the time I got with the pretrained model when predicting on the entire training dataset on my end:
\nAfter loading pretrained model
\nCPU times: user 3 µs, sys: 0 ns, total: 3 µs
\nWall time: 5.25 µs
After prediction using pretrained model
\nCPU times: user 2 µs, sys: 0 ns, total: 2 µs
\nWall time: 5.96 µs
Seems a bit strange. Or is this correct?
", "votes": 0, "replies": [] }, { "id": 2447634, "author_name": "AIDLRE001", "author_url": "", "post_date": "2023-09-20T07:11:42.607000", "content": "Meanwhile, still awaiting scoring @23 mins and counting…
", "votes": 0, "replies": [] }, { "id": 2447646, "author_name": "AIDLRE001", "author_url": "", "post_date": "2023-09-20T07:16:40.953000", "content": "I understand that but why wait till after 9 hours and then throwing an exception. Lost 4 submissions just like that.
\nEven on the GPU starter notebook in another competition ran out of memory just rerunning it without doing anything. Is it from my end or what? Other people run the same or similar code and it gets scored.
", "votes": 0, "replies": [] }, { "id": 2447651, "author_name": "AIDLRE001", "author_url": "", "post_date": "2023-09-20T07:21:03.327000", "content": "Out of memory or timeout messages I can understand but this exception throwing messages are ambiguous because the code runs fine offline and online even with larger offline datasets run elsewhere even if it takes hours.
", "votes": 0, "replies": [] }, { "id": 2447654, "author_name": "Anil Reddy", "author_url": "", "post_date": "2023-09-20T07:22:59.060000", "content": "did you measure you feature engineering time as well?
", "votes": 0, "replies": [] }, { "id": 2447656, "author_name": "AIDLRE001", "author_url": "", "post_date": "2023-09-20T07:23:19.930000", "content": "Another weird thing is that the same code submitted twice or more, the newer submitted one fails or goes through while the earlier submissions keep on until a very long time and then fail as well.
", "votes": 0, "replies": [] }, { "id": 2447662, "author_name": "Anil Reddy", "author_url": "", "post_date": "2023-09-20T07:29:43.283000", "content": "The problem when using GPU is you get less RAM, and if you are doing multiple pandas operation like timestamp conversion on entire test set, it is going to run out of memory. You should probably consider loading maybe one series like
\ntest = pd.read_parquet(TEST_FILE, filters=[('series_id', '=', series)]).
\nand using
\ndel test
\ngc.collect() and load next batch
\nYou can refer here
Is it scoring error or any other error?
Here are the times:
\nCPU times: user 4 µs, sys: 0 ns, total: 4 µs
\nWall time: 6.68 µs
\nCPU times: user 4 µs, sys: 1 µs, total: 5 µs
\nWall time: 6.68 µs
\nCPU times: user 3 µs, sys: 0 ns, total: 3 µs
\nWall time: 6.2 µs
\nCPU times: user 3 µs, sys: 0 ns, total: 3 µs
\nWall time: 6.2 µs
\nCPU times: user 5 µs, sys: 1 µs, total: 6 µs
\nWall time: 7.39 µs
\nCPU times: user 3 µs, sys: 0 ns, total: 3 µs
\nWall time: 6.2 µs
I think you are doing something wrong because all the logs shows timing as microseconds
", "votes": 1, "replies": [] }, { "id": 2447670, "author_name": "AIDLRE001", "author_url": "", "post_date": "2023-09-20T07:36:16.630000", "content": "thrown exception is the message I keep getting after long periods of scoring. On the Active events tab, it always says notebook \"successful\" but competition scoring after hours, says \"failed\" and opening it shows thrown exception errors.
", "votes": 0, "replies": [] }, { "id": 2447676, "author_name": "AIDLRE001", "author_url": "", "post_date": "2023-09-20T07:39:07.937000", "content": "I am using the edited lgbm notebook of renatoreggian
\nTrained and saved model and relaunched model to save time in prediction. Another weird thing is that my earlier saved gradient boosting model even when uploaded unto kaggle folders is no where to be found and when I upload it again it says that a file with the same name exists. Even when trying to upload to a new created folder, it still prevents me from uploading the file again.
", "votes": 0, "replies": [] }, { "id": 2447684, "author_name": "AIDLRE001", "author_url": "", "post_date": "2023-09-20T07:45:53.120000", "content": "Hi, I have been using this your exact solution and it runs okay on my end. Only when its time for scoring is where there is problem.
", "votes": 0, "replies": [] }, { "id": 2447691, "author_name": "Anil Reddy", "author_url": "", "post_date": "2023-09-20T07:53:23.587000", "content": "I upload it again it says that a file with the same name exists, You can create a fresh dataset if you have issues uploading to existing dataset. The notebook you posted looks like ran without any issues, debug the changes you made on top of that notebook.
Thank you so much @anilreddyvv for this idea about save and reload model. Best regards!!
", "votes": 0, "replies": [] } ] }, { "id": 2444181, "author_name": "Rimba Erlangga", "author_url": "", "post_date": "2023-09-18T07:48:20.963000", "content": "Hi @aidlre001
\nIt really depends on your workflow.
\nHow many percent of the datasets are you using on training? The lightweight dataset from dr @carlmcbrideellis could be the fastest to work on since it only contains ~30 series from total ~200 from the whole training set.
\nWhat algorithm did you use? Based on the public notebook, XGBoost or sklearn GradientBoostingClassifier could take hours, while my work using LGBM is done in under 1h.
\nHow many features did you use? The more features, the longer time your solution will run.
\nOther aspects like the accelerators or the libraries you used to load data might impact your running time as well.
", "votes": 4, "replies": [ { "id": 2444259, "author_name": "AIDLRE001", "author_url": "", "post_date": "2023-09-18T08:24:38.323000", "content": "Hi Rimba,
\nI used the entire dataset but processed in parquet file batches and using the earlier three features from Carl McBride Ellis's notebook. The weird thing is that it runs fast in my notebook session and when I submitted it says successful but the scoring stage is what takes a long time. It basically stays at
\n\"Scoring… \"
Yes probably it is because of the size of the hidden test set. The hidden test set contains roughly similar number of series ID as in the training set (~200), so I think if you want to project the scoring time, you can try using the full ~200 series IDs in the training set first.
", "votes": 0, "replies": [] }, { "id": 2444305, "author_name": "AIDLRE001", "author_url": "", "post_date": "2023-09-18T08:44:15.950000", "content": "But the thing is that my earlier submissions using the lightweight dataset for training using the same classifier was relatively faster in the submission scoring stages. So if they are testing my submission with the same hidden dataset of same size, why the runtime disparity?
", "votes": 0, "replies": [] }, { "id": 2444319, "author_name": "AIDLRE001", "author_url": "", "post_date": "2023-09-18T08:47:16.290000", "content": "Using the entire training set without batch processing crashes both online and offline, thus the reason for breaking it down to parquet file batches. Oh well, will try with the multi feature training set and see. Thanks for all your help and taking the time. I hope I can get something that would be useful for others here too.
", "votes": 0, "replies": [] }, { "id": 2447905, "author_name": "Rimba Erlangga", "author_url": "", "post_date": "2023-09-20T09:58:38.327000", "content": "I think because the lightweight datasets contains much less than 200 series, and the hidden one is 200-ish, hence the disparity. You can try your own suggestion as well. Good luck!
", "votes": 1, "replies": [] } ] } ] }, { "id": 2447733, "author_name": "AIDLRE001", "author_url": "", "post_date": "2023-09-20T08:16:28.043000", "content": "@Anil Reddy and @Rimba Erlangga,
\nThank you for your helpful responses on here. It seems my complaining worked. The code has finally been scored for the first time. However, its saying it is not an improvement over my previous scores (which never scored). I wish I did more so I could have something to share but the time wasted on this scoring issue has me drained so I will go back offline to continue working and hopefully have something good to share. Thanks again to all involved commenters.
", "votes": 2, "replies": [ { "id": 2447743, "author_name": "Anil Reddy", "author_url": "", "post_date": "2023-09-20T08:24:35.360000", "content": "Good luck @aidlre001
", "votes": 1, "replies": [ { "id": 2448994, "author_name": "AIDLRE001", "author_url": "", "post_date": "2023-09-20T23:28:58.520000", "content": "Thanks. Good luck too.
", "votes": 0, "replies": [] } ] }, { "id": 2447907, "author_name": "Rimba Erlangga", "author_url": "", "post_date": "2023-09-20T09:59:41.447000", "content": "Great to hear @aidlre001 ! Looking forward for your maneuver.
", "votes": 1, "replies": [ { "id": 2448993, "author_name": "AIDLRE001", "author_url": "", "post_date": "2023-09-20T23:28:33.087000", "content": "Thanks guys. Good luck to all of us.
", "votes": 1, "replies": [] } ] } ] }, { "id": 2443815, "author_name": "Mario", "author_url": "", "post_date": "2023-09-18T02:46:31.853000", "content": "Not more than 9 hours according to the competition rules. In my case, it looks like it will be close to 9 hours due to the data size and limited computing power.
", "votes": 2, "replies": [ { "id": 2444264, "author_name": "AIDLRE001", "author_url": "", "post_date": "2023-09-18T08:27:16.560000", "content": "It runs fast on my own end until the scoring part of the submission. That's the weird thing. Thought I had solved the memory bottleneck by breaking down the files into batches. I don't know if it will be useful to folks but I will have to redo it with the 15+ features and see how it goes.
", "votes": 0, "replies": [] } ] }, { "id": 2447547, "author_name": "AIDLRE001", "author_url": "", "post_date": "2023-09-20T06:24:43.097000", "content": "Got a \"Notebook threw an exception\" after over 9 hours waiting for scoring. Now that's sure to motivate anyone to keep trying.
", "votes": 0, "replies": [ { "id": 2447587, "author_name": "Anil Reddy", "author_url": "", "post_date": "2023-09-20T06:52:33.197000", "content": "Open your submitted notebook and check all the cell outputs, you might find the code where the issue is happening. If you still can not find the issue, consider making notebook public, maybe someone can help you debug it.
", "votes": 1, "replies": [ { "id": 2447605, "author_name": "AIDLRE001", "author_url": "", "post_date": "2023-09-20T06:58:12.140000", "content": "Thanks @Anit Reddy,
\nI will do that. Thanks for taking the time. My face has been bashed in from banging my head on the wall repeatedly for some time now.
", "votes": 0, "replies": [] } ] } ] }, { "id": 2443681, "author_name": "awqatak", "author_url": "", "post_date": "2023-09-17T21:14:40.393000", "content": "It depends on your pipeline. However, if you check the public notebooks using GBDT models, the training on lightweight dataset takes 3-4 hours not taking feature engineering and inference time into consideration.
", "votes": 0, "replies": [ { "id": 2444267, "author_name": "AIDLRE001", "author_url": "", "post_date": "2023-09-18T08:29:00.997000", "content": "Yikes! I an in a place with unstable power so I can only come online when power and network is available. I work mostly offline on this. I hope that the notebook scoring can keep running even with a power and network outage on my end?
", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "2444564": "Consider saving your model and reload it for submission rather than training again. One of the major bottleneck I observed is conversion of timestamp column to timestamp object.", "2444181": "Hi @aidlre001 \n\nIt really depends on your workflow.\n\nHow many percent of the datasets are you using on training? The lightweight dataset from dr @carlmcbrideellis could be the fastest to work on since it only contains ~30 series from total ~200 from the whole training set.\n\nWhat algorithm did you use? Based on the public notebook, XGBoost or sklearn GradientBoostingClassifier could take hours, while my work using LGBM is done in under 1h.\n\nHow many features did you use? The more features, the longer time your solution will run.\n\nOther aspects like the accelerators or the libraries you used to load data might impact your running time as well.", "2443566": "Hello all. A novice such as myself is wondering, how long does a scoring take on kaggle after a submission? <= 30 mins or > 30 mins? Should I be worried if it is taking too long to score? Is it anything to do with the user or not? Thanks.", "2447733": "@Anil Reddy and @Rimba Erlangga,\n\nThank you for your helpful responses on here. It seems my complaining worked. The code has finally been scored for the first time. However, its saying it is not an improvement over my previous scores (which never scored). I wish I did more so I could have something to share but the time wasted on this scoring issue has me drained so I will go back offline to continue working and hopefully have something good to share. Thanks again to all involved commenters.", "2443815": "Not more than 9 hours according to the competition rules. In my case, it looks like it will be close to 9 hours due to the data size and limited computing power.", "2447547": "Got a \"Notebook threw an exception\" after over 9 hours waiting for scoring. Now that's sure to motivate anyone to keep trying.", "2443681": "It depends on your pipeline. However, if you check the public notebooks using GBDT models, the training on lightweight dataset takes 3-4 hours not taking feature engineering and inference time into consideration." } }