Hi!
\nI have question about submission notebook.
\nIs it ok to train model on local pc and download trained model parameter to kaggle dataset and
\nsubmit that notebook?
Hi!
\nI have question about submission notebook.
\nIs it ok to train model on local pc and download trained model parameter to kaggle dataset and
\nsubmit that notebook?
Yes; anything you make yourself can be used in a private dataset. If it's a model produced by someone else, then that's when it must be freely available to all participants.
", "rawMarkdown": "Yes; anything you make yourself can be used in a private dataset. If it's a model produced by someone else, then that's when it must be freely available to all participants.", "votes": null }, { "id": "2511299", "postDate": "11/03/2023 15:07:50", "content": "Hi, chris
\nthanks for reply. I understood.
\nI'm trying to learn Lgbm model but, memory error occurs because of feature columns are too much. So, I'm thinking to train model on mac pc. That why I asked this question.
\nThank you for answering.
Yes of course, this is a preferred method in code competitions @kazumamori
", "rawMarkdown": "Yes of course, this is a preferred method in code competitions @kazumamori", "votes": null }, { "id": "2511413", "postDate": "11/03/2023 16:32:44", "content": "You can use joblib to save the model and then load this using a Kaggle dataset and use it. Just ensure that the LGBM version you use are the same between the Kaggle kernel and your local environment @kazumamori
", "rawMarkdown": "You can use joblib to save the model and then load this using a Kaggle dataset and use it. Just ensure that the LGBM version you use are the same between the Kaggle kernel and your local environment @kazumamori", "votes": null }, { "id": "2512305", "postDate": "11/04/2023 10:52:21", "content": "@ravi20076
\nThank you for sharing your knowledge.
\nI didn't check version, thanks to notice.
\nby the way, I used pickle to save my model.
I am new to Kaggle and not familiar with its tools. Can you point me to where I can find out how to save a trained model so it can be used in the submission process? my notebook takes about 3 hours to run to submission is painful.
\nThank you
", "rawMarkdown": "I am new to Kaggle and not familiar with its tools. Can you point me to where I can find out how to save a trained model so it can be used in the submission process? my notebook takes about 3 hours to run to submission is painful.\n\nThank you", "votes": null }, { "id": "2518340", "postDate": "11/09/2023 08:07:10", "content": "If you are using torch, you can use the following :
\n\ntorch.save(model.state_dict(), PATH)\n\n\nmodel = YourModel(*args)\nmodel.load_state_dict(torch.load(PATH))\nmodel.()\n@kazumamori @nadereafshar yes, for sure you can train the model outside of kaggle, add the trained weights to kaggle as the dataset and use it during your submission. But be careful in this case with the environment you used outside kaggle - to receive the same model predictions outside Kaggle and in kernels.
\nAnother way to reduce submission time is to divide you train+inference kernel into 2 parts: first for train and second for inference. The output of the first kernel (model weights) will go to the second one (inference part) and you now submit only the inference kernel.
\nHope this helps :)
", "rawMarkdown": "kazumamori @nadereafshar yes, for sure you can train the model outside of kaggle, add the trained weights to kaggle as the dataset and use it during your submission. But be careful in this case with the environment you used outside kaggle - to receive the same model predictions outside Kaggle and in kernels.\n\nAnother way to reduce submission time is to divide you train+inference kernel into 2 parts: first for train and second for inference. The output of the first kernel (model weights) will go to the second one (inference part) and you now submit only the inference kernel.\n\nHope this helps :)", "votes": null }, { "id": "2518976", "postDate": "11/09/2023 16:28:25", "content": "hello again Alexander!
\nThank your for sharing your tip.
\nyou mean making two notebooks for train and inference?
\nI use local mac computer to train model because kaggle notebook get memory error while training.So, I'm using outside environment.
\n@alexryzhkov
@kazumamori yes, I mean 2 separated notebooks or the notebook in the local env + inference on kaggle 👍
", "rawMarkdown": "kazumamori yes, I mean 2 separated notebooks or the notebook in the local env + inference on kaggle 👍", "votes": null }, { "id": "2518997", "postDate": "11/09/2023 16:40:27", "content": "@nadereafshar
\nHi,Nader
\nI used this code to save my model.
pickle\n\n\nfile_name = \n\n\nmodel = model_instance\n\n\n (file_name, ) f:\n pickle.dump(model, f)\nif you want to download saved file, write this
\n\nfolder_path = \n\n\nfile_name = \n\n\n (file_name), ) f:\n trained_model = pickle.load(f)\nYes; anything you make yourself can be used in a private dataset. If it's a model produced by someone else, then that's when it must be freely available to all participants.
", "votes": null, "replies": [ { "id": 2511299, "author_name": "kazumamori", "author_url": "", "post_date": "11/03/2023 15:07:50", "content": "Hi, chris
\nthanks for reply. I understood.
\nI'm trying to learn Lgbm model but, memory error occurs because of feature columns are too much. So, I'm thinking to train model on mac pc. That why I asked this question.
\nThank you for answering.
You can use joblib to save the model and then load this using a Kaggle dataset and use it. Just ensure that the LGBM version you use are the same between the Kaggle kernel and your local environment @kazumamori
", "votes": null, "replies": [ { "id": 2512305, "author_name": "kazumamori", "author_url": "", "post_date": "11/04/2023 10:52:21", "content": "@ravi20076
\nThank you for sharing your knowledge.
\nI didn't check version, thanks to notice.
\nby the way, I used pickle to save my model.
Yes of course, this is a preferred method in code competitions @kazumamori
", "votes": null, "replies": [] }, { "id": 2517907, "author_name": "nadereafshar", "author_url": "", "post_date": "11/08/2023 21:11:28", "content": "I am new to Kaggle and not familiar with its tools. Can you point me to where I can find out how to save a trained model so it can be used in the submission process? my notebook takes about 3 hours to run to submission is painful.
\nThank you
", "votes": null, "replies": [ { "id": 2518340, "author_name": "fahmiayari", "author_url": "", "post_date": "11/09/2023 08:07:10", "content": "If you are using torch, you can use the following :
\n\ntorch.save(model.state_dict(), PATH)\n\n\nmodel = YourModel(*args)\nmodel.load_state_dict(torch.load(PATH))\nmodel.()\n@nadereafshar
\nHi,Nader
\nI used this code to save my model.
pickle\n\n\nfile_name = \n\n\nmodel = model_instance\n\n\n (file_name, ) f:\n pickle.dump(model, f)\nif you want to download saved file, write this
\n\nfolder_path = \n\n\nfile_name = \n\n\n (file_name), ) f:\n trained_model = pickle.load(f)\n@kazumamori @nadereafshar yes, for sure you can train the model outside of kaggle, add the trained weights to kaggle as the dataset and use it during your submission. But be careful in this case with the environment you used outside kaggle - to receive the same model predictions outside Kaggle and in kernels.
\nAnother way to reduce submission time is to divide you train+inference kernel into 2 parts: first for train and second for inference. The output of the first kernel (model weights) will go to the second one (inference part) and you now submit only the inference kernel.
\nHope this helps :)
", "votes": null, "replies": [ { "id": 2518976, "author_name": "kazumamori", "author_url": "", "post_date": "11/09/2023 16:28:25", "content": "hello again Alexander!
\nThank your for sharing your tip.
\nyou mean making two notebooks for train and inference?
\nI use local mac computer to train model because kaggle notebook get memory error while training.So, I'm using outside environment.
\n@alexryzhkov
@kazumamori yes, I mean 2 separated notebooks or the notebook in the local env + inference on kaggle 👍
", "votes": null, "replies": [] } ] } ] } ], "raw_markdown_by_id": { "2511154": "Hi!\nI have question about submission notebook.\nIs it ok to train model on local pc and download trained model parameter to kaggle dataset and\nsubmit that notebook?", "2511201": "Yes; anything you make yourself can be used in a private dataset. If it's a model produced by someone else, then that's when it must be freely available to all participants.", "2511299": "Hi, chris\nthanks for reply. I understood.\nI'm trying to learn Lgbm model but, memory error occurs because of feature columns are too much. So, I'm thinking to train model on mac pc. That why I asked this question.\nThank you for answering.", "2511319": "Yes of course, this is a preferred method in code competitions @kazumamori", "2511413": "You can use joblib to save the model and then load this using a Kaggle dataset and use it. Just ensure that the LGBM version you use are the same between the Kaggle kernel and your local environment @kazumamori", "2512305": "ravi20076 \nThank you for sharing your knowledge.\nI didn't check version, thanks to notice.\nby the way, I used pickle to save my model.", "2517907": "I am new to Kaggle and not familiar with its tools. Can you point me to where I can find out how to save a trained model so it can be used in the submission process? my notebook takes about 3 hours to run to submission is painful.\n\nThank you", "2518340": "If you are using torch, you can use the following : \n\n```python\n# save model after training\ntorch.save(model.state_dict(), PATH)\n\n#load model when doing inference\nmodel = YourModel(*args)\nmodel.load_state_dict(torch.load(PATH))\nmodel.eval()\n```", "2518786": "kazumamori @nadereafshar yes, for sure you can train the model outside of kaggle, add the trained weights to kaggle as the dataset and use it during your submission. But be careful in this case with the environment you used outside kaggle - to receive the same model predictions outside Kaggle and in kernels.\n\nAnother way to reduce submission time is to divide you train+inference kernel into 2 parts: first for train and second for inference. The output of the first kernel (model weights) will go to the second one (inference part) and you now submit only the inference kernel.\n\nHope this helps :)", "2518976": "hello again Alexander!\nThank your for sharing your tip.\nyou mean making two notebooks for train and inference?\nI use local mac computer to train model because kaggle notebook get memory error while training.So, I'm using outside environment.\n@alexryzhkov", "2518993": "kazumamori yes, I mean 2 separated notebooks or the notebook in the local env + inference on kaggle 👍", "2518997": "nadereafshar \nHi,Nader\nI used this code to save my model.\n\n```python\nimport pickle\n\n# Set the file name to save the model\nfile_name = 'trained_model.pkl'\n\n# Set the model instance to save\nmodel = model_instance\n\n# Save the model to a file using pickle\nwith open(file_name, 'wb') as f:\n pickle.dump(model, f)\n```\nif you want to download saved file, write this\n\n```python\n# Set the folder path where the trained model file is located\nfolder_path = '/path/to/folder/'\n\n# Set the filename of the trained model to load\nfile_name = 'trained_model.pkl'\n\n# Load the trained model from the file using pickle\nwith open(file_name), 'rb') as f:\n trained_model = pickle.load(f)\n```" }, "source": "meta" }