![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2745910%2Fb599335565970271af41965885cabd85%2FScreen%20Shot%202019-07-24%20at%201.31.53%20PM.png?generation=1564000537181446&alt=media\")
This is my second Kaggle competition and I didn't have a good habit of log things down since everything I did before was easy and small so that my bare brain was still capable of memorizing most of them, or at least I could re-run them in no time. As things getting complicated, running several different pipelines with all kinds of hyper-parameter and architectures, I finally realized it is crucial to write down everything. I'm currently using a probably out of fashion spreadsheet style and I'm really curious about how you guys keep track of your different models.
", "messages": [ { "id": "583204", "postDate": "07/24/2019 06:44:43", "content": "This is my second Kaggle competition and I didn't have a good habit of log things down since everything I did before was easy and small so that my bare brain was still capable of memorizing most of them, or at least I could re-run them in no time. As things getting complicated, running several different pipelines with all kinds of hyper-parameter and architectures, I finally realized it is crucial to write down everything. I'm currently using a probably out of fashion spreadsheet style and I'm really curious about how you guys keep track of your different models.
", "rawMarkdown": "This is my second Kaggle competition and I didn't have a good habit of log things down since everything I did before was easy and small so that my bare brain was still capable of memorizing most of them, or at least I could re-run them in no time. As things getting complicated, running several different pipelines with all kinds of hyper-parameter and architectures, I finally realized it is crucial to write down everything. I'm currently using a probably out of fashion spreadsheet style and I'm really curious about how you guys keep track of your different models.", "votes": null }, { "id": "583686", "postDate": "07/24/2019 20:35:43", "content": "Since Kaggle's cloud kernel doesn't have a commit message, only comparing differences. This is something I did.
\n\n
Nice
", "rawMarkdown": "Nice", "votes": null }, { "id": "584172", "postDate": "07/25/2019 14:16:15", "content": "This is really neat. I'll start incorporating this style from now. Thanks for the demo.
", "rawMarkdown": "This is really neat. I'll start incorporating this style from now. Thanks for the demo.", "votes": null }, { "id": "584205", "postDate": "07/25/2019 14:49:52", "content": "Recently I've been been generating a run_id/model_id every time I run a model training script. This is usually either a datetime value or a generated string based on various model parameters that get passed as arguments (e.g. ResNet50_512_bs_4_epochs_50). A subdirectory gets created using the run_id/model_id and then everything gets saved under there. I save a copy of the script currently being ran and any other scripts/modules that need saving. This way if I ever need to run it again exactly the same I have all of the files exactly as they were at runtime. I basically end up with a directory structure that looks something like this:
./models/{run_id}/train.py\n./models/{run_id}/utils.py\n./models/{run_id}/args.yaml\n./models/{run_id}/submission.csv\n./models/{run_id}/log.txt\n./models/{run_id}/scoring_history.csv
Then if you want to compare models you can have another script that goes through all of your run_id directories and extracts any information you need e.g. best score from scoring_history.py, hyperparameters settings from args.yaml, etc. You can also add a \"note\" argument so that you can save something similar to a commit message to each of your runs.
I've also seen sacred used to log experiments in Python (https://github.com/IDSIA/sacred) but I've never used it myself. There might be other similar things out there. I haven't seen anything similar for R.
Thanks for the sharing, I really like the robustness of your style. I'll definitely try it. Sacred looks really promising and I wanted to try it before. But a little bit search revealed its seeming lack of support for jupyter notebook since I'm mainly using AWS Sagemaker for the majority of training.
", "rawMarkdown": "Thanks for the sharing, I really like the robustness of your style. I'll definitely try it. Sacred looks really promising and I wanted to try it before. But a little bit search revealed its seeming lack of support for jupyter notebook since I'm mainly using AWS Sagemaker for the majority of training.", "votes": null } ], "comments": [ { "id": 583686, "author_name": "scaomath", "author_url": "", "post_date": "07/24/2019 20:35:43", "content": "Since Kaggle's cloud kernel doesn't have a commit message, only comparing differences. This is something I did.
\n\n
Nice
", "votes": null, "replies": [] }, { "id": 584172, "author_name": "zachariahpang", "author_url": "", "post_date": "07/25/2019 14:16:15", "content": "This is really neat. I'll start incorporating this style from now. Thanks for the demo.
", "votes": null, "replies": [] } ] }, { "id": 584205, "author_name": "brandenkmurray", "author_url": "", "post_date": "07/25/2019 14:49:52", "content": "Recently I've been been generating a run_id/model_id every time I run a model training script. This is usually either a datetime value or a generated string based on various model parameters that get passed as arguments (e.g. ResNet50_512_bs_4_epochs_50). A subdirectory gets created using the run_id/model_id and then everything gets saved under there. I save a copy of the script currently being ran and any other scripts/modules that need saving. This way if I ever need to run it again exactly the same I have all of the files exactly as they were at runtime. I basically end up with a directory structure that looks something like this:
./models/{run_id}/train.py\n./models/{run_id}/utils.py\n./models/{run_id}/args.yaml\n./models/{run_id}/submission.csv\n./models/{run_id}/log.txt\n./models/{run_id}/scoring_history.csv
Then if you want to compare models you can have another script that goes through all of your run_id directories and extracts any information you need e.g. best score from scoring_history.py, hyperparameters settings from args.yaml, etc. You can also add a \"note\" argument so that you can save something similar to a commit message to each of your runs.
I've also seen sacred used to log experiments in Python (https://github.com/IDSIA/sacred) but I've never used it myself. There might be other similar things out there. I haven't seen anything similar for R.
Thanks for the sharing, I really like the robustness of your style. I'll definitely try it. Sacred looks really promising and I wanted to try it before. But a little bit search revealed its seeming lack of support for jupyter notebook since I'm mainly using AWS Sagemaker for the majority of training.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "583204": "This is my second Kaggle competition and I didn't have a good habit of log things down since everything I did before was easy and small so that my bare brain was still capable of memorizing most of them, or at least I could re-run them in no time. As things getting complicated, running several different pipelines with all kinds of hyper-parameter and architectures, I finally realized it is crucial to write down everything. I'm currently using a probably out of fashion spreadsheet style and I'm really curious about how you guys keep track of your different models.", "583686": "* Log the changes. \n\nSince Kaggle's cloud kernel doesn't have a commit message, only comparing differences. This is something I did.\n\n", "583950": "Nice", "584172": "This is really neat. I'll start incorporating this style from now. Thanks for the demo.", "584205": "Recently I've been been generating a `run_id/model_id` every time I run a model training script. This is usually either a `datetime` value or a generated string based on various model parameters that get passed as arguments (e.g. `ResNet50_512_bs_4_epochs_50`). A subdirectory gets created using the `run_id/model_id` and then everything gets saved under there. I save a copy of the script currently being ran and any other scripts/modules that need saving. This way if I ever need to run it again exactly the same I have all of the files exactly as they were at runtime. I basically end up with a directory structure that looks something like this:\n\n`./models/{run_id}/train.py`\n`./models/{run_id}/utils.py`\n`./models/{run_id}/args.yaml`\n`./models/{run_id}/submission.csv`\n`./models/{run_id}/log.txt`\n`./models/{run_id}/scoring_history.csv`\n\nThen if you want to compare models you can have another script that goes through all of your `run_id` directories and extracts any information you need e.g. best score from `scoring_history.py`, hyperparameters settings from `args.yaml`, etc. You can also add a \"note\" argument so that you can save something similar to a commit message to each of your runs.\n\nI've also seen `sacred` used to log experiments in Python (https://github.com/IDSIA/sacred) but I've never used it myself. There might be other similar things out there. I haven't seen anything similar for R.", "584465": "Thanks for the sharing, I really like the robustness of your style. I'll definitely try it. Sacred looks really promising and I wanted to try it before. But a little bit search revealed its seeming lack of support for jupyter notebook since I'm mainly using AWS Sagemaker for the majority of training." }, "source": "meta" }