![\"enter](\"https://storage.googleapis.com/kaggle-forum-message-attachments/315500/9208/post-submission.png\")
my submission code (training, inference, ensemble, model) is at stage2: https://www.kaggle.com/c/data-science-bowl-2018/discussion/53720 or here https://drive.google.com/drive/folders/1QeMFRfiW95rudGLwKYNy-weLpP-5L6-S
\n\nAt the submission :
\n\nfirst submission(set FP reject threshold = 4)
\n\npublic LB = 0.505
private LB = 0.574
second submission(set FP reject threshold = 0.5)
\n\npublic LB = 0.501
private LB = 0.505
Post submission:
\n\nprivate LB (2800 gray images only)= 0.506 (set FP reject threshold = 4)
private LB (2800 gray images only)= 0.521 (set FP reject threshold = 6)
private LB (2800 gray images only)= 0.539 (set FP reject threshold = 8)
private LB (2800 gray images only)= 0.535 (set FP reject threshold =10)
current best post submission results is \"privateLB-0.607__submission-stage2-LB0.505-ensemble-post-s8.0-gray-old_color.csv\", using threshold=8 for gray images
\n\n
I guessed that is the difference between \"black box\" and \"gray box\" testing. When you get to see your results, you can think of many tricks or parameter adjustment or pre/post processing to improve results.
\n\nto set reject threshold, please refer to s value in ensemble of my code
\n\ns = center[binary].sum()/binary.sum()*num_members\n if s<6: continue\nmy submission code (training, inference, ensemble, model) is at stage2: https://www.kaggle.com/c/data-science-bowl-2018/discussion/53720 or here https://drive.google.com/drive/folders/1QeMFRfiW95rudGLwKYNy-weLpP-5L6-S
\n\nAt the submission :
\n\nfirst submission(set FP reject threshold = 4)
\n\npublic LB = 0.505
private LB = 0.574
second submission(set FP reject threshold = 0.5)
\n\npublic LB = 0.501
private LB = 0.505
Post submission:
\n\nprivate LB (2800 gray images only)= 0.506 (set FP reject threshold = 4)
private LB (2800 gray images only)= 0.521 (set FP reject threshold = 6)
private LB (2800 gray images only)= 0.539 (set FP reject threshold = 8)
private LB (2800 gray images only)= 0.535 (set FP reject threshold =10)
current best post submission results is \"privateLB-0.607__submission-stage2-LB0.505-ensemble-post-s8.0-gray-old_color.csv\", using threshold=8 for gray images
\n\n
I guessed that is the difference between \"black box\" and \"gray box\" testing. When you get to see your results, you can think of many tricks or parameter adjustment or pre/post processing to improve results.
\n\nto set reject threshold, please refer to s value in ensemble of my code
\n\ns = center[binary].sum()/binary.sum()*num_members\n if s<6: continue\nThanks for sharing. This is interesting enough for me to really learn Pytorch!
", "rawMarkdown": "Thanks for sharing. This is interesting enough for me to really learn Pytorch!", "votes": null }, { "id": "315565", "postDate": "04/17/2018 09:24:49", "content": "Thank you for great kernel and I've learned a lot from your previous posts!
", "rawMarkdown": "Thank you for great kernel and I've learned a lot from your previous posts!", "votes": null }, { "id": "315623", "postDate": "04/17/2018 11:15:54", "content": "Yes, same here.
\n\n@heng could be single handedly responsible for PyTorch becoming the new Kaggle standard!
", "rawMarkdown": "Yes, same here. \n\n@heng could be single handedly responsible for PyTorch becoming the new Kaggle standard!", "votes": null }, { "id": "315631", "postDate": "04/17/2018 11:28:36", "content": "Thanks a lot. Why did not use mosaic image? Could you also provide train and val list file?
", "rawMarkdown": "Thanks a lot. Why did not use mosaic image? Could you also provide train and val list file?", "votes": null }, { "id": "315635", "postDate": "04/17/2018 11:33:52", "content": "mosaic is for training. later i hand fixed the training mask and has no time to regenerated mosaic for the hand fixed mask. so the model was fine tuned on hand fixed mask.
\n\nfor testing, i use original images. this is because, i think the ground truth remove annotations near the edge of the images.
", "rawMarkdown": "mosaic is for training. later i hand fixed the training mask and has no time to regenerated mosaic for the hand fixed mask. so the model was fine tuned on hand fixed mask.\n\nfor testing, i use original images. this is because, i think the ground truth remove annotations near the edge of the images.", "votes": null }, { "id": "315640", "postDate": "04/17/2018 11:47:49", "content": "Heng, We found mosaic to help on test prediction.
", "rawMarkdown": "Heng, We found mosaic to help on test prediction.", "votes": null }, { "id": "315648", "postDate": "04/17/2018 12:01:16", "content": "And have you use stage1 mask for training or just original training set? I do not think that manually correct mask improve performance because you can think it is noisy label
", "rawMarkdown": "And have you use stage1 mask for training or just original training set? I do not think that manually correct mask improve performance because you can think it is noisy label", "votes": null }, { "id": "315695", "postDate": "04/17/2018 13:42:55", "content": "Definitely worth a look. Intriguing thing about Pytorch is it's immediate python execution versus the graph definition then session execution in tensorflow. One would think this would give tensorflow an edge in speed, but I'm not sure that is true.
", "rawMarkdown": "Definitely worth a look. Intriguing thing about Pytorch is it's immediate python execution versus the graph definition then session execution in tensorflow. One would think this would give tensorflow an edge in speed, but I'm not sure that is true.", "votes": null }, { "id": "315697", "postDate": "04/17/2018 13:48:30", "content": "We have used the training masks from https://github.com/lopuhin/kaggle-dsbowl-2018-dataset-fixes and the stage1 test labels as is.
", "rawMarkdown": "We have used the training masks from https://github.com/lopuhin/kaggle-dsbowl-2018-dataset-fixes and the stage1 test labels as is.", "votes": null }, { "id": "315701", "postDate": "04/17/2018 13:54:23", "content": "In another competition(CDiscount?) someone reported that Pytorch is faster than Keras. That's anecdotal evidence, not enough to draw a definitive conclusion, but interesting still.
", "rawMarkdown": "In another competition(CDiscount?) someone reported that Pytorch is faster than Keras. That's anecdotal evidence, not enough to draw a definitive conclusion, but interesting still.", "votes": null }, { "id": "315702", "postDate": "04/17/2018 13:55:51", "content": "TF has the edge when you are distributing the model between 6 different machines each with 8 GPUs. PyT can't do that. The overhead for TF being able to do distributed learning is that it is slow for single machines. And that is making PyT better than Keras and TF for Kaggle competitions.
\n\nTF has a new 'Eager Execution Mode' that enables it to do immediate execution and is therefore easier to debug. I haven't used it but I am guessing it will be even slower.
\n\nAbout the only advantage that TF has (over PyT) is the fact that it has lots of models already written for it. But who needs extra models when you have @heng to write them for you?!
", "rawMarkdown": "TF has the edge when you are distributing the model between 6 different machines each with 8 GPUs. PyT can't do that. The overhead for TF being able to do distributed learning is that it is slow for single machines. And that is making PyT better than Keras and TF for Kaggle competitions. \n\nTF has a new 'Eager Execution Mode' that enables it to do immediate execution and is therefore easier to debug. I haven't used it but I am guessing it will be even slower.\n\nAbout the only advantage that TF has (over PyT) is the fact that it has lots of models already written for it. But who needs extra models when you have @heng to write them for you?!", "votes": null }, { "id": "315703", "postDate": "04/17/2018 13:56:03", "content": "i feel that advantage of pytorch is debugging. Maybe there is not so much difference in speed.
\n\nin pytorch, you can set breakpoint and inspect tensor values in gpu memory. pytorch tensor is just like np array, very easy to code. Also, the data loader in pytorch is more user friendly.
\n\nTensorflow strength is probably commercial application deployment.
", "rawMarkdown": "i feel that advantage of pytorch is debugging. Maybe there is not so much difference in speed.\n\nin pytorch, you can set breakpoint and inspect tensor values in gpu memory. pytorch tensor is just like np array, very easy to code. Also, the data loader in pytorch is more user friendly.\n\nTensorflow strength is probably commercial application deployment.", "votes": null }, { "id": "315705", "postDate": "04/17/2018 14:01:12", "content": "Right, Pytorch flexibility makes it appealing for research purposes, but it is definitely not to be used as a deployment platform.
", "rawMarkdown": "Right, Pytorch flexibility makes it appealing for research purposes, but it is definitely not to be used as a deployment platform.", "votes": null }, { "id": "315727", "postDate": "04/17/2018 14:23:49", "content": "I'd rather deploy on something that is properly debugged ;)
\n\nSeriously, though, I don't have much deployment experience. What would be your biggest worry about using PyTorch in deployment. Is distributed learning that useful for most commercial applications? What else would you be worried about. I'd be interested to know.
", "rawMarkdown": "I'd rather deploy on something that is properly debugged ;)\n\nSeriously, though, I don't have much deployment experience. What would be your biggest worry about using PyTorch in deployment. Is distributed learning that useful for most commercial applications? What else would you be worried about. I'd be interested to know.", "votes": null }, { "id": "315744", "postDate": "04/17/2018 14:37:27", "content": "Even Facebooks says Pytorch is not for deployment. They use Caffe2 for that.
", "rawMarkdown": "Even Facebooks says Pytorch is not for deployment. They use Caffe2 for that.", "votes": null }, { "id": "315756", "postDate": "04/17/2018 14:51:35", "content": "Yes, I have read that that is the conventional wisdom, but can you explain to someone who has not worked in commercial ML development (me) what a deployable system can do that I un-deployable one cannot?
\n\nAt what point do I go \"Dang, I wish I had chosen TensorFlow for my new startup\"?
", "rawMarkdown": "Yes, I have read that that is the conventional wisdom, but can you explain to someone who has not worked in commercial ML development (me) what a deployable system can do that I un-deployable one cannot?\n\nAt what point do I go \"Dang, I wish I had chosen TensorFlow for my new startup\"?", "votes": null }, { "id": "315762", "postDate": "04/17/2018 15:02:01", "content": "You need a way to serialize models so that they can be transported where you need the predictions. You then need a small footprint yet scalable prediction layer to be able to read these serialized models and make predictions at scale. You need to be able to version model versions. There is a lot more, and going further I'd be forced to start making publicity for what we do. If you stick to open source then there is little effort aiming at supporting the deployment needs (often called serving in deep learning context). Tensorflow Serving is a notable exception: https://www.tensorflow.org/serving/.
\n\nFacebook teamed with Microsoft Amazon, and others to establish ONNX as a way to exchange deep learning models. This could become the basis for a generic serving layer: https://onnx.ai/\nFacebook also tries to bridge the gap between Pytorch and Caffe2 as the latter is designed for deployment, see https://medium.com/@Synced/caffe2-merges-with-pytorch-a89c70ad9eb7
", "rawMarkdown": "You need a way to serialize models so that they can be transported where you need the predictions. You then need a small footprint yet scalable prediction layer to be able to read these serialized models and make predictions at scale. You need to be able to version model versions. There is a lot more, and going further I'd be forced to start making publicity for what we do. If you stick to open source then there is little effort aiming at supporting the deployment needs (often called *serving* in deep learning context). Tensorflow Serving is a notable exception: https://www.tensorflow.org/serving/.\n\nFacebook teamed with Microsoft Amazon, and others to establish ONNX as a way to exchange deep learning models. This could become the basis for a generic serving layer: https://onnx.ai/\nFacebook also tries to bridge the gap between Pytorch and Caffe2 as the latter is designed for deployment, see https://medium.com/@Synced/caffe2-merges-with-pytorch-a89c70ad9eb7", "votes": null }, { "id": "315763", "postDate": "04/17/2018 15:06:54", "content": "there are many kind of commercial deployment. I mainly deal with computer vision, image recognition on embedded system like on-board system-on-chip, especially ARM processor.
\n\nAs such, the first consideration is cost. Low cost means low cpu/gpu power, low memory and low power. At such, nvidia mobile gpu like TX2 is usually not an option (too expensive).
\n\nThe deep learning framework we decide to use usually depends on the support given by the chip. Most vendors only support caffe (not caffe2) and tensorflow model.
\n\nIf you are talking about mobile app development or server app deployment, it would be a different story.
", "rawMarkdown": "there are many kind of commercial deployment. I mainly deal with computer vision, image recognition on embedded system like on-board system-on-chip, especially ARM processor.\n\nAs such, the first consideration is cost. Low cost means low cpu/gpu power, low memory and low power. At such, nvidia mobile gpu like TX2 is usually not an option (too expensive).\n\nThe deep learning framework we decide to use usually depends on the support given by the chip. Most vendors only support caffe (not caffe2) and tensorflow model.\n\nIf you are talking about mobile app development or server app deployment, it would be a different story.", "votes": null }, { "id": "315768", "postDate": "04/17/2018 15:11:32", "content": "I also have testimony from people that tried both Pytorch and TF for production environments. I'll ask them if I can post their testimony here.
", "rawMarkdown": "I also have testimony from people that tried both Pytorch and TF for production environments. I'll ask them if I can post their testimony here.", "votes": null }, { "id": "315794", "postDate": "04/17/2018 15:59:36", "content": "Hi Heng, thank you for your amazing posts! If you're open, can I be your apprentice? ;)
", "rawMarkdown": "Hi Heng, thank you for your amazing posts! If you're open, can I be your apprentice? ;)", "votes": null }, { "id": "315828", "postDate": "04/17/2018 16:34:15", "content": "\"I see\", said the blind man.
\n\nSo, by deployment, you are talking about AFTER the model has been created and trained? Though it is obviously more efficient to train a model in the same framework in which it is deployed.
\n\nSorry if I am being slow, but my only experience in DL is with Kaggle.
\n\nAnd I am guessing that, as Pros, you guys are bi-lingual in PyT and (TF or caffe).
", "rawMarkdown": "\"I see\", said the blind man. \n\nSo, by deployment, you are talking about AFTER the model has been created and trained? Though it is obviously more efficient to train a model in the same framework in which it is deployed.\n\nSorry if I am being slow, but my only experience in DL is with Kaggle.\n\nAnd I am guessing that, as Pros, you guys are bi-lingual in PyT and (TF or caffe).", "votes": null }, { "id": "315834", "postDate": "04/17/2018 16:43:36", "content": "i started off with tensorflow. Now i need to get back to it because of work.
\n\nPersonally, i preferred pytorch.
\n\nAs for caffe, I used to train in caffe before. But now I, I do everything in pytorch and wrote a converter to convert model from pytorch to caffe.
\n\nActually i tried all frameworks before, including mxnet, or even code from scratch using c++ and cudnn. After trying all, I have to say that pytorch is the best to try experiments and unusual models.
", "rawMarkdown": "i started off with tensorflow. Now i need to get back to it because of work.\n\nPersonally, i preferred pytorch.\n\nAs for caffe, I used to train in caffe before. But now I, I do everything in pytorch and wrote a converter to convert model from pytorch to caffe.\n\nActually i tried all frameworks before, including mxnet, or even code from scratch using c++ and cudnn. After trying all, I have to say that pytorch is the best to try experiments and unusual models.", "votes": null }, { "id": "315837", "postDate": "04/17/2018 16:45:08", "content": "Yes, deployment is what happens after models are trained. And yes, better use the same framework. But when you serve models you don't need all the bells and whistles you need for training. You need a lightweight, and scalable, runtime. Lightweight is the key word here.
", "rawMarkdown": "Yes, deployment is what happens after models are trained. And yes, better use the same framework. But when you serve models you don't need all the bells and whistles you need for training. You need a lightweight, and scalable, runtime. Lightweight is the key word here.", "votes": null }, { "id": "316029", "postDate": "04/18/2018 03:25:02", "content": "Just an FYI, in tensorflow you can wrap a session so that it opens up a nice CLI every time a run is called on the session where you can inspect the tensors. Something like this:
from tensorflow.python import debug as tf_debug\nsess = tf_debug.LocalCLIDebugWrapperSession(sess) \nThanks. This is useful. I am going to try it in future
", "rawMarkdown": "Thanks. This is useful. I am going to try it in future", "votes": null }, { "id": "316056", "postDate": "04/18/2018 05:28:33", "content": "Hi Heng, Could you also release the corrected dataset \"fixed the training mask\" for us that we can reproduce your result easily? I mean train from scratch without using your initial checkpoint file? Thanks
", "rawMarkdown": "Hi Heng, Could you also release the corrected dataset \"fixed the training mask\" for us that we can reproduce your result easily? I mean train from scratch without using your initial checkpoint file? Thanks", "votes": null }, { "id": "316435", "postDate": "04/19/2018 03:06:23", "content": "Give me some time to organise the data and training log and post them. But from my experiments, i think it is not so important. Becuse the ensemble will average out the results
", "rawMarkdown": "Give me some time to organise the data and training log and post them. But from my experiments, i think it is not so important. Becuse the ensemble will average out the results", "votes": null }, { "id": "316967", "postDate": "04/20/2018 09:28:13", "content": "Blimey! I just watched this video from the TF Dev Summit on Visual Debugging using Tensorboard
\n\n\n\nForget the boring pre-able by the lady, skip to 6:30 to see the demo from the guy who wrote it.
\n\nIt has certainly convinced me that debugging TF is going to be a lot easier. In the demo he sets a break point to look for NaN and it halts and takes you to the code where there errors occur. Easy fix: log(0) error.
\n\nAlthough he doesn't give a demo of stepping through the code, I noticed that there was a button in the bottom left corner of the screen at 14:17 marked 'STEP' so I am guessing it does.
", "rawMarkdown": "Blimey! I just watched this video from the TF Dev Summit on Visual Debugging using Tensorboard\n\n[video here][1]\n\n\n [1]: https://www.youtube.com/watch?v=XcHWLsVmHvk\n\nForget the boring pre-able by the lady, skip to 6:30 to see the demo from the guy who wrote it.\n\nIt has certainly convinced me that debugging TF is going to be a lot easier. In the demo he sets a break point to look for NaN and it halts and takes you to the code where there errors occur. Easy fix: log(0) error.\n\nAlthough he doesn't give a demo of stepping through the code, I noticed that there was a button in the bottom left corner of the screen at 14:17 marked 'STEP' so I am guessing it does.", "votes": null } ], "comments": [ { "id": 315510, "author_name": "cpmpml", "author_url": "", "post_date": "04/17/2018 07:42:24", "content": "Thanks for sharing. This is interesting enough for me to really learn Pytorch!
", "votes": null, "replies": [ { "id": 315623, "author_name": "jinkos", "author_url": "", "post_date": "04/17/2018 11:15:54", "content": "Yes, same here.
\n\n@heng could be single handedly responsible for PyTorch becoming the new Kaggle standard!
", "votes": null, "replies": [] }, { "id": 315695, "author_name": "sasrdw", "author_url": "", "post_date": "04/17/2018 13:42:55", "content": "Definitely worth a look. Intriguing thing about Pytorch is it's immediate python execution versus the graph definition then session execution in tensorflow. One would think this would give tensorflow an edge in speed, but I'm not sure that is true.
", "votes": null, "replies": [] }, { "id": 315701, "author_name": "cpmpml", "author_url": "", "post_date": "04/17/2018 13:54:23", "content": "In another competition(CDiscount?) someone reported that Pytorch is faster than Keras. That's anecdotal evidence, not enough to draw a definitive conclusion, but interesting still.
", "votes": null, "replies": [] }, { "id": 315702, "author_name": "jinkos", "author_url": "", "post_date": "04/17/2018 13:55:51", "content": "TF has the edge when you are distributing the model between 6 different machines each with 8 GPUs. PyT can't do that. The overhead for TF being able to do distributed learning is that it is slow for single machines. And that is making PyT better than Keras and TF for Kaggle competitions.
\n\nTF has a new 'Eager Execution Mode' that enables it to do immediate execution and is therefore easier to debug. I haven't used it but I am guessing it will be even slower.
\n\nAbout the only advantage that TF has (over PyT) is the fact that it has lots of models already written for it. But who needs extra models when you have @heng to write them for you?!
", "votes": null, "replies": [] } ] }, { "id": 315565, "author_name": "soonhwankwon", "author_url": "", "post_date": "04/17/2018 09:24:49", "content": "Thank you for great kernel and I've learned a lot from your previous posts!
", "votes": null, "replies": [] }, { "id": 315631, "author_name": "john1231983", "author_url": "", "post_date": "04/17/2018 11:28:36", "content": "Thanks a lot. Why did not use mosaic image? Could you also provide train and val list file?
", "votes": null, "replies": [ { "id": 315635, "author_name": "hengck23", "author_url": "", "post_date": "04/17/2018 11:33:52", "content": "mosaic is for training. later i hand fixed the training mask and has no time to regenerated mosaic for the hand fixed mask. so the model was fine tuned on hand fixed mask.
\n\nfor testing, i use original images. this is because, i think the ground truth remove annotations near the edge of the images.
", "votes": null, "replies": [] }, { "id": 315640, "author_name": "cpmpml", "author_url": "", "post_date": "04/17/2018 11:47:49", "content": "Heng, We found mosaic to help on test prediction.
", "votes": null, "replies": [] }, { "id": 315648, "author_name": "john1231983", "author_url": "", "post_date": "04/17/2018 12:01:16", "content": "And have you use stage1 mask for training or just original training set? I do not think that manually correct mask improve performance because you can think it is noisy label
", "votes": null, "replies": [] }, { "id": 315697, "author_name": "cpmpml", "author_url": "", "post_date": "04/17/2018 13:48:30", "content": "We have used the training masks from https://github.com/lopuhin/kaggle-dsbowl-2018-dataset-fixes and the stage1 test labels as is.
", "votes": null, "replies": [] }, { "id": 316056, "author_name": "john1231983", "author_url": "", "post_date": "04/18/2018 05:28:33", "content": "Hi Heng, Could you also release the corrected dataset \"fixed the training mask\" for us that we can reproduce your result easily? I mean train from scratch without using your initial checkpoint file? Thanks
", "votes": null, "replies": [] }, { "id": 316435, "author_name": "hengck23", "author_url": "", "post_date": "04/19/2018 03:06:23", "content": "Give me some time to organise the data and training log and post them. But from my experiments, i think it is not so important. Becuse the ensemble will average out the results
", "votes": null, "replies": [] } ] }, { "id": 315703, "author_name": "hengck23", "author_url": "", "post_date": "04/17/2018 13:56:03", "content": "i feel that advantage of pytorch is debugging. Maybe there is not so much difference in speed.
\n\nin pytorch, you can set breakpoint and inspect tensor values in gpu memory. pytorch tensor is just like np array, very easy to code. Also, the data loader in pytorch is more user friendly.
\n\nTensorflow strength is probably commercial application deployment.
", "votes": null, "replies": [ { "id": 315705, "author_name": "cpmpml", "author_url": "", "post_date": "04/17/2018 14:01:12", "content": "Right, Pytorch flexibility makes it appealing for research purposes, but it is definitely not to be used as a deployment platform.
", "votes": null, "replies": [] }, { "id": 315727, "author_name": "jinkos", "author_url": "", "post_date": "04/17/2018 14:23:49", "content": "I'd rather deploy on something that is properly debugged ;)
\n\nSeriously, though, I don't have much deployment experience. What would be your biggest worry about using PyTorch in deployment. Is distributed learning that useful for most commercial applications? What else would you be worried about. I'd be interested to know.
", "votes": null, "replies": [] }, { "id": 315744, "author_name": "cpmpml", "author_url": "", "post_date": "04/17/2018 14:37:27", "content": "Even Facebooks says Pytorch is not for deployment. They use Caffe2 for that.
", "votes": null, "replies": [] }, { "id": 315756, "author_name": "jinkos", "author_url": "", "post_date": "04/17/2018 14:51:35", "content": "Yes, I have read that that is the conventional wisdom, but can you explain to someone who has not worked in commercial ML development (me) what a deployable system can do that I un-deployable one cannot?
\n\nAt what point do I go \"Dang, I wish I had chosen TensorFlow for my new startup\"?
", "votes": null, "replies": [] }, { "id": 315762, "author_name": "cpmpml", "author_url": "", "post_date": "04/17/2018 15:02:01", "content": "You need a way to serialize models so that they can be transported where you need the predictions. You then need a small footprint yet scalable prediction layer to be able to read these serialized models and make predictions at scale. You need to be able to version model versions. There is a lot more, and going further I'd be forced to start making publicity for what we do. If you stick to open source then there is little effort aiming at supporting the deployment needs (often called serving in deep learning context). Tensorflow Serving is a notable exception: https://www.tensorflow.org/serving/.
\n\nFacebook teamed with Microsoft Amazon, and others to establish ONNX as a way to exchange deep learning models. This could become the basis for a generic serving layer: https://onnx.ai/\nFacebook also tries to bridge the gap between Pytorch and Caffe2 as the latter is designed for deployment, see https://medium.com/@Synced/caffe2-merges-with-pytorch-a89c70ad9eb7
", "votes": null, "replies": [] }, { "id": 315763, "author_name": "hengck23", "author_url": "", "post_date": "04/17/2018 15:06:54", "content": "there are many kind of commercial deployment. I mainly deal with computer vision, image recognition on embedded system like on-board system-on-chip, especially ARM processor.
\n\nAs such, the first consideration is cost. Low cost means low cpu/gpu power, low memory and low power. At such, nvidia mobile gpu like TX2 is usually not an option (too expensive).
\n\nThe deep learning framework we decide to use usually depends on the support given by the chip. Most vendors only support caffe (not caffe2) and tensorflow model.
\n\nIf you are talking about mobile app development or server app deployment, it would be a different story.
", "votes": null, "replies": [] }, { "id": 315768, "author_name": "cpmpml", "author_url": "", "post_date": "04/17/2018 15:11:32", "content": "I also have testimony from people that tried both Pytorch and TF for production environments. I'll ask them if I can post their testimony here.
", "votes": null, "replies": [] }, { "id": 315828, "author_name": "jinkos", "author_url": "", "post_date": "04/17/2018 16:34:15", "content": "\"I see\", said the blind man.
\n\nSo, by deployment, you are talking about AFTER the model has been created and trained? Though it is obviously more efficient to train a model in the same framework in which it is deployed.
\n\nSorry if I am being slow, but my only experience in DL is with Kaggle.
\n\nAnd I am guessing that, as Pros, you guys are bi-lingual in PyT and (TF or caffe).
", "votes": null, "replies": [] }, { "id": 315834, "author_name": "hengck23", "author_url": "", "post_date": "04/17/2018 16:43:36", "content": "i started off with tensorflow. Now i need to get back to it because of work.
\n\nPersonally, i preferred pytorch.
\n\nAs for caffe, I used to train in caffe before. But now I, I do everything in pytorch and wrote a converter to convert model from pytorch to caffe.
\n\nActually i tried all frameworks before, including mxnet, or even code from scratch using c++ and cudnn. After trying all, I have to say that pytorch is the best to try experiments and unusual models.
", "votes": null, "replies": [] }, { "id": 315837, "author_name": "cpmpml", "author_url": "", "post_date": "04/17/2018 16:45:08", "content": "Yes, deployment is what happens after models are trained. And yes, better use the same framework. But when you serve models you don't need all the bells and whistles you need for training. You need a lightweight, and scalable, runtime. Lightweight is the key word here.
", "votes": null, "replies": [] }, { "id": 316029, "author_name": "jfaath", "author_url": "", "post_date": "04/18/2018 03:25:02", "content": "Just an FYI, in tensorflow you can wrap a session so that it opens up a nice CLI every time a run is called on the session where you can inspect the tensors. Something like this:
from tensorflow.python import debug as tf_debug\nsess = tf_debug.LocalCLIDebugWrapperSession(sess) \nThanks. This is useful. I am going to try it in future
", "votes": null, "replies": [] }, { "id": 316967, "author_name": "jinkos", "author_url": "", "post_date": "04/20/2018 09:28:13", "content": "Blimey! I just watched this video from the TF Dev Summit on Visual Debugging using Tensorboard
\n\n\n\nForget the boring pre-able by the lady, skip to 6:30 to see the demo from the guy who wrote it.
\n\nIt has certainly convinced me that debugging TF is going to be a lot easier. In the demo he sets a break point to look for NaN and it halts and takes you to the code where there errors occur. Easy fix: log(0) error.
\n\nAlthough he doesn't give a demo of stepping through the code, I noticed that there was a button in the bottom left corner of the screen at 14:17 marked 'STEP' so I am guessing it does.
", "votes": null, "replies": [] } ] }, { "id": 315794, "author_name": "joek47", "author_url": "", "post_date": "04/17/2018 15:59:36", "content": "Hi Heng, thank you for your amazing posts! If you're open, can I be your apprentice? ;)
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "315500": "my submission code (training, inference, ensemble, model) is at stage2: https://www.kaggle.com/c/data-science-bowl-2018/discussion/53720 or here https://drive.google.com/drive/folders/1QeMFRfiW95rudGLwKYNy-weLpP-5L6-S\n\nAt the submission :\n\nfirst submission(set FP reject threshold = 4)\n\n - public LB = 0.505\n\n - private LB = 0.574\n\n\nsecond submission(set FP reject threshold = 0.5)\n\n - public LB = 0.501\n\n - private LB = 0.505\n\n\n\nPost submission:\n\n - private LB (2800 gray images only)= 0.506 (set FP reject threshold = 4)\n\n - private LB (2800 gray images only)= 0.521 (set FP reject threshold = 6)\n\n - private LB (2800 gray images only)= 0.539 (set FP reject threshold = 8)\n\n - private LB (2800 gray images only)= 0.535 (set FP reject threshold =10)\n\ncurrent best post submission results is \"privateLB-0.607__submission-stage2-LB0.505-ensemble-post-s8.0-gray-old_color.csv\", using threshold=8 for gray images\n\n ![enter image description here][1]\n\n\nI guessed that is the difference between \"black box\" and \"gray box\" testing. When you get to see your results, you can think of many tricks or parameter adjustment or pre/post processing to improve results.\n\n---\n\nto set reject threshold, please refer to s value in ensemble of my code\n\n s = center[binary].sum()/binary.sum()*num_members\n if s<6: continue\n\n\n [1]: https://kaggle2.blob.core.windows.net/forum-message-attachments/315500/9208/post-submission.png", "315510": "Thanks for sharing. This is interesting enough for me to really learn Pytorch!", "315565": "Thank you for great kernel and I've learned a lot from your previous posts!", "315623": "Yes, same here. \n\n@heng could be single handedly responsible for PyTorch becoming the new Kaggle standard!", "315631": "Thanks a lot. Why did not use mosaic image? Could you also provide train and val list file?", "315635": "mosaic is for training. later i hand fixed the training mask and has no time to regenerated mosaic for the hand fixed mask. so the model was fine tuned on hand fixed mask.\n\nfor testing, i use original images. this is because, i think the ground truth remove annotations near the edge of the images.", "315640": "Heng, We found mosaic to help on test prediction.", "315648": "And have you use stage1 mask for training or just original training set? I do not think that manually correct mask improve performance because you can think it is noisy label", "315695": "Definitely worth a look. Intriguing thing about Pytorch is it's immediate python execution versus the graph definition then session execution in tensorflow. One would think this would give tensorflow an edge in speed, but I'm not sure that is true.", "315697": "We have used the training masks from https://github.com/lopuhin/kaggle-dsbowl-2018-dataset-fixes and the stage1 test labels as is.", "315701": "In another competition(CDiscount?) someone reported that Pytorch is faster than Keras. That's anecdotal evidence, not enough to draw a definitive conclusion, but interesting still.", "315702": "TF has the edge when you are distributing the model between 6 different machines each with 8 GPUs. PyT can't do that. The overhead for TF being able to do distributed learning is that it is slow for single machines. And that is making PyT better than Keras and TF for Kaggle competitions. \n\nTF has a new 'Eager Execution Mode' that enables it to do immediate execution and is therefore easier to debug. I haven't used it but I am guessing it will be even slower.\n\nAbout the only advantage that TF has (over PyT) is the fact that it has lots of models already written for it. But who needs extra models when you have @heng to write them for you?!", "315703": "i feel that advantage of pytorch is debugging. Maybe there is not so much difference in speed.\n\nin pytorch, you can set breakpoint and inspect tensor values in gpu memory. pytorch tensor is just like np array, very easy to code. Also, the data loader in pytorch is more user friendly.\n\nTensorflow strength is probably commercial application deployment.", "315705": "Right, Pytorch flexibility makes it appealing for research purposes, but it is definitely not to be used as a deployment platform.", "315727": "I'd rather deploy on something that is properly debugged ;)\n\nSeriously, though, I don't have much deployment experience. What would be your biggest worry about using PyTorch in deployment. Is distributed learning that useful for most commercial applications? What else would you be worried about. I'd be interested to know.", "315744": "Even Facebooks says Pytorch is not for deployment. They use Caffe2 for that.", "315756": "Yes, I have read that that is the conventional wisdom, but can you explain to someone who has not worked in commercial ML development (me) what a deployable system can do that I un-deployable one cannot?\n\nAt what point do I go \"Dang, I wish I had chosen TensorFlow for my new startup\"?", "315762": "You need a way to serialize models so that they can be transported where you need the predictions. You then need a small footprint yet scalable prediction layer to be able to read these serialized models and make predictions at scale. You need to be able to version model versions. There is a lot more, and going further I'd be forced to start making publicity for what we do. If you stick to open source then there is little effort aiming at supporting the deployment needs (often called *serving* in deep learning context). Tensorflow Serving is a notable exception: https://www.tensorflow.org/serving/.\n\nFacebook teamed with Microsoft Amazon, and others to establish ONNX as a way to exchange deep learning models. This could become the basis for a generic serving layer: https://onnx.ai/\nFacebook also tries to bridge the gap between Pytorch and Caffe2 as the latter is designed for deployment, see https://medium.com/@Synced/caffe2-merges-with-pytorch-a89c70ad9eb7", "315763": "there are many kind of commercial deployment. I mainly deal with computer vision, image recognition on embedded system like on-board system-on-chip, especially ARM processor.\n\nAs such, the first consideration is cost. Low cost means low cpu/gpu power, low memory and low power. At such, nvidia mobile gpu like TX2 is usually not an option (too expensive).\n\nThe deep learning framework we decide to use usually depends on the support given by the chip. Most vendors only support caffe (not caffe2) and tensorflow model.\n\nIf you are talking about mobile app development or server app deployment, it would be a different story.", "315768": "I also have testimony from people that tried both Pytorch and TF for production environments. I'll ask them if I can post their testimony here.", "315794": "Hi Heng, thank you for your amazing posts! If you're open, can I be your apprentice? ;)", "315828": "\"I see\", said the blind man. \n\nSo, by deployment, you are talking about AFTER the model has been created and trained? Though it is obviously more efficient to train a model in the same framework in which it is deployed.\n\nSorry if I am being slow, but my only experience in DL is with Kaggle.\n\nAnd I am guessing that, as Pros, you guys are bi-lingual in PyT and (TF or caffe).", "315834": "i started off with tensorflow. Now i need to get back to it because of work.\n\nPersonally, i preferred pytorch.\n\nAs for caffe, I used to train in caffe before. But now I, I do everything in pytorch and wrote a converter to convert model from pytorch to caffe.\n\nActually i tried all frameworks before, including mxnet, or even code from scratch using c++ and cudnn. After trying all, I have to say that pytorch is the best to try experiments and unusual models.", "315837": "Yes, deployment is what happens after models are trained. And yes, better use the same framework. But when you serve models you don't need all the bells and whistles you need for training. You need a lightweight, and scalable, runtime. Lightweight is the key word here.", "316029": "Just an FYI, in tensorflow you can wrap a session so that it opens up a nice CLI every time a ```run``` is called on the session where you can inspect the tensors. Something like this:\n\n from tensorflow.python import debug as tf_debug\n sess = tf_debug.LocalCLIDebugWrapperSession(sess)", "316053": "Thanks. This is useful. I am going to try it in future", "316056": "Hi Heng, Could you also release the corrected dataset \"fixed the training mask\" for us that we can reproduce your result easily? I mean train from scratch without using your initial checkpoint file? Thanks", "316435": "Give me some time to organise the data and training log and post them. But from my experiments, i think it is not so important. Becuse the ensemble will average out the results", "316967": "Blimey! I just watched this video from the TF Dev Summit on Visual Debugging using Tensorboard\n\n[video here][1]\n\n\n [1]: https://www.youtube.com/watch?v=XcHWLsVmHvk\n\nForget the boring pre-able by the lady, skip to 6:30 to see the demo from the guy who wrote it.\n\nIt has certainly convinced me that debugging TF is going to be a lot easier. In the demo he sets a break point to look for NaN and it halts and takes you to the code where there errors occur. Easy fix: log(0) error.\n\nAlthough he doesn't give a demo of stepping through the code, I noticed that there was a button in the bottom left corner of the screen at 14:17 marked 'STEP' so I am guessing it does." }, "source": "meta" }