I've worked super hard on yolov4 pseudolabelling with darknet, and it all works perfectly in Google Colab. However, when I execute the same code in Kaggle, darknet doesn't even build, and I don't understand the differences between Colab and Kaggle (this was run in Kaggle with internet on). I have the following code:\n\n!nvcc --version\n!git clone https://github.com/AlexeyAB/darknet\n!sed -i 's/OPENCV=0/OPENCV=1/' Makefile\n!sed -i 's/GPU=0/GPU=1/' Makefile\n!sed -i 's/CUDNN=0/CUDNN=1/' Makefile\n!sed -i 's/CUDNN_HALF=0/CUDNN_HALF=1/' Makefile\n!make\n
I get the following error: \n\ncollect2: error: ld returned 1 exit status\nMakefile:168: recipe for target 'darknet' failed\nmake: *** [darknet] Error 1\n
Thanks so much.
", "messages": [ { "id": 1595272, "postDate": "2021-11-25T14:50:36.050Z", "content": "Hello, after following many solutions I managed to run yolov4 (compiled with OPENCV=1) without libcuda.so nor
\nlibopencv_highgui.so errors. Here I created a notebook with all the steps. Hope it helps.
\nhttps://www.kaggle.com/gimarcecaml/build-darknet-yolo4/
I successfully ran yolov4 to train on my own dataset with your notebook! Your notebook is really helpful! You are an absolute lifesaver for me. Judging from your notebook, you know a lot about linux. Even though it's been two years, your valuable experience can still help me.
", "rawMarkdown": "I successfully ran yolov4 to train on my own dataset with your notebook! Your notebook is really helpful! You are an absolute lifesaver for me. Judging from your notebook, you know a lot about linux. Even though it's been two years, your valuable experience can still help me." } ] }, { "id": 953920, "postDate": "2020-08-01T08:38:49.207Z", "content": "Hi @stanleyjzheng , you may want to check out the notebook that I just shared, it should be able to let you run darknet without problems on kaggle. Hope it helps.\nhttps://www.kaggle.com/markpeng/darknet-gpu-on-kaggle/
", "rawMarkdown": "Hi @stanleyjzheng , you may want to check out the notebook that I just shared, it should be able to let you run darknet without problems on kaggle. Hope it helps.\nhttps://www.kaggle.com/markpeng/darknet-gpu-on-kaggle/", "votes": 4, "replies": [ { "id": 954144, "postDate": "2020-08-01T12:29:00.183Z", "content": "You, sir, are an absolute lifesaver. Thanks so so so so much, I really appreciate it!
", "rawMarkdown": "You, sir, are an absolute lifesaver. Thanks so so so so much, I really appreciate it!", "votes": 1 }, { "id": 954249, "postDate": "2020-08-01T14:47:52.083Z", "rawMarkdown": "", "isDeleted": true }, { "id": 954254, "postDate": "2020-08-01T14:57:31.323Z", "content": "@stanleyjzheng I think it is possible if we compile darknet using the same version of OpenCV installed on Kaggle Notebook. My machine is installed with OpenCV branch 4.0.1, which looks different from the Kaggle one, therefore I disabled the OPENCV=1 flag because it won't work in the notebook.
But I think without OPENCV=1 you should still be able to train the model with mosaic=1 in the cfg file. I was training the models locally with Mosaic augmentation without any problem (OPENCV=0), but data augmentation could be slower.
@stanleyjzheng It could be that I was actually using OPENCV=1 compiled to train locally, will double check. But even if I could be wrong in the previous comment, it should be possible to recompile darknet with a compatible OpenCV version.
@stanleyjzheng A follow-up update, you were right, we need to enable for OPENCV=1 in the Makefile to use mosaic=1 mode.\n\nmosaic=1 - compile Darknet with OpenCV for using mosaic=1\n\nKaggle notebook dockerfile seems missing libopencv-highgui-dev, it needs to be installed through apt-get install -y libopencv-dev, which requires to enable Internet (not allowed in notebook competition).
Another direction to try is to prebuilt libopencv_highgui.so.4.3.0 locally and push it as a dataset (like what I did for darknet).
@stanleyjzheng I can feel the pain of you. I had tried a lot but couldn't able to use darknet command in Kaggle. You can check this thread for the error your are facing- darknet issue in kaggle
", "rawMarkdown": "@stanleyjzheng I can feel the pain of you. I had tried a lot but couldn't able to use darknet command in Kaggle. You can check this thread for the error your are facing- [darknet issue in kaggle](https://github.com/AlexeyAB/darknet/issues/6163)", "votes": 1, "replies": [ { "id": 953771, "postDate": "2020-08-01T04:56:29.827Z", "content": "Thanks a ton man, I've spent like 6 hours trying to get it to work, and I have a lot of bash experience, couldn't get it to work for the life of me. It's a libcuda.so issue as he mentioned, quite unfortunate. I guess that just confirms it, there's no way to use darknet due to a crappy driver by Kaggle. I'm still quite confused why it works in Colab but not Kaggle, they're both google owned.
", "rawMarkdown": "Thanks a ton man, I've spent like 6 hours trying to get it to work, and I have a lot of bash experience, couldn't get it to work for the life of me. It's a libcuda.so issue as he mentioned, quite unfortunate. I guess that just confirms it, there's no way to use darknet due to a crappy driver by Kaggle. I'm still quite confused why it works in Colab but not Kaggle, they're both google owned. ", "votes": 1 }, { "id": 953776, "postDate": "2020-08-01T05:04:16.423Z", "content": "I'm sorry to hear you're having technical difficulties at this point in the competition. Just my thought: is it possible to build in an online notebook and then add it to the offline notebook as a dataset? I see in that github thread they say it won't work because it's read-only but can't you just copy it into the working directory and modify the permissions if needed? Sorry if this is something you've already tried
", "rawMarkdown": "I'm sorry to hear you're having technical difficulties at this point in the competition. Just my thought: is it possible to build in an online notebook and then add it to the offline notebook as a dataset? I see in that github thread they say it won't work because it's read-only but can't you just copy it into the working directory and modify the permissions if needed? Sorry if this is something you've already tried", "votes": 1 }, { "id": 954243, "postDate": "2020-08-01T14:44:13.973Z", "content": "Yea so the issue then is that it doesn't compile properly with the makefile. However, Mark Peng provided a brilliant solution, he compiled it on his machine and uploaded it to a dataset to use in Kaggle. Thanks for your reply Matthew, I really appreciate your activity in the discussions throughout this competition!
", "rawMarkdown": "Yea so the issue then is that it doesn't compile properly with the makefile. However, Mark Peng provided a brilliant solution, he compiled it on his machine and uploaded it to a dataset to use in Kaggle. Thanks for your reply Matthew, I really appreciate your activity in the discussions throughout this competition!", "votes": 1 }, { "id": 954625, "postDate": "2020-08-01T23:25:48.017Z", "content": "Thanks for the kind words! That's awesome of Mark for creating and sharing that notebook. Best of luck in the final standings!
", "rawMarkdown": "Thanks for the kind words! That's awesome of Mark for creating and sharing that notebook. Best of luck in the final standings!", "votes": 2 } ] }, { "id": 953399, "postDate": "2020-07-31T18:53:43.157Z", "content": "If I understood correctly, Kaggle has a wrong file to compile: see here
", "rawMarkdown": "If I understood correctly, Kaggle has a wrong file to compile: see [here](https://github.com/ultralytics/yolov5/issues/6#issuecomment-659693164)", "votes": 1, "replies": [ { "id": 953417, "postDate": "2020-07-31T19:19:02.880Z", "rawMarkdown": "", "isDeleted": true } ] }, { "id": 953384, "postDate": "2020-07-31T18:35:03.113Z", "content": "I've worked super hard on yolov4 pseudolabelling with darknet, and it all works perfectly in Google Colab. However, when I execute the same code in Kaggle, darknet doesn't even build, and I don't understand the differences between Colab and Kaggle (this was run in Kaggle with internet on). I have the following code:\n\n!nvcc --version\n!git clone https://github.com/AlexeyAB/darknet\n!sed -i 's/OPENCV=0/OPENCV=1/' Makefile\n!sed -i 's/GPU=0/GPU=1/' Makefile\n!sed -i 's/CUDNN=0/CUDNN=1/' Makefile\n!sed -i 's/CUDNN_HALF=0/CUDNN_HALF=1/' Makefile\n!make\n
I get the following error: \n\ncollect2: error: ld returned 1 exit status\nMakefile:168: recipe for target 'darknet' failed\nmake: *** [darknet] Error 1\n
Thanks so much.
", "rawMarkdown": "I've worked super hard on yolov4 pseudolabelling with darknet, and it all works perfectly in Google Colab. However, when I execute the same code in Kaggle, darknet doesn't even build, and I don't understand the differences between Colab and Kaggle (this was run in Kaggle with internet on). I have the following code:\n```\n!nvcc --version\n!git clone https://github.com/AlexeyAB/darknet\n!sed -i 's/OPENCV=0/OPENCV=1/' Makefile\n!sed -i 's/GPU=0/GPU=1/' Makefile\n!sed -i 's/CUDNN=0/CUDNN=1/' Makefile\n!sed -i 's/CUDNN_HALF=0/CUDNN_HALF=1/' Makefile\n!make\n```\n\nI get the following error: \n```\ncollect2: error: ld returned 1 exit status\nMakefile:168: recipe for target 'darknet' failed\nmake: *** [darknet] Error 1\n```\n\nThanks so much. ", "votes": 2 } ], "comments": [ { "id": 1595272, "author_name": "Gino Marceca", "author_url": "", "post_date": "2021-11-25T14:50:36.050000", "content": "Hello, after following many solutions I managed to run yolov4 (compiled with OPENCV=1) without libcuda.so nor
\nlibopencv_highgui.so errors. Here I created a notebook with all the steps. Hope it helps.
\nhttps://www.kaggle.com/gimarcecaml/build-darknet-yolo4/
I successfully ran yolov4 to train on my own dataset with your notebook! Your notebook is really helpful! You are an absolute lifesaver for me. Judging from your notebook, you know a lot about linux. Even though it's been two years, your valuable experience can still help me.
", "votes": 0, "replies": [] } ] }, { "id": 953920, "author_name": "Mark Peng", "author_url": "", "post_date": "2020-08-01T08:38:49.207000", "content": "Hi @stanleyjzheng , you may want to check out the notebook that I just shared, it should be able to let you run darknet without problems on kaggle. Hope it helps.\nhttps://www.kaggle.com/markpeng/darknet-gpu-on-kaggle/
", "votes": 4, "replies": [ { "id": 954144, "author_name": "Stanley Zheng", "author_url": "", "post_date": "2020-08-01T12:29:00.183000", "content": "You, sir, are an absolute lifesaver. Thanks so so so so much, I really appreciate it!
", "votes": 1, "replies": [] }, { "id": 954249, "author_name": "", "author_url": "", "post_date": "2020-08-01T14:47:52.083000", "content": "", "votes": 0, "replies": [] }, { "id": 954254, "author_name": "Mark Peng", "author_url": "", "post_date": "2020-08-01T14:57:31.323000", "content": "@stanleyjzheng I think it is possible if we compile darknet using the same version of OpenCV installed on Kaggle Notebook. My machine is installed with OpenCV branch 4.0.1, which looks different from the Kaggle one, therefore I disabled the OPENCV=1 flag because it won't work in the notebook.
But I think without OPENCV=1 you should still be able to train the model with mosaic=1 in the cfg file. I was training the models locally with Mosaic augmentation without any problem (OPENCV=0), but data augmentation could be slower.
@stanleyjzheng It could be that I was actually using OPENCV=1 compiled to train locally, will double check. But even if I could be wrong in the previous comment, it should be possible to recompile darknet with a compatible OpenCV version.
@stanleyjzheng A follow-up update, you were right, we need to enable for OPENCV=1 in the Makefile to use mosaic=1 mode.\n\nmosaic=1 - compile Darknet with OpenCV for using mosaic=1\n\nKaggle notebook dockerfile seems missing libopencv-highgui-dev, it needs to be installed through apt-get install -y libopencv-dev, which requires to enable Internet (not allowed in notebook competition).
Another direction to try is to prebuilt libopencv_highgui.so.4.3.0 locally and push it as a dataset (like what I did for darknet).
@stanleyjzheng I can feel the pain of you. I had tried a lot but couldn't able to use darknet command in Kaggle. You can check this thread for the error your are facing- darknet issue in kaggle
", "votes": 1, "replies": [ { "id": 953771, "author_name": "Stanley Zheng", "author_url": "", "post_date": "2020-08-01T04:56:29.827000", "content": "Thanks a ton man, I've spent like 6 hours trying to get it to work, and I have a lot of bash experience, couldn't get it to work for the life of me. It's a libcuda.so issue as he mentioned, quite unfortunate. I guess that just confirms it, there's no way to use darknet due to a crappy driver by Kaggle. I'm still quite confused why it works in Colab but not Kaggle, they're both google owned.
", "votes": 1, "replies": [] }, { "id": 953776, "author_name": "Matt", "author_url": "", "post_date": "2020-08-01T05:04:16.423000", "content": "I'm sorry to hear you're having technical difficulties at this point in the competition. Just my thought: is it possible to build in an online notebook and then add it to the offline notebook as a dataset? I see in that github thread they say it won't work because it's read-only but can't you just copy it into the working directory and modify the permissions if needed? Sorry if this is something you've already tried
", "votes": 1, "replies": [] }, { "id": 954243, "author_name": "Stanley Zheng", "author_url": "", "post_date": "2020-08-01T14:44:13.973000", "content": "Yea so the issue then is that it doesn't compile properly with the makefile. However, Mark Peng provided a brilliant solution, he compiled it on his machine and uploaded it to a dataset to use in Kaggle. Thanks for your reply Matthew, I really appreciate your activity in the discussions throughout this competition!
", "votes": 1, "replies": [] }, { "id": 954625, "author_name": "Matt", "author_url": "", "post_date": "2020-08-01T23:25:48.017000", "content": "Thanks for the kind words! That's awesome of Mark for creating and sharing that notebook. Best of luck in the final standings!
", "votes": 2, "replies": [] } ] }, { "id": 953399, "author_name": "eagle4", "author_url": "", "post_date": "2020-07-31T18:53:43.157000", "content": "If I understood correctly, Kaggle has a wrong file to compile: see here
", "votes": 1, "replies": [ { "id": 953417, "author_name": "", "author_url": "", "post_date": "2020-07-31T19:19:02.880000", "content": "", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "1595272": "Hello, after following many solutions I managed to run yolov4 (compiled with `OPENCV=1`) without `libcuda.so` nor \n`libopencv_highgui.so` errors. Here I created a notebook with all the steps. Hope it helps.\nhttps://www.kaggle.com/gimarcecaml/build-darknet-yolo4/", "953920": "Hi @stanleyjzheng , you may want to check out the notebook that I just shared, it should be able to let you run darknet without problems on kaggle. Hope it helps.\nhttps://www.kaggle.com/markpeng/darknet-gpu-on-kaggle/", "953768": "@stanleyjzheng I can feel the pain of you. I had tried a lot but couldn't able to use darknet command in Kaggle. You can check this thread for the error your are facing- [darknet issue in kaggle](https://github.com/AlexeyAB/darknet/issues/6163)", "953399": "If I understood correctly, Kaggle has a wrong file to compile: see [here](https://github.com/ultralytics/yolov5/issues/6#issuecomment-659693164)", "953384": "I've worked super hard on yolov4 pseudolabelling with darknet, and it all works perfectly in Google Colab. However, when I execute the same code in Kaggle, darknet doesn't even build, and I don't understand the differences between Colab and Kaggle (this was run in Kaggle with internet on). I have the following code:\n```\n!nvcc --version\n!git clone https://github.com/AlexeyAB/darknet\n!sed -i 's/OPENCV=0/OPENCV=1/' Makefile\n!sed -i 's/GPU=0/GPU=1/' Makefile\n!sed -i 's/CUDNN=0/CUDNN=1/' Makefile\n!sed -i 's/CUDNN_HALF=0/CUDNN_HALF=1/' Makefile\n!make\n```\n\nI get the following error: \n```\ncollect2: error: ld returned 1 exit status\nMakefile:168: recipe for target 'darknet' failed\nmake: *** [darknet] Error 1\n```\n\nThanks so much. " } }