Is it possible to win using only kernels? Has this ever happened, that somebody won using only kernels?\nAs for this competition, I think that it is very difficult to use only kernels as even if the data are split (e.g. by types) and memory requirements are reduced by the script (mentioned in almost every kernel) 16 GB of memory is just not enough.\nBut I am a novice here, so I was just wondering if it ever happened.
", "messages": [ { "id": "570727", "postDate": "07/08/2019 17:49:09", "content": "Is it possible to win using only kernels? Has this ever happened, that somebody won using only kernels?\nAs for this competition, I think that it is very difficult to use only kernels as even if the data are split (e.g. by types) and memory requirements are reduced by the script (mentioned in almost every kernel) 16 GB of memory is just not enough.\nBut I am a novice here, so I was just wondering if it ever happened.
", "rawMarkdown": "Is it possible to win using only kernels? Has this ever happened, that somebody won using only kernels?\nAs for this competition, I think that it is very difficult to use only kernels as even if the data are split (e.g. by types) and memory requirements are reduced by the script (mentioned in almost every kernel) 16 GB of memory is just not enough.\nBut I am a novice here, so I was just wondering if it ever happened.", "votes": null }, { "id": "570816", "postDate": "07/08/2019 19:47:02", "content": "I think that you can say that you won the competition if you received at least gold medal, so no. However, sometimes there are competitions in which you can get a bronze for solution from public Kernel.
", "rawMarkdown": "I think that you can say that you won the competition if you received at least gold medal, so no. However, sometimes there are competitions in which you can get a bronze for solution from public Kernel.", "votes": null }, { "id": "570827", "postDate": "07/08/2019 20:07:06", "content": "It happens all the time in kernels only competitions ;)
", "rawMarkdown": "It happens all the time in kernels only competitions ;)", "votes": null }, { "id": "570857", "postDate": "07/08/2019 21:02:29", "content": "16 GB is likely a real issue - but I don't know how Linux works with regards to memory compression and using hard drive for memory cache. My 64GB Windows RAM is at max running a lightgbm script as I type this with a lot of SSD disk cache activity going on - perhaps someone more knowledgeable about Linux can enlighten us with regards to how hard of a roof is the 16GB memory.
", "rawMarkdown": "16 GB is likely a real issue - but I don't know how Linux works with regards to memory compression and using hard drive for memory cache. My 64GB Windows RAM is at max running a lightgbm script as I type this with a lot of SSD disk cache activity going on - perhaps someone more knowledgeable about Linux can enlighten us with regards to how hard of a roof is the 16GB memory.", "votes": null }, { "id": "572867", "postDate": "07/11/2019 13:59:01", "content": "The memory shouldn't be an issue for doing ML if you use a generator. That is, you only store the full dataset in a compact form without feature, and create features on the fly only by minibatch as you train.
", "rawMarkdown": "The memory shouldn't be an issue for doing ML if you use a generator. That is, you only store the full dataset in a compact form without feature, and create features on the fly only by minibatch as you train.", "votes": null }, { "id": "573178", "postDate": "07/11/2019 22:12:11", "content": "I am guessing 🥉 is possible if you figure out the main tricks early on and split your training into chunks. One month of kernels is a lot of cpu/gpu.
", "rawMarkdown": "I am guessing 🥉 is possible if you figure out the main tricks early on and split your training into chunks. One month of kernels is a lot of cpu/gpu.", "votes": null }, { "id": "573357", "postDate": "07/12/2019 06:55:24", "content": "Thanks for disclosing you're using deep learning ;)
", "rawMarkdown": "Thanks for disclosing you're using deep learning ;)", "votes": null }, { "id": "573530", "postDate": "07/12/2019 11:24:29", "content": "Thanks more generally for the generation idea. Was getting stuck with RAM for high dimensional inputs even using local PC and this opens up some options.
", "rawMarkdown": "Thanks more generally for the generation idea. Was getting stuck with RAM for high dimensional inputs even using local PC and this opens up some options.", "votes": null } ], "comments": [ { "id": 570816, "author_name": "sorrow321", "author_url": "", "post_date": "07/08/2019 19:47:02", "content": "I think that you can say that you won the competition if you received at least gold medal, so no. However, sometimes there are competitions in which you can get a bronze for solution from public Kernel.
", "votes": null, "replies": [] }, { "id": 570827, "author_name": "scirpus", "author_url": "", "post_date": "07/08/2019 20:07:06", "content": "It happens all the time in kernels only competitions ;)
", "votes": null, "replies": [] }, { "id": 570857, "author_name": "pcjimmmy", "author_url": "", "post_date": "07/08/2019 21:02:29", "content": "16 GB is likely a real issue - but I don't know how Linux works with regards to memory compression and using hard drive for memory cache. My 64GB Windows RAM is at max running a lightgbm script as I type this with a lot of SSD disk cache activity going on - perhaps someone more knowledgeable about Linux can enlighten us with regards to how hard of a roof is the 16GB memory.
", "votes": null, "replies": [] }, { "id": 572867, "author_name": "cedben", "author_url": "", "post_date": "07/11/2019 13:59:01", "content": "The memory shouldn't be an issue for doing ML if you use a generator. That is, you only store the full dataset in a compact form without feature, and create features on the fly only by minibatch as you train.
", "votes": null, "replies": [ { "id": 573357, "author_name": "cpmpml", "author_url": "", "post_date": "07/12/2019 06:55:24", "content": "Thanks for disclosing you're using deep learning ;)
", "votes": null, "replies": [] }, { "id": 573530, "author_name": "jameschapman19", "author_url": "", "post_date": "07/12/2019 11:24:29", "content": "Thanks more generally for the generation idea. Was getting stuck with RAM for high dimensional inputs even using local PC and this opens up some options.
", "votes": null, "replies": [] } ] }, { "id": 573178, "author_name": "stocks", "author_url": "", "post_date": "07/11/2019 22:12:11", "content": "I am guessing 🥉 is possible if you figure out the main tricks early on and split your training into chunks. One month of kernels is a lot of cpu/gpu.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "570727": "Is it possible to win using only kernels? Has this ever happened, that somebody won using only kernels?\nAs for this competition, I think that it is very difficult to use only kernels as even if the data are split (e.g. by types) and memory requirements are reduced by the script (mentioned in almost every kernel) 16 GB of memory is just not enough.\nBut I am a novice here, so I was just wondering if it ever happened.", "570816": "I think that you can say that you won the competition if you received at least gold medal, so no. However, sometimes there are competitions in which you can get a bronze for solution from public Kernel.", "570827": "It happens all the time in kernels only competitions ;)", "570857": "16 GB is likely a real issue - but I don't know how Linux works with regards to memory compression and using hard drive for memory cache. My 64GB Windows RAM is at max running a lightgbm script as I type this with a lot of SSD disk cache activity going on - perhaps someone more knowledgeable about Linux can enlighten us with regards to how hard of a roof is the 16GB memory.", "572867": "The memory shouldn't be an issue for doing ML if you use a generator. That is, you only store the full dataset in a compact form without feature, and create features on the fly only by minibatch as you train.", "573178": "I am guessing 🥉 is possible if you figure out the main tricks early on and split your training into chunks. One month of kernels is a lot of cpu/gpu.", "573357": "Thanks for disclosing you're using deep learning ;)", "573530": "Thanks more generally for the generation idea. Was getting stuck with RAM for high dimensional inputs even using local PC and this opens up some options." }, "source": "meta" }