Looking to take on this as my first featured competition. I however have found that simply reading data takes a lot of time and worried training may take days. To be concise, how can I best reduce the time spent for instance on reading the data?\n I'm using R if it makes a difference. Thanks in advance.
", "messages": [ { "id": "436375", "postDate": "12/10/2018 07:46:26", "content": "Looking to take on this as my first featured competition. I however have found that simply reading data takes a lot of time and worried training may take days. To be concise, how can I best reduce the time spent for instance on reading the data?\n I'm using R if it makes a difference. Thanks in advance.
", "rawMarkdown": "Looking to take on this as my first featured competition. I however have found that simply reading data takes a lot of time and worried training may take days. To be concise, how can I best reduce the time spent for instance on reading the data?\n I'm using R if it makes a difference. Thanks in advance.", "votes": null }, { "id": "436493", "postDate": "12/10/2018 12:19:32", "content": "I use a kernel to load, cleaning, do feature engineering then save all of it to pkl or h5 files.\nThen load that kernel's output to do trianing. And it much faster and lighter.
", "rawMarkdown": "I use a kernel to load, cleaning, do feature engineering then save all of it to pkl or h5 files.\nThen load that kernel's output to do trianing. And it much faster and lighter.", "votes": null }, { "id": "436560", "postDate": "12/10/2018 14:21:13", "content": "Thanks, I'll look into that.
", "rawMarkdown": "Thanks, I'll look into that.", "votes": null }, { "id": "436563", "postDate": "12/10/2018 14:26:03", "content": "What R function are you using to read the data? Some read functions are faster than others. But I'd recommend that you use multiple kernels. Perhaps one for aggregating and cleaning data, one for hyperparameter optimization and/or training and one for the final stacking/blending.
", "rawMarkdown": "What R function are you using to read the data? Some read functions are faster than others. But I'd recommend that you use multiple kernels. Perhaps one for aggregating and cleaning data, one for hyperparameter optimization and/or training and one for the final stacking/blending.", "votes": null }, { "id": "436566", "postDate": "12/10/2018 14:27:39", "content": "Will look into this too. Thanks!
", "rawMarkdown": "Will look into this too. Thanks!", "votes": null }, { "id": "437614", "postDate": "12/12/2018 07:35:50", "content": "Is there a similar way to do that in Python? Most of the data compression/read-time improvements are being done by changing the data itself.
", "rawMarkdown": "Is there a similar way to do that in Python? Most of the data compression/read-time improvements are being done by changing the data itself.", "votes": null } ], "comments": [ { "id": 436493, "author_name": "qnkhuat", "author_url": "", "post_date": "12/10/2018 12:19:32", "content": "I use a kernel to load, cleaning, do feature engineering then save all of it to pkl or h5 files.\nThen load that kernel's output to do trianing. And it much faster and lighter.
", "votes": null, "replies": [ { "id": 436560, "author_name": "gonnel", "author_url": "", "post_date": "12/10/2018 14:21:13", "content": "Thanks, I'll look into that.
", "votes": null, "replies": [] } ] }, { "id": 436563, "author_name": "frjo0083", "author_url": "", "post_date": "12/10/2018 14:26:03", "content": "What R function are you using to read the data? Some read functions are faster than others. But I'd recommend that you use multiple kernels. Perhaps one for aggregating and cleaning data, one for hyperparameter optimization and/or training and one for the final stacking/blending.
", "votes": null, "replies": [ { "id": 436566, "author_name": "gonnel", "author_url": "", "post_date": "12/10/2018 14:27:39", "content": "Will look into this too. Thanks!
", "votes": null, "replies": [] } ] }, { "id": 437614, "author_name": "", "author_url": "", "post_date": "12/12/2018 07:35:50", "content": "Is there a similar way to do that in Python? Most of the data compression/read-time improvements are being done by changing the data itself.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "436375": "Looking to take on this as my first featured competition. I however have found that simply reading data takes a lot of time and worried training may take days. To be concise, how can I best reduce the time spent for instance on reading the data?\n I'm using R if it makes a difference. Thanks in advance.", "436493": "I use a kernel to load, cleaning, do feature engineering then save all of it to pkl or h5 files.\nThen load that kernel's output to do trianing. And it much faster and lighter.", "436560": "Thanks, I'll look into that.", "436563": "What R function are you using to read the data? Some read functions are faster than others. But I'd recommend that you use multiple kernels. Perhaps one for aggregating and cleaning data, one for hyperparameter optimization and/or training and one for the final stacking/blending.", "436566": "Will look into this too. Thanks!", "437614": "Is there a similar way to do that in Python? Most of the data compression/read-time improvements are being done by changing the data itself." }, "source": "meta" }