{ "id": 68148, "title": "Are you using Parallelism?", "url": "/competitions/ga-customer-revenue-prediction/discussion/68148", "author_name": "", "post_date": "2018-10-09T19:53:04.263272200Z", "votes": 1, "comment_count": 11, "views": 0, "content": "

I wonder whether you are using parallelism in order to make your code run faster. I am using a local computer having i7 processor, 4 cores and 8 threads. However, the running time looks so big for me. Are you using multiprocessing module in Python or any other library to accelerate your training ?

", "messages": [ { "id": "401306", "postDate": "10/09/2018 19:53:04", "content": "

I wonder whether you are using parallelism in order to make your code run faster. I am using a local computer having i7 processor, 4 cores and 8 threads. However, the running time looks so big for me. Are you using multiprocessing module in Python or any other library to accelerate your training ?

", "rawMarkdown": "I wonder whether you are using parallelism in order to make your code run faster. I am using a local computer having i7 processor, 4 cores and 8 threads. However, the running time looks so big for me. Are you using *multiprocessing* module in Python or any other library to accelerate your training ?", "votes": null }, { "id": "401327", "postDate": "10/09/2018 21:09:19", "content": "

Just for running the model (lightgbm, xgboost, tensorflow...), which you can set in the model parameters to 4. For the pipeline you can use pandas with python's multiprocessing library or Dask, but I dont think it is necessary in this competition. If a group of feature is taking too long to process you can pre-save with pickle or feather.

\n\n

For the number of jobs, I don't know if using more than the number of physical cores does help

", "rawMarkdown": "Just for running the model (lightgbm, xgboost, tensorflow...), which you can set in the model parameters to 4. For the pipeline you can use pandas with python's multiprocessing library or [Dask][1], but I dont think it is necessary in this competition. If a group of feature is taking too long to process you can pre-save with pickle or feather.\n\nFor the number of jobs, I don't know if using more than the number of physical cores does help\n\n\n [1]: https://dask.org/", "votes": null }, { "id": "401343", "postDate": "10/09/2018 21:48:41", "content": "

Are you running R or python?

\n\n

If you have a look at some of the great kernels from Olivier (https://www.kaggle.com/ogrellier) the runtime is ca. 811 seconds (example https://www.kaggle.com/ogrellier/i-have-seen-the-future/notebook).

\n\n

In R also some scripts with ca. 500 seconds run time (example https://www.kaggle.com/pratikmirani/pmlightgbm-in-r-time-based-cv/code).

\n\n

In my humble opinion this is quite fast. Maybe you can have a look and see how it was done.

", "rawMarkdown": "Are you running R or python? \n\nIf you have a look at some of the great kernels from Olivier (https://www.kaggle.com/ogrellier) the runtime is ca. 811 seconds (example https://www.kaggle.com/ogrellier/i-have-seen-the-future/notebook). \n\nIn R also some scripts with ca. 500 seconds run time (example https://www.kaggle.com/pratikmirani/pmlightgbm-in-r-time-based-cv/code).\n\nIn my humble opinion this is quite fast. Maybe you can have a look and see how it was done.", "votes": null }, { "id": "401778", "postDate": "10/10/2018 17:36:57", "content": "

I am using Python.

", "rawMarkdown": "I am using Python.", "votes": null }, { "id": "401802", "postDate": "10/10/2018 18:10:31", "content": "

Instead of CPU multicores, try using GPU computation which is miles away ahead.

\n\n

You can use it with XGBoost, Lgbm, CatBoost...

", "rawMarkdown": "Instead of CPU multicores, try using GPU computation which is miles away ahead.\n\nYou can use it with XGBoost, Lgbm, CatBoost...", "votes": null }, { "id": "401808", "postDate": "10/10/2018 18:16:09", "content": "

I know GPU is much much faster but I only use TensorFlow and Keras with GPU on my local machine. I didn't install Lightgbm with gpu or xgboost with gpu. My OS is Windows 10. I thought it was so hard to install lightgbm gpu version. Therefore, I am using CPU version.

", "rawMarkdown": "I know GPU is much much faster but I only use TensorFlow and Keras with GPU on my local machine. I didn't install Lightgbm with gpu or xgboost with gpu. My OS is Windows 10. I thought it was so hard to install lightgbm gpu version. Therefore, I am using CPU version.", "votes": null }, { "id": "401814", "postDate": "10/10/2018 18:26:39", "content": "

With Linux it's so easy... I don't use Windows, except at the office.

", "rawMarkdown": "With Linux it's so easy... I don't use Windows, except at the office.", "votes": null }, { "id": "401816", "postDate": "10/10/2018 18:33:11", "content": "

You are right. I will pass to Linux ecosystem as soon as possible.

", "rawMarkdown": "You are right. I will pass to Linux ecosystem as soon as possible.", "votes": null }, { "id": "401824", "postDate": "10/10/2018 18:45:41", "content": "

And if you really need Windows, you can use a dual boot or a virtual machine under Linux : VMware player is really easy to install, free and does the job for Windows.

", "rawMarkdown": "And if you really *need* Windows, you can use a dual boot or a virtual machine under Linux : **VMware** player is really easy to install, free and does the job for Windows.", "votes": null }, { "id": "401836", "postDate": "10/10/2018 18:59:20", "content": "

Dask for python.\nForeach for R

", "rawMarkdown": "Dask for python.\nForeach for R", "votes": null }, { "id": "401896", "postDate": "10/10/2018 21:03:08", "content": "

If you are using LightGBM, It doesnt work well with old GPUs (700 series or older)

", "rawMarkdown": "If you are using LightGBM, It doesnt work well with old GPUs (700 series or older)", "votes": null }, { "id": "402456", "postDate": "10/11/2018 18:50:35", "content": "

I have a GTX 1050, which has 640 CUDA Cores, 4 GB RAM and Pascal architecture.

", "rawMarkdown": "I have a GTX 1050, which has 640 CUDA Cores, 4 GB RAM and Pascal architecture.", "votes": null } ], "comments": [ { "id": 401327, "author_name": "jsaguiar", "author_url": "", "post_date": "10/09/2018 21:09:19", "content": "

Just for running the model (lightgbm, xgboost, tensorflow...), which you can set in the model parameters to 4. For the pipeline you can use pandas with python's multiprocessing library or Dask, but I dont think it is necessary in this competition. If a group of feature is taking too long to process you can pre-save with pickle or feather.

\n\n

For the number of jobs, I don't know if using more than the number of physical cores does help

", "votes": null, "replies": [] }, { "id": 401343, "author_name": "justfor", "author_url": "", "post_date": "10/09/2018 21:48:41", "content": "

Are you running R or python?

\n\n

If you have a look at some of the great kernels from Olivier (https://www.kaggle.com/ogrellier) the runtime is ca. 811 seconds (example https://www.kaggle.com/ogrellier/i-have-seen-the-future/notebook).

\n\n

In R also some scripts with ca. 500 seconds run time (example https://www.kaggle.com/pratikmirani/pmlightgbm-in-r-time-based-cv/code).

\n\n

In my humble opinion this is quite fast. Maybe you can have a look and see how it was done.

", "votes": null, "replies": [ { "id": 401778, "author_name": "mbkinaci", "author_url": "", "post_date": "10/10/2018 17:36:57", "content": "

I am using Python.

", "votes": null, "replies": [] } ] }, { "id": 401802, "author_name": "pfichou", "author_url": "", "post_date": "10/10/2018 18:10:31", "content": "

Instead of CPU multicores, try using GPU computation which is miles away ahead.

\n\n

You can use it with XGBoost, Lgbm, CatBoost...

", "votes": null, "replies": [ { "id": 401808, "author_name": "mbkinaci", "author_url": "", "post_date": "10/10/2018 18:16:09", "content": "

I know GPU is much much faster but I only use TensorFlow and Keras with GPU on my local machine. I didn't install Lightgbm with gpu or xgboost with gpu. My OS is Windows 10. I thought it was so hard to install lightgbm gpu version. Therefore, I am using CPU version.

", "votes": null, "replies": [] }, { "id": 401814, "author_name": "pfichou", "author_url": "", "post_date": "10/10/2018 18:26:39", "content": "

With Linux it's so easy... I don't use Windows, except at the office.

", "votes": null, "replies": [] }, { "id": 401816, "author_name": "mbkinaci", "author_url": "", "post_date": "10/10/2018 18:33:11", "content": "

You are right. I will pass to Linux ecosystem as soon as possible.

", "votes": null, "replies": [] }, { "id": 401824, "author_name": "pfichou", "author_url": "", "post_date": "10/10/2018 18:45:41", "content": "

And if you really need Windows, you can use a dual boot or a virtual machine under Linux : VMware player is really easy to install, free and does the job for Windows.

", "votes": null, "replies": [] }, { "id": 401896, "author_name": "jsaguiar", "author_url": "", "post_date": "10/10/2018 21:03:08", "content": "

If you are using LightGBM, It doesnt work well with old GPUs (700 series or older)

", "votes": null, "replies": [] }, { "id": 402456, "author_name": "mbkinaci", "author_url": "", "post_date": "10/11/2018 18:50:35", "content": "

I have a GTX 1050, which has 640 CUDA Cores, 4 GB RAM and Pascal architecture.

", "votes": null, "replies": [] } ] }, { "id": 401836, "author_name": "freedomslaves22", "author_url": "", "post_date": "10/10/2018 18:59:20", "content": "

Dask for python.\nForeach for R

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "401306": "I wonder whether you are using parallelism in order to make your code run faster. I am using a local computer having i7 processor, 4 cores and 8 threads. However, the running time looks so big for me. Are you using *multiprocessing* module in Python or any other library to accelerate your training ?", "401327": "Just for running the model (lightgbm, xgboost, tensorflow...), which you can set in the model parameters to 4. For the pipeline you can use pandas with python's multiprocessing library or [Dask][1], but I dont think it is necessary in this competition. If a group of feature is taking too long to process you can pre-save with pickle or feather.\n\nFor the number of jobs, I don't know if using more than the number of physical cores does help\n\n\n [1]: https://dask.org/", "401343": "Are you running R or python? \n\nIf you have a look at some of the great kernels from Olivier (https://www.kaggle.com/ogrellier) the runtime is ca. 811 seconds (example https://www.kaggle.com/ogrellier/i-have-seen-the-future/notebook). \n\nIn R also some scripts with ca. 500 seconds run time (example https://www.kaggle.com/pratikmirani/pmlightgbm-in-r-time-based-cv/code).\n\nIn my humble opinion this is quite fast. Maybe you can have a look and see how it was done.", "401778": "I am using Python.", "401802": "Instead of CPU multicores, try using GPU computation which is miles away ahead.\n\nYou can use it with XGBoost, Lgbm, CatBoost...", "401808": "I know GPU is much much faster but I only use TensorFlow and Keras with GPU on my local machine. I didn't install Lightgbm with gpu or xgboost with gpu. My OS is Windows 10. I thought it was so hard to install lightgbm gpu version. Therefore, I am using CPU version.", "401814": "With Linux it's so easy... I don't use Windows, except at the office.", "401816": "You are right. I will pass to Linux ecosystem as soon as possible.", "401824": "And if you really *need* Windows, you can use a dual boot or a virtual machine under Linux : **VMware** player is really easy to install, free and does the job for Windows.", "401836": "Dask for python.\nForeach for R", "401896": "If you are using LightGBM, It doesnt work well with old GPUs (700 series or older)", "402456": "I have a GTX 1050, which has 640 CUDA Cores, 4 GB RAM and Pascal architecture." }, "source": "meta" }