I'm now training xgb model on my school's machine( 32 cores, 192GB RAM). I split the data into 50*50 cell. I estimate it'll take me 70+ hours to train the model. Is it normal?
", "messages": [ { "id": "124515", "postDate": "06/19/2016 15:15:30", "content": "I'm now training xgb model on my school's machine( 32 cores, 192GB RAM). I split the data into 50*50 cell. I estimate it'll take me 70+ hours to train the model. Is it normal?
", "rawMarkdown": "I'm now training xgb model on my school's machine( 32 cores, 192GB RAM). I split the data into 50*50 cell. I estimate it'll take me 70+ hours to train the model. Is it normal?", "votes": null }, { "id": "124516", "postDate": "06/19/2016 15:33:49", "content": "There are two major factors affecting the runtime for xgboost.
\n\nn observations in the cell in order to consider this category)So for example, with 75(X) and 150(Y) cells, with a cut of 10 observations for that category in the bin will run much faster than your current setup. My runtimes are about 1 day on 16 cores.
", "rawMarkdown": "There are two major factors affecting the runtime for xgboost.\r\n\r\n - Number of categories per cell. The smaller this is, the faster it runs. You can change this with a threshold cut (must be at least `n` observations in the cell in order to consider this category)\r\n - How big are your cells. The smaller the cells, the faster this cell runs, which is just a function of the amount of data. Smaller cells also \"naturally\" reduce the number of categories the classifier has to consider. In my experience, smaller cells decrease total computation time, which is not obvious since the number of cells is increasing.\r\n\r\nSo for example, with 75(X) and 150(Y) cells, with a cut of 10 observations for that category in the bin will run much faster than your current setup. My runtimes are about 1 day on 16 cores.", "votes": null }, { "id": "124518", "postDate": "06/19/2016 15:40:15", "content": "Thanks Ravi, in your model, how many classes are there in each bin on average?
", "rawMarkdown": "Thanks Ravi, in your model, how many classes are there in each bin on average?", "votes": null }, { "id": "124522", "postDate": "06/19/2016 15:54:10", "content": "I'm not sure offhand without writing code to check. My threshold cut is between 5 and 10 depending on the cell with a 75x150 grid.
", "rawMarkdown": "I'm not sure offhand without writing code to check. My threshold cut is between 5 and 10 depending on the cell with a 75x150 grid.", "votes": null }, { "id": "124534", "postDate": "06/19/2016 18:42:21", "content": "On 4 cores i5-6500 it takes 1 minute per 50*50 cell with ~500 classes (no cutoff). On 32 cores, 192GB RAM it should run really fast unless you set number of trees or their depth too high. XGBoost default # estimators/trees is 100. I'd suggest to keep all defaults and see.
", "rawMarkdown": "On 4 cores i5-6500 it takes 1 minute per 50*50 cell with ~500 classes (no cutoff). On 32 cores, 192GB RAM it should run really fast unless you set number of trees or their depth too high. XGBoost default # estimators/trees is 100. I'd suggest to keep all defaults and see.", "votes": null }, { "id": "124539", "postDate": "06/19/2016 20:15:58", "content": "@rakhlin It seems you need 2500 minutes to run the whole dataset . It's strange because all my parameter is like default ( n_rounds=120, depth=3) . Now it has just finished 690 cells which takes 19 hours.
", "rawMarkdown": "rakhlin It seems you need 2500 minutes to run the whole dataset . It's strange because all my parameter is like default ( n_rounds=120, depth=3) . Now it has just finished 690 cells which takes 19 hours.", "votes": null }, { "id": "124540", "postDate": "06/19/2016 20:29:07", "content": "@FengLi then it's possible that this school machine isn't in your exclusive access
", "rawMarkdown": "FengLi then it's possible that this school machine isn't in your exclusive access", "votes": null }, { "id": "124543", "postDate": "06/19/2016 20:31:09", "content": "@rakhlin I've checked top in terminal. I'm actually using 32 cores to run my code.
", "rawMarkdown": "rakhlin I've checked *top* in terminal. I'm actually using 32 cores to run my code.", "votes": null }, { "id": "124547", "postDate": "06/19/2016 20:47:58", "content": "No idea if this is your issue, but, with xgboost, I've had large slowdowns if hyperthreading is enabled on the computer. So if you have 16 physical cores that are hyperthreaded to look like 32 cores, try using only 16 cores for xgboost.
", "rawMarkdown": "No idea if this is your issue, but, with xgboost, I've had large slowdowns if hyperthreading is enabled on the computer. So if you have 16 physical cores that are hyperthreaded to look like 32 cores, try using only 16 cores for xgboost.", "votes": null } ], "comments": [ { "id": 124516, "author_name": "rshekhar2", "author_url": "", "post_date": "06/19/2016 15:33:49", "content": "There are two major factors affecting the runtime for xgboost.
\n\nn observations in the cell in order to consider this category)So for example, with 75(X) and 150(Y) cells, with a cut of 10 observations for that category in the bin will run much faster than your current setup. My runtimes are about 1 day on 16 cores.
", "votes": null, "replies": [] }, { "id": 124518, "author_name": "beedata", "author_url": "", "post_date": "06/19/2016 15:40:15", "content": "Thanks Ravi, in your model, how many classes are there in each bin on average?
", "votes": null, "replies": [] }, { "id": 124522, "author_name": "rshekhar2", "author_url": "", "post_date": "06/19/2016 15:54:10", "content": "I'm not sure offhand without writing code to check. My threshold cut is between 5 and 10 depending on the cell with a 75x150 grid.
", "votes": null, "replies": [] }, { "id": 124534, "author_name": "rakhlin", "author_url": "", "post_date": "06/19/2016 18:42:21", "content": "On 4 cores i5-6500 it takes 1 minute per 50*50 cell with ~500 classes (no cutoff). On 32 cores, 192GB RAM it should run really fast unless you set number of trees or their depth too high. XGBoost default # estimators/trees is 100. I'd suggest to keep all defaults and see.
", "votes": null, "replies": [] }, { "id": 124539, "author_name": "beedata", "author_url": "", "post_date": "06/19/2016 20:15:58", "content": "@rakhlin It seems you need 2500 minutes to run the whole dataset . It's strange because all my parameter is like default ( n_rounds=120, depth=3) . Now it has just finished 690 cells which takes 19 hours.
", "votes": null, "replies": [] }, { "id": 124540, "author_name": "rakhlin", "author_url": "", "post_date": "06/19/2016 20:29:07", "content": "@FengLi then it's possible that this school machine isn't in your exclusive access
", "votes": null, "replies": [] }, { "id": 124543, "author_name": "beedata", "author_url": "", "post_date": "06/19/2016 20:31:09", "content": "@rakhlin I've checked top in terminal. I'm actually using 32 cores to run my code.
", "votes": null, "replies": [] }, { "id": 124547, "author_name": "rshekhar2", "author_url": "", "post_date": "06/19/2016 20:47:58", "content": "No idea if this is your issue, but, with xgboost, I've had large slowdowns if hyperthreading is enabled on the computer. So if you have 16 physical cores that are hyperthreaded to look like 32 cores, try using only 16 cores for xgboost.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "124515": "I'm now training xgb model on my school's machine( 32 cores, 192GB RAM). I split the data into 50*50 cell. I estimate it'll take me 70+ hours to train the model. Is it normal?", "124516": "There are two major factors affecting the runtime for xgboost.\r\n\r\n - Number of categories per cell. The smaller this is, the faster it runs. You can change this with a threshold cut (must be at least `n` observations in the cell in order to consider this category)\r\n - How big are your cells. The smaller the cells, the faster this cell runs, which is just a function of the amount of data. Smaller cells also \"naturally\" reduce the number of categories the classifier has to consider. In my experience, smaller cells decrease total computation time, which is not obvious since the number of cells is increasing.\r\n\r\nSo for example, with 75(X) and 150(Y) cells, with a cut of 10 observations for that category in the bin will run much faster than your current setup. My runtimes are about 1 day on 16 cores.", "124518": "Thanks Ravi, in your model, how many classes are there in each bin on average?", "124522": "I'm not sure offhand without writing code to check. My threshold cut is between 5 and 10 depending on the cell with a 75x150 grid.", "124534": "On 4 cores i5-6500 it takes 1 minute per 50*50 cell with ~500 classes (no cutoff). On 32 cores, 192GB RAM it should run really fast unless you set number of trees or their depth too high. XGBoost default # estimators/trees is 100. I'd suggest to keep all defaults and see.", "124539": "rakhlin It seems you need 2500 minutes to run the whole dataset . It's strange because all my parameter is like default ( n_rounds=120, depth=3) . Now it has just finished 690 cells which takes 19 hours.", "124540": "FengLi then it's possible that this school machine isn't in your exclusive access", "124543": "rakhlin I've checked *top* in terminal. I'm actually using 32 cores to run my code.", "124547": "No idea if this is your issue, but, with xgboost, I've had large slowdowns if hyperthreading is enabled on the computer. So if you have 16 physical cores that are hyperthreaded to look like 32 cores, try using only 16 cores for xgboost." }, "source": "meta" }