Personally, for this competition I've been using caret more than ever before (setting the 'method' argument to xgbTree in caret's "train" function gets you the xgboost implementation).
With caret it's really easy to do cross-validation and compare different parameter settings (currently you can vary "eta", "max_depth", and "nrounds").
\n\nIf anyone's interested, here's an example of what the code would look like:
\n\nctrl <- trainControl('cv', number=5, summaryFunction=twoClassSummary,\n verboseIter=TRUE, classProbs=TRUE)\nxgb.grid <- expand.grid(nrounds=2:20*100, max_depth=c(6, 8),\n eta=c(0.01, 0.035))\nfit.xgbTree <- train(x=training, y=target, method='xgbTree', metric='ROC',\n trControl=ctrl, tuneGrid=xgb.grid, subsample=0.5, \n colsample.bytree=0.8)\nTime to re-ask this question... especially since the private LB results are generally ~.003-005 lower! (I personally can't complain, I never cracked the top 20 on the public LB... ;) )
", "rawMarkdown": "Time to re-ask this question... especially since the private LB results are generally ~.003-005 lower! (I personally can't complain, I never cracked the top 20 on the public LB... ;) )", "votes": 1 }, { "id": 96684, "postDate": "2015-10-19T17:24:55.787Z", "content": "[quote=KeithMessina;96668]
\n\nHas everybody been rounding or transforming the data from the XGBoost to get "1"s and "0"s or have you been leaving it as-is when you predict using the XGBoost model? I've been getting much worse values when I round, but the output as-is doesn't really make sense from a yes/no decision making on whether to send marketing info to the prospects that we are predicting.
\n\n[/quote]
\n\nThis has been discussed before on this board, so look around, but the short version is to look at the AUC algorithm. What you are predicting is the probability that it is a 1. Therefore it allows more nuance. For example the marketing dept could decide that it is profitable to send mail at a cuttoff of 0.8 (80% sure that it is a yes), or some other threshold. The AUC takes into account that variable thresholds may be used to determine what is yes / no, so a fine grade probability is more useful than just a binary classifier as 0 or 1.
", "rawMarkdown": "[quote=KeithMessina;96668]\r\n\r\nHas everybody been rounding or transforming the data from the XGBoost to get \"1\"s and \"0\"s or have you been leaving it as-is when you predict using the XGBoost model? I've been getting much worse values when I round, but the output as-is doesn't really make sense from a yes/no decision making on whether to send marketing info to the prospects that we are predicting.\r\n\r\n[/quote]\r\n\r\nThis has been discussed before on this board, so look around, but the short version is to look at the AUC algorithm. What you are predicting is the probability that it is a 1. Therefore it allows more nuance. For example the marketing dept could decide that it is profitable to send mail at a cuttoff of 0.8 (80% sure that it is a yes), or some other threshold. The AUC takes into account that variable thresholds may be used to determine what is yes / no, so a fine grade probability is more useful than just a binary classifier as 0 or 1.", "votes": 1 }, { "id": 96663, "postDate": "2015-10-19T14:58:44.037Z", "content": "[quote=.Григорій.;96660]
\n\nYes, it's possible to get AUC=.8137 by usual Logistic Regression on validation sample after extensive data transformation work. xgBoost give AUC=.8369 in internal test.
\n\nUnfortunately as a newcomer I cannot submit the results and looking for a cooperation in a team.
\n\n[/quote]
\n\nI recommend to double check your internal validation procedure. If your result is correct then you can be at the first place with large margin.
", "rawMarkdown": "[quote=.Григорій.;96660]\r\n\r\nYes, it's possible to get AUC=.8137 by usual Logistic Regression on validation sample after extensive data transformation work. xgBoost give AUC=.8369 in internal test.\r\n\r\nUnfortunately as a newcomer I cannot submit the results and looking for a cooperation in a team. \r\n\r\n\r\n[/quote]\r\n\r\nI recommend to double check your internal validation procedure. If your result is correct then you can be at the first place with large margin.", "votes": 2 }, { "id": 96227, "postDate": "2015-10-15T04:58:36.887Z", "content": "I got .80+ using a single XGB.
\n\nYou can get .70&change if you take this approach: I am at test[x,y] - what is the probability that the next value, be it, say, 834, is positive, and negative - tally and move to the next variable - basically, take the probability of "what's next?"
", "rawMarkdown": "I got .80+ using a single XGB.\r\n\r\nYou can get .70&change if you take this approach: I am at test[x,y] - what is the probability that the next value, be it, say, 834, is positive, and negative - tally and move to the next variable - basically, take the probability of \"what's next?\"", "votes": 2 }, { "id": 96666, "postDate": "2015-10-19T15:12:57.940Z", "content": "@Maxx The models were evaluated correctly after scoring on hidden validation sample. \nSolution of such problem is a part of production process in my company.
", "rawMarkdown": "@Maxx The models were evaluated correctly after scoring on hidden validation sample. \r\nSolution of such problem is a part of production process in my company.\r\n", "votes": -2 }, { "id": 96660, "postDate": "2015-10-19T14:36:09.397Z", "content": "Yes, it's possible to get AUC=.8137 by usual Logistic Regression on validation sample after extensive data transformation work. xgBoost give AUC=.8369 in internal test.
\n\nUnfortunately as a newcomer I cannot submit the results and looking for a cooperation in a team.
", "rawMarkdown": "Yes, it's possible to get AUC=.8137 by usual Logistic Regression on validation sample after extensive data transformation work. xgBoost give AUC=.8369 in internal test.\r\n\r\nUnfortunately as a newcomer I cannot submit the results and looking for a cooperation in a team. \r\n", "votes": -5 }, { "id": 97096, "postDate": "2015-10-22T16:56:18.737Z", "content": "Thanks a lot [quote=SteveS;96384]
\n\nPersonally, for this competition I've been using caret more than ever before (setting the 'method' argument to xgbTree in caret's "train" function gets you the xgboost implementation).
With caret it's really easy to do cross-validation and compare different parameter settings (currently you can vary "eta", "max_depth", and "nrounds").
\n\nIf anyone's interested, here's an example of what the code would look like:
\n\nctrl <- trainControl('cv', number=5, summaryFunction=twoClassSummary,\n verboseIter=TRUE, classProbs=TRUE)\nxgb.grid <- expand.grid(nrounds=2:20*100, max_depth=c(6, 8),\n eta=c(0.01, 0.035))\nfit.xgbTree <- train(x=training, y=target, method='xgbTree', metric='ROC',\n trControl=ctrl, tuneGrid=xgb.grid, subsample=0.5, \n colsample.bytree=0.8)\n[/quote]
", "rawMarkdown": "Thanks a lot [quote=SteveS;96384]\r\n\r\nPersonally, for this competition I've been using **caret** more than ever before (setting the 'method' argument to `xgbTree` in caret's \"train\" function gets you the **xgboost** implementation). \r\n\r\nWith caret it's really easy to do cross-validation and compare different parameter settings (currently you can vary \"eta\", \"max_depth\", and \"nrounds\").\r\n\r\nIf anyone's interested, here's an example of what the code would look like:\r\n\r\n ctrl <- trainControl('cv', number=5, summaryFunction=twoClassSummary,\r\n verboseIter=TRUE, classProbs=TRUE)\r\n xgb.grid <- expand.grid(nrounds=2:20*100, max_depth=c(6, 8),\r\n eta=c(0.01, 0.035))\r\n fit.xgbTree <- train(x=training, y=target, method='xgbTree', metric='ROC',\r\n trControl=ctrl, tuneGrid=xgb.grid, subsample=0.5, \r\n colsample.bytree=0.8)\r\n\r\n[/quote]\r\n" }, { "id": 96668, "postDate": "2015-10-19T15:14:20.733Z", "content": "Has everybody been rounding or transforming the data from the XGBoost to get "1"s and "0"s or have you been leaving it as-is when you predict using the XGBoost model? I've been getting much worse values when I round, but the output as-is doesn't really make sense from a yes/no decision making on whether to send marketing info to the prospects that we are predicting.
", "rawMarkdown": "Has everybody been rounding or transforming the data from the XGBoost to get \"1\"s and \"0\"s or have you been leaving it as-is when you predict using the XGBoost model? I've been getting much worse values when I round, but the output as-is doesn't really make sense from a yes/no decision making on whether to send marketing info to the prospects that we are predicting." }, { "id": 96667, "postDate": "2015-10-19T15:13:41.237Z", "content": "Has everybody been rounding or transforming the data from the XGBoost to get "1"s and "0"s or have you been leaving it as-is when you predict using the XGBoost model? I've been getting much worse values when I found, but the output as-is doesn't really make sense from a yes/no decision making on whether to send marketing info to the prospects that we are predicting.
", "rawMarkdown": "Has everybody been rounding or transforming the data from the XGBoost to get \"1\"s and \"0\"s or have you been leaving it as-is when you predict using the XGBoost model? I've been getting much worse values when I found, but the output as-is doesn't really make sense from a yes/no decision making on whether to send marketing info to the prospects that we are predicting." }, { "id": 96661, "postDate": "2015-10-19T14:49:19.360Z", "content": "I tried many things but neither did my internal AUC improve much after 0.79xx nor did LB. \nGetting 0.8+ with a single model is commendable!
", "rawMarkdown": "I tried many things but neither did my internal AUC improve much after 0.79xx nor did LB. \r\nGetting 0.8+ with a single model is commendable!" }, { "id": 96377, "postDate": "2015-10-15T22:40:55.867Z", "content": "First - an apology all since I think this post is definitely going off-topic for this thread!
\n\nKeith - I just use xgb.cv to figure out how many rounds to run when I use xgb.train for real. Assuming you are using "R"... save the result and you will have a matrix and you can use which.max() function (assuming you want the max AUC - if you are using logloss for example then use which.min()) to figure out an appropriate number of rounds.
\n\nRegarding RStudio - and memory - there is an excellent post in these forums from Eric Couto about using less memory to hold the data. But - as I understand XGBoost - it takes a copy of the "R" data into its own c++ data structure (i may be wrong about this but I think thats what it is doing). So it may be that the XGBoost is holding a copy of the data PLUS a copy of all the trees (one for each round) that it is generating. Hence the 8Gb.
\n\nBut if anyone else actually knows how XGB is using memory I would certainly be interested in understanding it.
", "rawMarkdown": "First - an apology all since I think this post is definitely going off-topic for this thread!\r\n\r\nKeith - I just use xgb.cv to figure out how many rounds to run when I use xgb.train for real. Assuming you are using \"R\"... save the result and you will have a matrix and you can use which.max() function (assuming you want the max AUC - if you are using logloss for example then use which.min()) to figure out an appropriate number of rounds.\r\n\r\nRegarding RStudio - and memory - there is an excellent post in these forums from Eric Couto about using less memory to hold the data. But - as I understand XGBoost - it takes a *copy* of the \"R\" data into its own c++ data structure (i may be wrong about this but I think thats what it is doing). So it may be that the XGBoost is holding a copy of the data PLUS a copy of *all* the trees (one for each round) that it is generating. Hence the 8Gb.\r\n\r\nBut if anyone else actually *knows* how XGB is using memory I would certainly be interested in understanding it.\r\n" }, { "id": 96373, "postDate": "2015-10-15T22:32:29.587Z", "content": "You should be able to get to .79x with a 10/90 test/train split. It's a big enough training set that you can get reasonable results without CV. Try different parameters and massage the data going into xgb a bit more and you should see nice gains.
\n\nMemory usage depends on how the data is unpacked and stored. A compact binary representation is about 900MB for each set IIRC.
", "rawMarkdown": "You should be able to get to .79x with a 10/90 test/train split. It's a big enough training set that you can get reasonable results without CV. Try different parameters and massage the data going into xgb a bit more and you should see nice gains.\r\n\r\nMemory usage depends on how the data is unpacked and stored. A compact binary representation is about 900MB for each set IIRC." }, { "id": 96370, "postDate": "2015-10-15T22:26:50.690Z", "content": "Haven't cracked .77 with XGBoost yet. I'm not cross-validating, which I think is my biggest issue. Can't figure out exactly how to use the xgb.cv function.
\n\n2-core Celeron G1840 with 16GB RAM on Ubuntu. I wasn't able to load the full train.csv at the 8GB, so I upgraded. Anyone know why loading what RStudio calls a 1.4GB dataset takes over 8GB when that is the only thing running? I have RStudio server setup and I remote into it via browser.
", "rawMarkdown": "Haven't cracked .77 with XGBoost yet. I'm not cross-validating, which I think is my biggest issue. Can't figure out exactly how to use the xgb.cv function. \r\n\r\n2-core Celeron G1840 with 16GB RAM on Ubuntu. I wasn't able to load the full train.csv at the 8GB, so I upgraded. Anyone know why loading what RStudio calls a 1.4GB dataset takes over 8GB when that is the only thing running? I have RStudio server setup and I remote into it via browser.\r\n\r\n" }, { "id": 96360, "postDate": "2015-10-15T20:42:10.567Z", "content": "Apshreyans - I started the competition with an i5 notebook w/16Gb RAM but I now have access to an (older) i7 with 20Gb RAM. With all the features I've been adding I am definitely finding 16Gb is useful, but a depth=30 tree still takes many many hours to run.
\n\nI've avoided being too selective on features because in the past I've found xgboost can deliver small increments to the evaluation criteria with even low-correlated features - and in these competitions even a small increase can gain you quite a few spots on the leaderboard - whereas in real life I would not usually bother. So I've just been adding new features and letting the algorithm decide the best ones - doesn't stop me running XGB.Importance to see what its doing though!
", "rawMarkdown": "Apshreyans - I started the competition with an i5 notebook w/16Gb RAM but I now have access to an (older) i7 with 20Gb RAM. With all the features I've been adding I am definitely finding 16Gb is useful, but a depth=30 tree still takes many many hours to run. \r\n\r\nI've avoided being too selective on features because in the past I've found xgboost can deliver small increments to the evaluation criteria with even low-correlated features - and in these competitions even a small increase can gain you quite a few spots on the leaderboard - whereas in real life I would not usually bother. So I've just been adding new features and letting the algorithm decide the best ones - doesn't stop me running XGB.Importance to see what its doing though!" }, { "id": 96275, "postDate": "2015-10-15T12:54:45.677Z", "content": "Your score sounds very good for resources you have. We currently need 16GB of RAM for our solution. However, we were not doing any activity to prune uninformative features, which is not good for sure.
", "rawMarkdown": "Your score sounds very good for resources you have. We currently need 16GB of RAM for our solution. However, we were not doing any activity to prune uninformative features, which is not good for sure." }, { "id": 96272, "postDate": "2015-10-15T12:47:51.117Z", "content": "@Maxx @Michael @Richard What are your PC configurations?? I am at .79776 with my i3 and 4GB RAM laptop.
", "rawMarkdown": "@Maxx @Michael @Richard What are your PC configurations?? I am at .79776 with my i3 and 4GB RAM laptop." }, { "id": 96246, "postDate": "2015-10-15T08:44:09.947Z", "content": "We got our current score (0.8005) with single xgboost on quite basic features.
", "rawMarkdown": "We got our current score (0.8005) with single xgboost on quite basic features." }, { "id": 96204, "postDate": "2015-10-15T02:32:13.433Z", "rawMarkdown": "" }, { "id": 96165, "postDate": "2015-10-14T22:01:00.940Z", "content": "Well I've finally managed to crack 0.80000 but... its an ensemble model of 3 XGBOOSTs - I keep trying new things and not making a lot of progress... but there's still a few days yet so I am seeing what I can come up with. The list of people with better models above me keeps growing and growing... .
\n\nbtw I did try a C5.0 model (it was a bit of a struggle to get it run) but it wasn't very effective - only around .785 and I didn't even submit it in the end...
", "rawMarkdown": "Well I've finally managed to crack 0.80000 but... its an ensemble model of 3 XGBOOSTs - I keep trying new things and not making a lot of progress... but there's still a few days yet so I am seeing what I can come up with. The list of people with better models above me keeps growing and growing...I have a feeling whoever did (and I'm not one of 'em!) won't tell until next week. It's just too much of an advantage ;)
", "rawMarkdown": "I have a feeling whoever did (and I'm not one of 'em!) won't tell until next week. It's just too much of an advantage ;)" }, { "id": 95870, "postDate": "2015-10-12T18:44:48.037Z", "content": "@Loknar right, i meant without xgboost nor GBC.
", "rawMarkdown": "@Loknar right, i meant without xgboost nor GBC." }, { "id": 95858, "postDate": "2015-10-12T17:56:43.640Z", "content": "i dont see any difference in AUCs in my tests between xgboost and RF/GBC from sklearn on same data, so i doubt the classifier itself makes the difference.
", "rawMarkdown": "i dont see any difference in AUCs in my tests between xgboost and RF/GBC from sklearn on same data, so i doubt the classifier itself makes the difference." }, { "id": 95736, "postDate": "2015-10-11T07:36:07.037Z", "rawMarkdown": "" }, { "id": 96664, "postDate": "2015-10-19T14:59:03.283Z", "rawMarkdown": "", "votes": 2, "isDeleted": true } ], "comments": [ { "id": 96384, "author_name": "SteveS", "author_url": "", "post_date": "2015-10-15T23:29:53.367000", "content": "Personally, for this competition I've been using caret more than ever before (setting the 'method' argument to xgbTree in caret's "train" function gets you the xgboost implementation).
With caret it's really easy to do cross-validation and compare different parameter settings (currently you can vary "eta", "max_depth", and "nrounds").
\n\nIf anyone's interested, here's an example of what the code would look like:
\n\nctrl <- trainControl('cv', number=5, summaryFunction=twoClassSummary,\n verboseIter=TRUE, classProbs=TRUE)\nxgb.grid <- expand.grid(nrounds=2:20*100, max_depth=c(6, 8),\n eta=c(0.01, 0.035))\nfit.xgbTree <- train(x=training, y=target, method='xgbTree', metric='ROC',\n trControl=ctrl, tuneGrid=xgb.grid, subsample=0.5, \n colsample.bytree=0.8)\nTime to re-ask this question... especially since the private LB results are generally ~.003-005 lower! (I personally can't complain, I never cracked the top 20 on the public LB... ;) )
", "votes": 1, "replies": [] }, { "id": 96684, "author_name": "pi_informatics", "author_url": "", "post_date": "2015-10-19T17:24:55.787000", "content": "[quote=KeithMessina;96668]
\n\nHas everybody been rounding or transforming the data from the XGBoost to get "1"s and "0"s or have you been leaving it as-is when you predict using the XGBoost model? I've been getting much worse values when I round, but the output as-is doesn't really make sense from a yes/no decision making on whether to send marketing info to the prospects that we are predicting.
\n\n[/quote]
\n\nThis has been discussed before on this board, so look around, but the short version is to look at the AUC algorithm. What you are predicting is the probability that it is a 1. Therefore it allows more nuance. For example the marketing dept could decide that it is profitable to send mail at a cuttoff of 0.8 (80% sure that it is a yes), or some other threshold. The AUC takes into account that variable thresholds may be used to determine what is yes / no, so a fine grade probability is more useful than just a binary classifier as 0 or 1.
", "votes": 1, "replies": [] }, { "id": 96663, "author_name": "Maxx", "author_url": "", "post_date": "2015-10-19T14:58:44.037000", "content": "[quote=.Григорій.;96660]
\n\nYes, it's possible to get AUC=.8137 by usual Logistic Regression on validation sample after extensive data transformation work. xgBoost give AUC=.8369 in internal test.
\n\nUnfortunately as a newcomer I cannot submit the results and looking for a cooperation in a team.
\n\n[/quote]
\n\nI recommend to double check your internal validation procedure. If your result is correct then you can be at the first place with large margin.
", "votes": 2, "replies": [] }, { "id": 96227, "author_name": "Michael Optional Anderson", "author_url": "", "post_date": "2015-10-15T04:58:36.887000", "content": "I got .80+ using a single XGB.
\n\nYou can get .70&change if you take this approach: I am at test[x,y] - what is the probability that the next value, be it, say, 834, is positive, and negative - tally and move to the next variable - basically, take the probability of "what's next?"
", "votes": 2, "replies": [] }, { "id": 96666, "author_name": "g_I", "author_url": "", "post_date": "2015-10-19T15:12:57.940000", "content": "@Maxx The models were evaluated correctly after scoring on hidden validation sample. \nSolution of such problem is a part of production process in my company.
", "votes": -2, "replies": [] }, { "id": 96660, "author_name": "g_I", "author_url": "", "post_date": "2015-10-19T14:36:09.397000", "content": "Yes, it's possible to get AUC=.8137 by usual Logistic Regression on validation sample after extensive data transformation work. xgBoost give AUC=.8369 in internal test.
\n\nUnfortunately as a newcomer I cannot submit the results and looking for a cooperation in a team.
", "votes": -5, "replies": [] }, { "id": 97096, "author_name": "Chris", "author_url": "", "post_date": "2015-10-22T16:56:18.737000", "content": "Thanks a lot [quote=SteveS;96384]
\n\nPersonally, for this competition I've been using caret more than ever before (setting the 'method' argument to xgbTree in caret's "train" function gets you the xgboost implementation).
With caret it's really easy to do cross-validation and compare different parameter settings (currently you can vary "eta", "max_depth", and "nrounds").
\n\nIf anyone's interested, here's an example of what the code would look like:
\n\nctrl <- trainControl('cv', number=5, summaryFunction=twoClassSummary,\n verboseIter=TRUE, classProbs=TRUE)\nxgb.grid <- expand.grid(nrounds=2:20*100, max_depth=c(6, 8),\n eta=c(0.01, 0.035))\nfit.xgbTree <- train(x=training, y=target, method='xgbTree', metric='ROC',\n trControl=ctrl, tuneGrid=xgb.grid, subsample=0.5, \n colsample.bytree=0.8)\n[/quote]
", "votes": 0, "replies": [] }, { "id": 96668, "author_name": "KeithMessina", "author_url": "", "post_date": "2015-10-19T15:14:20.733000", "content": "Has everybody been rounding or transforming the data from the XGBoost to get "1"s and "0"s or have you been leaving it as-is when you predict using the XGBoost model? I've been getting much worse values when I round, but the output as-is doesn't really make sense from a yes/no decision making on whether to send marketing info to the prospects that we are predicting.
", "votes": 0, "replies": [] }, { "id": 96667, "author_name": "KeithMessina", "author_url": "", "post_date": "2015-10-19T15:13:41.237000", "content": "Has everybody been rounding or transforming the data from the XGBoost to get "1"s and "0"s or have you been leaving it as-is when you predict using the XGBoost model? I've been getting much worse values when I found, but the output as-is doesn't really make sense from a yes/no decision making on whether to send marketing info to the prospects that we are predicting.
", "votes": 0, "replies": [] }, { "id": 96661, "author_name": "Abhay Shukla", "author_url": "", "post_date": "2015-10-19T14:49:19.360000", "content": "I tried many things but neither did my internal AUC improve much after 0.79xx nor did LB. \nGetting 0.8+ with a single model is commendable!
", "votes": 0, "replies": [] }, { "id": 96377, "author_name": "Richard Hunt", "author_url": "", "post_date": "2015-10-15T22:40:55.867000", "content": "First - an apology all since I think this post is definitely going off-topic for this thread!
\n\nKeith - I just use xgb.cv to figure out how many rounds to run when I use xgb.train for real. Assuming you are using "R"... save the result and you will have a matrix and you can use which.max() function (assuming you want the max AUC - if you are using logloss for example then use which.min()) to figure out an appropriate number of rounds.
\n\nRegarding RStudio - and memory - there is an excellent post in these forums from Eric Couto about using less memory to hold the data. But - as I understand XGBoost - it takes a copy of the "R" data into its own c++ data structure (i may be wrong about this but I think thats what it is doing). So it may be that the XGBoost is holding a copy of the data PLUS a copy of all the trees (one for each round) that it is generating. Hence the 8Gb.
\n\nBut if anyone else actually knows how XGB is using memory I would certainly be interested in understanding it.
", "votes": 0, "replies": [] }, { "id": 96373, "author_name": "happycube", "author_url": "", "post_date": "2015-10-15T22:32:29.587000", "content": "You should be able to get to .79x with a 10/90 test/train split. It's a big enough training set that you can get reasonable results without CV. Try different parameters and massage the data going into xgb a bit more and you should see nice gains.
\n\nMemory usage depends on how the data is unpacked and stored. A compact binary representation is about 900MB for each set IIRC.
", "votes": 0, "replies": [] }, { "id": 96370, "author_name": "KeithMessina", "author_url": "", "post_date": "2015-10-15T22:26:50.690000", "content": "Haven't cracked .77 with XGBoost yet. I'm not cross-validating, which I think is my biggest issue. Can't figure out exactly how to use the xgb.cv function.
\n\n2-core Celeron G1840 with 16GB RAM on Ubuntu. I wasn't able to load the full train.csv at the 8GB, so I upgraded. Anyone know why loading what RStudio calls a 1.4GB dataset takes over 8GB when that is the only thing running? I have RStudio server setup and I remote into it via browser.
", "votes": 0, "replies": [] }, { "id": 96360, "author_name": "Richard Hunt", "author_url": "", "post_date": "2015-10-15T20:42:10.567000", "content": "Apshreyans - I started the competition with an i5 notebook w/16Gb RAM but I now have access to an (older) i7 with 20Gb RAM. With all the features I've been adding I am definitely finding 16Gb is useful, but a depth=30 tree still takes many many hours to run.
\n\nI've avoided being too selective on features because in the past I've found xgboost can deliver small increments to the evaluation criteria with even low-correlated features - and in these competitions even a small increase can gain you quite a few spots on the leaderboard - whereas in real life I would not usually bother. So I've just been adding new features and letting the algorithm decide the best ones - doesn't stop me running XGB.Importance to see what its doing though!
", "votes": 0, "replies": [] }, { "id": 96275, "author_name": "Maxx", "author_url": "", "post_date": "2015-10-15T12:54:45.677000", "content": "Your score sounds very good for resources you have. We currently need 16GB of RAM for our solution. However, we were not doing any activity to prune uninformative features, which is not good for sure.
", "votes": 0, "replies": [] }, { "id": 96272, "author_name": "apshreyans", "author_url": "", "post_date": "2015-10-15T12:47:51.117000", "content": "@Maxx @Michael @Richard What are your PC configurations?? I am at .79776 with my i3 and 4GB RAM laptop.
", "votes": 0, "replies": [] }, { "id": 96246, "author_name": "Maxx", "author_url": "", "post_date": "2015-10-15T08:44:09.947000", "content": "We got our current score (0.8005) with single xgboost on quite basic features.
", "votes": 0, "replies": [] }, { "id": 96204, "author_name": "Vincent.Y", "author_url": "", "post_date": "2015-10-15T02:32:13.433000", "content": "", "votes": 0, "replies": [] }, { "id": 96165, "author_name": "Richard Hunt", "author_url": "", "post_date": "2015-10-14T22:01:00.940000", "content": "Well I've finally managed to crack 0.80000 but... its an ensemble model of 3 XGBOOSTs - I keep trying new things and not making a lot of progress... but there's still a few days yet so I am seeing what I can come up with. The list of people with better models above me keeps growing and growing... .
\n\nbtw I did try a C5.0 model (it was a bit of a struggle to get it run) but it wasn't very effective - only around .785 and I didn't even submit it in the end...
", "votes": 0, "replies": [] }, { "id": 96125, "author_name": "happycube", "author_url": "", "post_date": "2015-10-14T16:37:12.117000", "content": "I have a feeling whoever did (and I'm not one of 'em!) won't tell until next week. It's just too much of an advantage ;)
", "votes": 0, "replies": [] }, { "id": 95870, "author_name": "Randombishop", "author_url": "", "post_date": "2015-10-12T18:44:48.037000", "content": "@Loknar right, i meant without xgboost nor GBC.
", "votes": 0, "replies": [] }, { "id": 95858, "author_name": "Loknar", "author_url": "", "post_date": "2015-10-12T17:56:43.640000", "content": "i dont see any difference in AUCs in my tests between xgboost and RF/GBC from sklearn on same data, so i doubt the classifier itself makes the difference.
", "votes": 0, "replies": [] }, { "id": 96664, "author_name": "", "author_url": "", "post_date": "2015-10-19T14:59:03.283000", "content": "", "votes": 2, "replies": [] } ], "raw_markdown_by_id": { "96384": "Personally, for this competition I've been using **caret** more than ever before (setting the 'method' argument to `xgbTree` in caret's \"train\" function gets you the **xgboost** implementation). \r\n\r\nWith caret it's really easy to do cross-validation and compare different parameter settings (currently you can vary \"eta\", \"max_depth\", and \"nrounds\").\r\n\r\nIf anyone's interested, here's an example of what the code would look like:\r\n\r\n ctrl <- trainControl('cv', number=5, summaryFunction=twoClassSummary,\r\n verboseIter=TRUE, classProbs=TRUE)\r\n xgb.grid <- expand.grid(nrounds=2:20*100, max_depth=c(6, 8),\r\n eta=c(0.01, 0.035))\r\n fit.xgbTree <- train(x=training, y=target, method='xgbTree', metric='ROC',\r\n trControl=ctrl, tuneGrid=xgb.grid, subsample=0.5, \r\n colsample.bytree=0.8)", "96717": "Time to re-ask this question... especially since the private LB results are generally ~.003-005 lower! (I personally can't complain, I never cracked the top 20 on the public LB... ;) )", "96684": "[quote=KeithMessina;96668]\r\n\r\nHas everybody been rounding or transforming the data from the XGBoost to get \"1\"s and \"0\"s or have you been leaving it as-is when you predict using the XGBoost model? I've been getting much worse values when I round, but the output as-is doesn't really make sense from a yes/no decision making on whether to send marketing info to the prospects that we are predicting.\r\n\r\n[/quote]\r\n\r\nThis has been discussed before on this board, so look around, but the short version is to look at the AUC algorithm. What you are predicting is the probability that it is a 1. Therefore it allows more nuance. For example the marketing dept could decide that it is profitable to send mail at a cuttoff of 0.8 (80% sure that it is a yes), or some other threshold. The AUC takes into account that variable thresholds may be used to determine what is yes / no, so a fine grade probability is more useful than just a binary classifier as 0 or 1.", "96663": "[quote=.Григорій.;96660]\r\n\r\nYes, it's possible to get AUC=.8137 by usual Logistic Regression on validation sample after extensive data transformation work. xgBoost give AUC=.8369 in internal test.\r\n\r\nUnfortunately as a newcomer I cannot submit the results and looking for a cooperation in a team. \r\n\r\n\r\n[/quote]\r\n\r\nI recommend to double check your internal validation procedure. If your result is correct then you can be at the first place with large margin.", "96227": "I got .80+ using a single XGB.\r\n\r\nYou can get .70&change if you take this approach: I am at test[x,y] - what is the probability that the next value, be it, say, 834, is positive, and negative - tally and move to the next variable - basically, take the probability of \"what's next?\"", "96666": "@Maxx The models were evaluated correctly after scoring on hidden validation sample. \r\nSolution of such problem is a part of production process in my company.\r\n", "96660": "Yes, it's possible to get AUC=.8137 by usual Logistic Regression on validation sample after extensive data transformation work. xgBoost give AUC=.8369 in internal test.\r\n\r\nUnfortunately as a newcomer I cannot submit the results and looking for a cooperation in a team. \r\n", "97096": "Thanks a lot [quote=SteveS;96384]\r\n\r\nPersonally, for this competition I've been using **caret** more than ever before (setting the 'method' argument to `xgbTree` in caret's \"train\" function gets you the **xgboost** implementation). \r\n\r\nWith caret it's really easy to do cross-validation and compare different parameter settings (currently you can vary \"eta\", \"max_depth\", and \"nrounds\").\r\n\r\nIf anyone's interested, here's an example of what the code would look like:\r\n\r\n ctrl <- trainControl('cv', number=5, summaryFunction=twoClassSummary,\r\n verboseIter=TRUE, classProbs=TRUE)\r\n xgb.grid <- expand.grid(nrounds=2:20*100, max_depth=c(6, 8),\r\n eta=c(0.01, 0.035))\r\n fit.xgbTree <- train(x=training, y=target, method='xgbTree', metric='ROC',\r\n trControl=ctrl, tuneGrid=xgb.grid, subsample=0.5, \r\n colsample.bytree=0.8)\r\n\r\n[/quote]\r\n", "96668": "Has everybody been rounding or transforming the data from the XGBoost to get \"1\"s and \"0\"s or have you been leaving it as-is when you predict using the XGBoost model? I've been getting much worse values when I round, but the output as-is doesn't really make sense from a yes/no decision making on whether to send marketing info to the prospects that we are predicting.", "96667": "Has everybody been rounding or transforming the data from the XGBoost to get \"1\"s and \"0\"s or have you been leaving it as-is when you predict using the XGBoost model? I've been getting much worse values when I found, but the output as-is doesn't really make sense from a yes/no decision making on whether to send marketing info to the prospects that we are predicting.", "96661": "I tried many things but neither did my internal AUC improve much after 0.79xx nor did LB. \r\nGetting 0.8+ with a single model is commendable!", "96377": "First - an apology all since I think this post is definitely going off-topic for this thread!\r\n\r\nKeith - I just use xgb.cv to figure out how many rounds to run when I use xgb.train for real. Assuming you are using \"R\"... save the result and you will have a matrix and you can use which.max() function (assuming you want the max AUC - if you are using logloss for example then use which.min()) to figure out an appropriate number of rounds.\r\n\r\nRegarding RStudio - and memory - there is an excellent post in these forums from Eric Couto about using less memory to hold the data. But - as I understand XGBoost - it takes a *copy* of the \"R\" data into its own c++ data structure (i may be wrong about this but I think thats what it is doing). So it may be that the XGBoost is holding a copy of the data PLUS a copy of *all* the trees (one for each round) that it is generating. Hence the 8Gb.\r\n\r\nBut if anyone else actually *knows* how XGB is using memory I would certainly be interested in understanding it.\r\n", "96373": "You should be able to get to .79x with a 10/90 test/train split. It's a big enough training set that you can get reasonable results without CV. Try different parameters and massage the data going into xgb a bit more and you should see nice gains.\r\n\r\nMemory usage depends on how the data is unpacked and stored. A compact binary representation is about 900MB for each set IIRC.", "96370": "Haven't cracked .77 with XGBoost yet. I'm not cross-validating, which I think is my biggest issue. Can't figure out exactly how to use the xgb.cv function. \r\n\r\n2-core Celeron G1840 with 16GB RAM on Ubuntu. I wasn't able to load the full train.csv at the 8GB, so I upgraded. Anyone know why loading what RStudio calls a 1.4GB dataset takes over 8GB when that is the only thing running? I have RStudio server setup and I remote into it via browser.\r\n\r\n", "96360": "Apshreyans - I started the competition with an i5 notebook w/16Gb RAM but I now have access to an (older) i7 with 20Gb RAM. With all the features I've been adding I am definitely finding 16Gb is useful, but a depth=30 tree still takes many many hours to run. \r\n\r\nI've avoided being too selective on features because in the past I've found xgboost can deliver small increments to the evaluation criteria with even low-correlated features - and in these competitions even a small increase can gain you quite a few spots on the leaderboard - whereas in real life I would not usually bother. So I've just been adding new features and letting the algorithm decide the best ones - doesn't stop me running XGB.Importance to see what its doing though!", "96275": "Your score sounds very good for resources you have. We currently need 16GB of RAM for our solution. However, we were not doing any activity to prune uninformative features, which is not good for sure.", "96272": "@Maxx @Michael @Richard What are your PC configurations?? I am at .79776 with my i3 and 4GB RAM laptop.", "96246": "We got our current score (0.8005) with single xgboost on quite basic features.", "96204": "", "96165": "Well I've finally managed to crack 0.80000 but... its an ensemble model of 3 XGBOOSTs - I keep trying new things and not making a lot of progress... but there's still a few days yet so I am seeing what I can come up with. The list of people with better models above me keeps growing and growing...