I'm new to Kaggle competitions and I'm trying my hand with xgboost. I'm trying to use the XGBoost CLI to produce a ranking of the top k (k=3) place_ids based on a small xy piece of the domain. Unfortunately I'm getting a list of floats instead of a ranked list of place_ids. My config file is below (and attached).
\n\nI've created train and validation files with the format
\n\n{place_id} 1:x 2:y 3:accuracy 4:day_of_week 5: hour\nAnd a test file with the format
\n\n1:x 2:y 3:accuracy 4:day_of_week 5: hour\nOf course, I've created group files for the train and validation, and not for the test, as I don't have that information.
\n\nWhen I run with xgboost conf && xgboost conf task=pred model_in=0006.model, I get a pred.txt that has one column of floats, one line per item in the bin2d.test input. How do I get a ranked list of k=3 place_ids instead?
--
\n\n# General Parameters, see comment for each definition\n\nbooster="gbtree"\n\n# specify objective\nobjective="rank:pairwise"\neval_metric = "map@3-"\n\n\n# Tree Booster Parameters\n# step size shrinkage\neta = 0.1\n# minimum loss reduction required to make a further partition\ngamma = 1.0\n# minimum sum of instance weight(hessian) needed in a child\nmin_child_weight = 0.1\n# maximum depth of a tree\nmax_depth = 6\n\nn_estimators=100\nn_thread=-1\nseed=1337\nsubsample=1\n\n# Task parameters\n# the number of round to do boosting\nnum_round = 6\n# 0 means do not save any model except the final round model\nsave_period = 0\n# The path of training data\ndata = "bin2d.train"\n# The path of validation data, used to monitor training process, here [test] sets name of the validation set\neval[test] = "bin2d.validation"\n# The path of test data\ntest:data = "bin2d.test"\n--
", "messages": [ { "id": "121017", "postDate": "05/22/2016 22:02:59", "content": "I'm new to Kaggle competitions and I'm trying my hand with xgboost. I'm trying to use the XGBoost CLI to produce a ranking of the top k (k=3) place_ids based on a small xy piece of the domain. Unfortunately I'm getting a list of floats instead of a ranked list of place_ids. My config file is below (and attached).
\n\nI've created train and validation files with the format
\n\n{place_id} 1:x 2:y 3:accuracy 4:day_of_week 5: hour\nAnd a test file with the format
\n\n1:x 2:y 3:accuracy 4:day_of_week 5: hour\nOf course, I've created group files for the train and validation, and not for the test, as I don't have that information.
\n\nWhen I run with xgboost conf && xgboost conf task=pred model_in=0006.model, I get a pred.txt that has one column of floats, one line per item in the bin2d.test input. How do I get a ranked list of k=3 place_ids instead?
--
\n\n# General Parameters, see comment for each definition\n\nbooster="gbtree"\n\n# specify objective\nobjective="rank:pairwise"\neval_metric = "map@3-"\n\n\n# Tree Booster Parameters\n# step size shrinkage\neta = 0.1\n# minimum loss reduction required to make a further partition\ngamma = 1.0\n# minimum sum of instance weight(hessian) needed in a child\nmin_child_weight = 0.1\n# maximum depth of a tree\nmax_depth = 6\n\nn_estimators=100\nn_thread=-1\nseed=1337\nsubsample=1\n\n# Task parameters\n# the number of round to do boosting\nnum_round = 6\n# 0 means do not save any model except the final round model\nsave_period = 0\n# The path of training data\ndata = "bin2d.train"\n# The path of validation data, used to monitor training process, here [test] sets name of the validation set\neval[test] = "bin2d.validation"\n# The path of test data\ntest:data = "bin2d.test"\n--
", "rawMarkdown": "I'm new to Kaggle competitions and I'm trying my hand with xgboost. I'm trying to use the XGBoost CLI to produce a ranking of the top k (k=3) place_ids based on a small xy piece of the domain. Unfortunately I'm getting a list of floats instead of a ranked list of place_ids. My config file is below (and attached).\r\n\r\nI've created train and validation files with the format\r\n\r\n {place_id} 1:x 2:y 3:accuracy 4:day_of_week 5: hour\r\n\r\nAnd a test file with the format\r\n\r\n 1:x 2:y 3:accuracy 4:day_of_week 5: hour\r\n\r\nOf course, I've created group files for the train and validation, and not for the test, as I don't have that information. \r\n\r\nWhen I run with `xgboost conf && xgboost conf task=pred model_in=0006.model`, I get a pred.txt that has one column of floats, one line per item in the bin2d.test input. How do I get a ranked list of k=3 place_ids instead?\r\n\r\n--\r\n\r\n # General Parameters, see comment for each definition\r\n \r\n booster=\"gbtree\"\r\n \r\n # specify objective\r\n objective=\"rank:pairwise\"\r\n eval_metric = \"map@3-\"\r\n \r\n \r\n # Tree Booster Parameters\r\n # step size shrinkage\r\n eta = 0.1\r\n # minimum loss reduction required to make a further partition\r\n gamma = 1.0\r\n # minimum sum of instance weight(hessian) needed in a child\r\n min_child_weight = 0.1\r\n # maximum depth of a tree\r\n max_depth = 6\r\n \r\n n_estimators=100\r\n n_thread=-1\r\n seed=1337\r\n subsample=1\r\n \r\n # Task parameters\r\n # the number of round to do boosting\r\n num_round = 6\r\n # 0 means do not save any model except the final round model\r\n save_period = 0\r\n # The path of training data\r\n data = \"bin2d.train\"\r\n # The path of validation data, used to monitor training process, here [test] sets name of the validation set\r\n eval[test] = \"bin2d.validation\"\r\n # The path of test data\r\n test:data = \"bin2d.test\"\r\n\r\n--", "votes": null }, { "id": "121025", "postDate": "05/23/2016 02:19:06", "content": "Here is how I do it:
\n\ndef get_top3_places_and_probs(row):\n row.sort()\n inds = row.index[-3:][::-1].tolist()\n return inds \ny_pred = xgb_classifier.predict(dtest)\nresult_xgb_df = pd.DataFrame(index=test.row_id,data=y_pred)\nresult_xgb_df['pred'] = result_xgb_df.apply(get_top3_places_and_probs,axis=1)\nresult_xgb_df['pred_0'] = result_xgb_df['pred'].map(lambda x: x[0])\nresult_xgb_df['pred_1'] = result_xgb_df['pred'].map(lambda x: x[1])\nresult_xgb_df['pred_2'] = result_xgb_df['pred'].map(lambda x: x[2])\nI tried a similar approach of sorting by the probabilities with the classifiers in the python sci-kit tool kit but even with a subset of 10^6 of the training data, there are too many distinct place_ids'.
\n\nSo the cols (or rows) the predicted probability matrix grows prohibitively large and it runs out of memory even with a 256G node.
\n\nSo I was just curious as to what proportion of the training set does the above code handle?
", "rawMarkdown": "I tried a similar approach of sorting by the probabilities with the classifiers in the python sci-kit tool kit but even with a subset of 10^6 of the training data, there are too many distinct place_ids'. \r\n\r\nSo the cols (or rows) the predicted probability matrix grows prohibitively large and it runs out of memory even with a 256G node. \r\n\r\nSo I was just curious as to what proportion of the training set does the above code handle?", "votes": null }, { "id": "121085", "postDate": "05/23/2016 18:10:56", "content": "Thanks so much @jturkewitz. Although I didn't end up using your implementation, the pointer to use xgb_classifier was what I needed to get my own implementation working.
Out of curiosity, what objective are you using? I have something working with "multi:softprob", but cannot seem to get it to work out with "rank:pairwise". I end up executing something like this.
\n\nmat1 = xgboost.DMatrix(x1, label=y1)\nbst = xgboost.train({'objective': 'multi:softprob', 'num_class': 122, 'eval_metric':'map@3-'}, mat1)\n@srimalj -- I separate the dataset into horizontal bins (e.g. 5 ≤ x ≤ 5.5, 3 ≤ y ≤ 3.25). Within each bin, I run a separate classifier, excluding all place_ids which don't have at least q observations in the bin from the classifier. With some careful choices of q and bin size, you can make the classification problem tractable. I think my classifier varies between 80 and 200 place_ids per bin, depending on the bin, with memory usage between 2 and 5GB.
", "rawMarkdown": "Thanks so much @jturkewitz. Although I didn't end up using your implementation, the pointer to use `xgb_classifier` was what I needed to get my own implementation working. \r\n\r\nOut of curiosity, what objective are you using? I have something working with \"multi:softprob\", but cannot seem to get it to work out with \"rank:pairwise\". I end up executing something like this.\r\n\r\n mat1 = xgboost.DMatrix(x1, label=y1)\r\n bst = xgboost.train({'objective': 'multi:softprob', 'num_class': 122, 'eval_metric':'map@3-'}, mat1)\r\n\r\n----------\r\n@srimalj -- I separate the dataset into horizontal bins (e.g. 5 ≤ x ≤ 5.5, 3 ≤ y ≤ 3.25). Within each bin, I run a separate classifier, excluding all place_ids which don't have at least q observations in the bin from the classifier. With some careful choices of q and bin size, you can make the classification problem tractable. I think my classifier varies between 80 and 200 place_ids per bin, depending on the bin, with memory usage between 2 and 5GB.", "votes": null }, { "id": "121103", "postDate": "05/23/2016 23:05:47", "content": "For rank:pairwise, xgboost expects you to:
\n\nIn this case you do not have any group, therefore everything is taken as a whole group. What xgboost gives, in any case, is a value which has to be interpreted on the whole group. There is no clear threshold between values: you only know one value can be higher than the other.
\n\nFor instance, if you have:
\n\nLabel Pred\n1 -0.5\n2 1.32\n3 1.87\n4 2.22\n5 2.98\n6 3.3\n7 3.27\n8 4.2\nBy ordinality of ranking, you know xgboost predicted the order {1, 2, 3, 4, 5, 7, 6, 8}. But that's it, nothing more.
\n\nSuppose the following:
\n\nGroup Label Pred\n1 1 -0.5\n1 2 1.32\n1 3 1.87\n1 4 2.22\n1 5 2.98\n2 1 2.22\n2 2 2.32\n2 3 2.44\n2 4 2.28\n2 5 2.10\nYou have two groups there, so the interpretation is spread by group:
\n\nFor real pairwise (two labels) ranking, you'd rather using {-1, 1} as labels to make it clear to xgboost (even if {0, 1} works).
\n\nIn your case, do you know if:
\n\nAn ID should not be an ordinal label, unless you sort it properly (label re-(de)coding).
\n\nIf you got ties (identical labels), it starts to be painful to infer. For instance...:
\n\nLabel Pred\n1 -0.5\n2 1.32\n2 1.87\n2 2.22\n3 2.98\n3 3.3\n3 3.27\n4 4.2\n... could be interpreted as {1, 2, 3, 4, 5, 6.5, 6.5, 8}, which would be "far from the reality".
", "rawMarkdown": "For rank:pairwise, xgboost expects you to:\r\n\r\n* Provide the groups for ranking (per group ranking) using xgb.DMatrix (set_group I think)\r\n* The **ordinal labels** of each group (it \"doesn't care\" about the values you put inside, although it can specify (not sure?) how far can be each ordinal value)\r\n\r\nIn this case you do not have any group, therefore everything is taken as a whole group. What xgboost gives, in any case, is a value which has to be interpreted on the whole group. There is no clear threshold between values: you only know one value can be higher than the other.\r\n\r\nFor instance, if you have:\r\n\r\n Label Pred\r\n 1 -0.5\r\n 2 1.32\r\n 3 1.87\r\n 4 2.22\r\n 5 2.98\r\n 6 3.3\r\n 7 3.27\r\n 8 4.2\r\n\r\nBy ordinality of ranking, you know xgboost predicted the order {1, 2, 3, 4, 5, 7, 6, 8}. But that's it, nothing more.\r\n\r\nSuppose the following:\r\n\r\n Group Label Pred\r\n 1 1 -0.5\r\n 1 2 1.32\r\n 1 3 1.87\r\n 1 4 2.22\r\n 1 5 2.98\r\n 2 1 2.22\r\n 2 2 2.32\r\n 2 3 2.44\r\n 2 4 2.28\r\n 2 5 2.10\r\n\r\nYou have two groups there, so the interpretation is spread by group:\r\n\r\n* Group 1: xgboost predicted {1, 2, 3, 4, 5}, which is right\r\n* Group 2: xgboost predicted {5, 1, 4, 2, 3}, which is wrong\r\n\r\nFor real pairwise (two labels) ranking, you'd rather using {-1, 1} as labels to make it clear to xgboost (even if {0, 1} works).\r\n\r\n\r\n----------\r\n\r\n\r\nIn your case, do you know if:\r\n\r\n* rank:pairwise is appropriate?\r\n* If it is appropriate, why?\r\n* **Do you have ordinal labels?** (= ordered labels, i.e 1 is before 2, 2 is before 3, 2482 is before 2894928, etc.)\r\n* **Are you able to re-map the predictions to the labels?** (prediction value in the group => label in the group)\r\n* **Do you know the structure of the test set?** (how many specific labels there are per group: in this case, how many times each label appears in the test set?)\r\n\r\nAn ID should not be an ordinal label, unless you sort it properly (**label re-(de)coding**).\r\n\r\nIf you got ties (identical labels), it starts to be painful to infer. For instance...:\r\n\r\n Label Pred\r\n 1 -0.5\r\n 2 1.32\r\n 2 1.87\r\n 2 2.22\r\n 3 2.98\r\n 3 3.3\r\n 3 3.27\r\n 4 4.2\r\n\r\n... could be interpreted as {1, 2, 3, 4, 5, 6.5, 6.5, 8}, which would be \"far from the reality\".", "votes": null }, { "id": "121174", "postDate": "05/24/2016 18:48:29", "content": "Thanks @Laurae2. You're correct, I did not have an understanding of what objective function to use, and I was trying to use rank:pairwise simply because that's what was in the xgboost rank example. I was specifying groups manually using set_group, but I do not have ordinal labels in this case, and I did not realize that was a requirement.
\n\nAs you put it, I definitely do not have test set information, so I understand why rank:pairwise is not appropriate now. Thanks!
", "rawMarkdown": "Thanks @Laurae2. You're correct, I did not have an understanding of what objective function to use, and I was trying to use rank:pairwise simply because that's what was in the xgboost rank example. I was specifying groups manually using set_group, but I do not have ordinal labels in this case, and I did not realize that was a requirement. \r\n\r\nAs you put it, I definitely do not have test set information, so I understand why rank:pairwise is not appropriate now. Thanks!", "votes": null } ], "comments": [ { "id": 121025, "author_name": "jturkewitz", "author_url": "", "post_date": "05/23/2016 02:19:06", "content": "Here is how I do it:
\n\ndef get_top3_places_and_probs(row):\n row.sort()\n inds = row.index[-3:][::-1].tolist()\n return inds \ny_pred = xgb_classifier.predict(dtest)\nresult_xgb_df = pd.DataFrame(index=test.row_id,data=y_pred)\nresult_xgb_df['pred'] = result_xgb_df.apply(get_top3_places_and_probs,axis=1)\nresult_xgb_df['pred_0'] = result_xgb_df['pred'].map(lambda x: x[0])\nresult_xgb_df['pred_1'] = result_xgb_df['pred'].map(lambda x: x[1])\nresult_xgb_df['pred_2'] = result_xgb_df['pred'].map(lambda x: x[2])\nI tried a similar approach of sorting by the probabilities with the classifiers in the python sci-kit tool kit but even with a subset of 10^6 of the training data, there are too many distinct place_ids'.
\n\nSo the cols (or rows) the predicted probability matrix grows prohibitively large and it runs out of memory even with a 256G node.
\n\nSo I was just curious as to what proportion of the training set does the above code handle?
", "votes": null, "replies": [] }, { "id": 121085, "author_name": "rshekhar2", "author_url": "", "post_date": "05/23/2016 18:10:56", "content": "Thanks so much @jturkewitz. Although I didn't end up using your implementation, the pointer to use xgb_classifier was what I needed to get my own implementation working.
Out of curiosity, what objective are you using? I have something working with "multi:softprob", but cannot seem to get it to work out with "rank:pairwise". I end up executing something like this.
\n\nmat1 = xgboost.DMatrix(x1, label=y1)\nbst = xgboost.train({'objective': 'multi:softprob', 'num_class': 122, 'eval_metric':'map@3-'}, mat1)\n@srimalj -- I separate the dataset into horizontal bins (e.g. 5 ≤ x ≤ 5.5, 3 ≤ y ≤ 3.25). Within each bin, I run a separate classifier, excluding all place_ids which don't have at least q observations in the bin from the classifier. With some careful choices of q and bin size, you can make the classification problem tractable. I think my classifier varies between 80 and 200 place_ids per bin, depending on the bin, with memory usage between 2 and 5GB.
", "votes": null, "replies": [] }, { "id": 121103, "author_name": "laurae2", "author_url": "", "post_date": "05/23/2016 23:05:47", "content": "For rank:pairwise, xgboost expects you to:
\n\nIn this case you do not have any group, therefore everything is taken as a whole group. What xgboost gives, in any case, is a value which has to be interpreted on the whole group. There is no clear threshold between values: you only know one value can be higher than the other.
\n\nFor instance, if you have:
\n\nLabel Pred\n1 -0.5\n2 1.32\n3 1.87\n4 2.22\n5 2.98\n6 3.3\n7 3.27\n8 4.2\nBy ordinality of ranking, you know xgboost predicted the order {1, 2, 3, 4, 5, 7, 6, 8}. But that's it, nothing more.
\n\nSuppose the following:
\n\nGroup Label Pred\n1 1 -0.5\n1 2 1.32\n1 3 1.87\n1 4 2.22\n1 5 2.98\n2 1 2.22\n2 2 2.32\n2 3 2.44\n2 4 2.28\n2 5 2.10\nYou have two groups there, so the interpretation is spread by group:
\n\nFor real pairwise (two labels) ranking, you'd rather using {-1, 1} as labels to make it clear to xgboost (even if {0, 1} works).
\n\nIn your case, do you know if:
\n\nAn ID should not be an ordinal label, unless you sort it properly (label re-(de)coding).
\n\nIf you got ties (identical labels), it starts to be painful to infer. For instance...:
\n\nLabel Pred\n1 -0.5\n2 1.32\n2 1.87\n2 2.22\n3 2.98\n3 3.3\n3 3.27\n4 4.2\n... could be interpreted as {1, 2, 3, 4, 5, 6.5, 6.5, 8}, which would be "far from the reality".
", "votes": null, "replies": [] }, { "id": 121174, "author_name": "rshekhar2", "author_url": "", "post_date": "05/24/2016 18:48:29", "content": "Thanks @Laurae2. You're correct, I did not have an understanding of what objective function to use, and I was trying to use rank:pairwise simply because that's what was in the xgboost rank example. I was specifying groups manually using set_group, but I do not have ordinal labels in this case, and I did not realize that was a requirement.
\n\nAs you put it, I definitely do not have test set information, so I understand why rank:pairwise is not appropriate now. Thanks!
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "121017": "I'm new to Kaggle competitions and I'm trying my hand with xgboost. I'm trying to use the XGBoost CLI to produce a ranking of the top k (k=3) place_ids based on a small xy piece of the domain. Unfortunately I'm getting a list of floats instead of a ranked list of place_ids. My config file is below (and attached).\r\n\r\nI've created train and validation files with the format\r\n\r\n {place_id} 1:x 2:y 3:accuracy 4:day_of_week 5: hour\r\n\r\nAnd a test file with the format\r\n\r\n 1:x 2:y 3:accuracy 4:day_of_week 5: hour\r\n\r\nOf course, I've created group files for the train and validation, and not for the test, as I don't have that information. \r\n\r\nWhen I run with `xgboost conf && xgboost conf task=pred model_in=0006.model`, I get a pred.txt that has one column of floats, one line per item in the bin2d.test input. How do I get a ranked list of k=3 place_ids instead?\r\n\r\n--\r\n\r\n # General Parameters, see comment for each definition\r\n \r\n booster=\"gbtree\"\r\n \r\n # specify objective\r\n objective=\"rank:pairwise\"\r\n eval_metric = \"map@3-\"\r\n \r\n \r\n # Tree Booster Parameters\r\n # step size shrinkage\r\n eta = 0.1\r\n # minimum loss reduction required to make a further partition\r\n gamma = 1.0\r\n # minimum sum of instance weight(hessian) needed in a child\r\n min_child_weight = 0.1\r\n # maximum depth of a tree\r\n max_depth = 6\r\n \r\n n_estimators=100\r\n n_thread=-1\r\n seed=1337\r\n subsample=1\r\n \r\n # Task parameters\r\n # the number of round to do boosting\r\n num_round = 6\r\n # 0 means do not save any model except the final round model\r\n save_period = 0\r\n # The path of training data\r\n data = \"bin2d.train\"\r\n # The path of validation data, used to monitor training process, here [test] sets name of the validation set\r\n eval[test] = \"bin2d.validation\"\r\n # The path of test data\r\n test:data = \"bin2d.test\"\r\n\r\n--", "121025": "Here is how I do it:\r\n \r\n\r\n\r\n\r\n def get_top3_places_and_probs(row):\r\n row.sort()\r\n inds = row.index[-3:][::-1].tolist()\r\n return inds \r\n y_pred = xgb_classifier.predict(dtest)\r\n result_xgb_df = pd.DataFrame(index=test.row_id,data=y_pred)\r\n result_xgb_df['pred'] = result_xgb_df.apply(get_top3_places_and_probs,axis=1)\r\n result_xgb_df['pred_0'] = result_xgb_df['pred'].map(lambda x: x[0])\r\n result_xgb_df['pred_1'] = result_xgb_df['pred'].map(lambda x: x[1])\r\n result_xgb_df['pred_2'] = result_xgb_df['pred'].map(lambda x: x[2])", "121031": "I tried a similar approach of sorting by the probabilities with the classifiers in the python sci-kit tool kit but even with a subset of 10^6 of the training data, there are too many distinct place_ids'. \r\n\r\nSo the cols (or rows) the predicted probability matrix grows prohibitively large and it runs out of memory even with a 256G node. \r\n\r\nSo I was just curious as to what proportion of the training set does the above code handle?", "121085": "Thanks so much @jturkewitz. Although I didn't end up using your implementation, the pointer to use `xgb_classifier` was what I needed to get my own implementation working. \r\n\r\nOut of curiosity, what objective are you using? I have something working with \"multi:softprob\", but cannot seem to get it to work out with \"rank:pairwise\". I end up executing something like this.\r\n\r\n mat1 = xgboost.DMatrix(x1, label=y1)\r\n bst = xgboost.train({'objective': 'multi:softprob', 'num_class': 122, 'eval_metric':'map@3-'}, mat1)\r\n\r\n----------\r\n@srimalj -- I separate the dataset into horizontal bins (e.g. 5 ≤ x ≤ 5.5, 3 ≤ y ≤ 3.25). Within each bin, I run a separate classifier, excluding all place_ids which don't have at least q observations in the bin from the classifier. With some careful choices of q and bin size, you can make the classification problem tractable. I think my classifier varies between 80 and 200 place_ids per bin, depending on the bin, with memory usage between 2 and 5GB.", "121103": "For rank:pairwise, xgboost expects you to:\r\n\r\n* Provide the groups for ranking (per group ranking) using xgb.DMatrix (set_group I think)\r\n* The **ordinal labels** of each group (it \"doesn't care\" about the values you put inside, although it can specify (not sure?) how far can be each ordinal value)\r\n\r\nIn this case you do not have any group, therefore everything is taken as a whole group. What xgboost gives, in any case, is a value which has to be interpreted on the whole group. There is no clear threshold between values: you only know one value can be higher than the other.\r\n\r\nFor instance, if you have:\r\n\r\n Label Pred\r\n 1 -0.5\r\n 2 1.32\r\n 3 1.87\r\n 4 2.22\r\n 5 2.98\r\n 6 3.3\r\n 7 3.27\r\n 8 4.2\r\n\r\nBy ordinality of ranking, you know xgboost predicted the order {1, 2, 3, 4, 5, 7, 6, 8}. But that's it, nothing more.\r\n\r\nSuppose the following:\r\n\r\n Group Label Pred\r\n 1 1 -0.5\r\n 1 2 1.32\r\n 1 3 1.87\r\n 1 4 2.22\r\n 1 5 2.98\r\n 2 1 2.22\r\n 2 2 2.32\r\n 2 3 2.44\r\n 2 4 2.28\r\n 2 5 2.10\r\n\r\nYou have two groups there, so the interpretation is spread by group:\r\n\r\n* Group 1: xgboost predicted {1, 2, 3, 4, 5}, which is right\r\n* Group 2: xgboost predicted {5, 1, 4, 2, 3}, which is wrong\r\n\r\nFor real pairwise (two labels) ranking, you'd rather using {-1, 1} as labels to make it clear to xgboost (even if {0, 1} works).\r\n\r\n\r\n----------\r\n\r\n\r\nIn your case, do you know if:\r\n\r\n* rank:pairwise is appropriate?\r\n* If it is appropriate, why?\r\n* **Do you have ordinal labels?** (= ordered labels, i.e 1 is before 2, 2 is before 3, 2482 is before 2894928, etc.)\r\n* **Are you able to re-map the predictions to the labels?** (prediction value in the group => label in the group)\r\n* **Do you know the structure of the test set?** (how many specific labels there are per group: in this case, how many times each label appears in the test set?)\r\n\r\nAn ID should not be an ordinal label, unless you sort it properly (**label re-(de)coding**).\r\n\r\nIf you got ties (identical labels), it starts to be painful to infer. For instance...:\r\n\r\n Label Pred\r\n 1 -0.5\r\n 2 1.32\r\n 2 1.87\r\n 2 2.22\r\n 3 2.98\r\n 3 3.3\r\n 3 3.27\r\n 4 4.2\r\n\r\n... could be interpreted as {1, 2, 3, 4, 5, 6.5, 6.5, 8}, which would be \"far from the reality\".", "121174": "Thanks @Laurae2. You're correct, I did not have an understanding of what objective function to use, and I was trying to use rank:pairwise simply because that's what was in the xgboost rank example. I was specifying groups manually using set_group, but I do not have ordinal labels in this case, and I did not realize that was a requirement. \r\n\r\nAs you put it, I definitely do not have test set information, so I understand why rank:pairwise is not appropriate now. Thanks!" }, "source": "meta" }