{
  "id": 9753,
  "title": "Maybe, a better evaluation metric",
  "url": "/competitions/acquire-valued-shoppers-challenge/discussion/9753",
  "author_name": "",
  "post_date": "2014-07-15T03:17:46.050Z",
  "votes": 1,
  "comment_count": 3,
  "views": 1506,
  "content": "<p>The evaluation metric here is the global AUC, which means the global ranking matters. However, does it make sense for the business? The global AUC implies that we need to rank not only how user A, B like item X, but also how user C likes item Y vs&nbsp;user D likes item Z. Is that necessary? In my opinion, it would be more useful to rank users given an offer. Accordingly, the performance metric would be mean of AUC_i's, where AUC_i is the AUC for offer i. This is a relaxing of the global ranking, which makes models more powerful. It would belong to learning to rank problems.</p>",
  "messages": [
    {
      "id": "50529",
      "postDate": "07/15/2014 03:17:46",
      "content": "<p>The evaluation metric here is the global AUC, which means the global ranking matters. However, does it make sense for the business? The global AUC implies that we need to rank not only how user A, B like item X, but also how user C likes item Y vs&nbsp;user D likes item Z. Is that necessary? In my opinion, it would be more useful to rank users given an offer. Accordingly, the performance metric would be mean of AUC_i's, where AUC_i is the AUC for offer i. This is a relaxing of the global ranking, which makes models more powerful. It would belong to learning to rank problems.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "50539",
      "postDate": "07/15/2014 05:52:34",
      "content": "<p>Agreed, the metric you suggest would have made validation easier. The global AUC is effected by within and between offer variation. The between offer variation made&nbsp;AUC unstable&nbsp;in the trainset and between train and test set. Within the test set, AUC seemed a bit more stable, probably because 60% of the records are on the same offered product.</p>\n<p>For the sponsor,&nbsp;an overall&nbsp;ranking/auc&nbsp;may be useful to decide what product to put on offer?&nbsp;But then the number of offered products was very small to model that well.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "50601",
      "postDate": "07/15/2014 18:13:08",
      "content": "<p>Gert, Congratulations!&nbsp;</p>\n<p>Average AUC or other ranking metrics are just suggestions for the sponsor, in case they are more meaningful in terms of their&nbsp;business goals.&nbsp;</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "50606",
      "postDate": "07/15/2014 19:38:08",
      "content": "<p>They probably want to distribute X&nbsp;number of coupons, and want to find the X offer-customer combinations that are most likely to result in&nbsp;repeat buys, so, global AUC.</p>",
      "rawMarkdown": "",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 50539,
      "author_name": "gertjac",
      "author_url": "",
      "post_date": "07/15/2014 05:52:34",
      "content": "<p>Agreed, the metric you suggest would have made validation easier. The global AUC is effected by within and between offer variation. The between offer variation made&nbsp;AUC unstable&nbsp;in the trainset and between train and test set. Within the test set, AUC seemed a bit more stable, probably because 60% of the records are on the same offered product.</p>\n<p>For the sponsor,&nbsp;an overall&nbsp;ranking/auc&nbsp;may be useful to decide what product to put on offer?&nbsp;But then the number of offered products was very small to model that well.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 50601,
      "author_name": "songgc",
      "author_url": "",
      "post_date": "07/15/2014 18:13:08",
      "content": "<p>Gert, Congratulations!&nbsp;</p>\n<p>Average AUC or other ranking metrics are just suggestions for the sponsor, in case they are more meaningful in terms of their&nbsp;business goals.&nbsp;</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 50606,
      "author_name": "byang1",
      "author_url": "",
      "post_date": "07/15/2014 19:38:08",
      "content": "<p>They probably want to distribute X&nbsp;number of coupons, and want to find the X offer-customer combinations that are most likely to result in&nbsp;repeat buys, so, global AUC.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "50529": "",
    "50539": "",
    "50601": "",
    "50606": ""
  },
  "source": "meta"
}