{
  "id": 98689,
  "title": "Objective Function different to Evaluation Metric",
  "url": "/competitions/champs-scalar-coupling/discussion/98689",
  "author_name": "",
  "post_date": "2019-07-05T14:01:15.985394200Z",
  "votes": null,
  "comment_count": 6,
  "views": 0,
  "content": "<p>Starting a discussion here after seeing lots of successful kernels using RMSE/L2 as LightGBM objective function despite our evaluation being based on MAE/L1. Does anybody have any insight here as to a) whether RMSE is consistently producing better scores and b) why that might be? </p>",
  "messages": [
    {
      "id": "568833",
      "postDate": "07/05/2019 14:01:15",
      "content": "<p>Starting a discussion here after seeing lots of successful kernels using RMSE/L2 as LightGBM objective function despite our evaluation being based on MAE/L1. Does anybody have any insight here as to a) whether RMSE is consistently producing better scores and b) why that might be? </p>",
      "rawMarkdown": "Starting a discussion here after seeing lots of successful kernels using RMSE/L2 as LightGBM objective function despite our evaluation being based on MAE/L1. Does anybody have any insight here as to a) whether RMSE is consistently producing better scores and b) why that might be?",
      "votes": null
    },
    {
      "id": "568836",
      "postDate": "07/05/2019 14:11:38",
      "content": "<p>I read this article a couple of years ago and I found it really interesting.\nIt might provide you with particular insights.\n<a href=\"https://medium.com/human-in-a-machine-world/mae-and-rmse-which-metric-is-better-e60ac3bde13d\">https://medium.com/human-in-a-machine-world/mae-and-rmse-which-metric-is-better-e60ac3bde13d</a></p>",
      "rawMarkdown": "I read this article a couple of years ago and I found it really interesting.\nIt might provide you with particular insights.\nhttps://medium.com/human-in-a-machine-world/mae-and-rmse-which-metric-is-better-e60ac3bde13d",
      "votes": null
    },
    {
      "id": "568984",
      "postDate": "07/05/2019 19:44:53",
      "content": "<p>Huber also works good, gamma - a bit worse. For my features at least</p>",
      "rawMarkdown": "Huber also works good, gamma - a bit worse. For my features at least",
      "votes": null
    },
    {
      "id": "569072",
      "postDate": "07/06/2019 02:11:59",
      "content": "<p>Under the hood MAE, obviosuly, looks at the mean error while RMSE is more \"interested\" in the median.</p>",
      "rawMarkdown": "Under the hood MAE, obviosuly, looks at the mean error while RMSE is more \"interested\" in the median.",
      "votes": null
    },
    {
      "id": "569220",
      "postDate": "07/06/2019 09:07:11",
      "content": "<p>That was a useful article that addresses OPs question... Since in this competition the types have different numbers of cases, MAE is more meaningful as RMSE would be bigger for the types with more cases.</p>",
      "rawMarkdown": "That was a useful article that addresses OPs question... Since in this competition the types have different numbers of cases, MAE is more meaningful as RMSE would be bigger for the types with more cases.",
      "votes": null
    },
    {
      "id": "571265",
      "postDate": "07/09/2019 12:09:23",
      "content": "<p>Issue with MAE as objective (or loss function) is that it has a constant derivative except at 0 where the derivative is not continuous, and even undefined.  This makes gradient descent ineffective.  Using other objective or loss functions is often better.</p>",
      "rawMarkdown": "Issue with MAE as objective (or loss function) is that it has a constant derivative except at 0 where the derivative is not continuous, and even undefined.  This makes gradient descent ineffective.  Using other objective or loss functions is often better.",
      "votes": null
    },
    {
      "id": "571312",
      "postDate": "07/09/2019 13:39:47",
      "content": "<p>It is the opposite actually.  If your prediction is constant, then MAE is minimal when using median, and RMSE is minimal when using mean.</p>",
      "rawMarkdown": "It is the opposite actually.  If your prediction is constant, then MAE is minimal when using median, and RMSE is minimal when using mean.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 568836,
      "author_name": "jolasa",
      "author_url": "",
      "post_date": "07/05/2019 14:11:38",
      "content": "<p>I read this article a couple of years ago and I found it really interesting.\nIt might provide you with particular insights.\n<a href=\"https://medium.com/human-in-a-machine-world/mae-and-rmse-which-metric-is-better-e60ac3bde13d\">https://medium.com/human-in-a-machine-world/mae-and-rmse-which-metric-is-better-e60ac3bde13d</a></p>",
      "votes": null,
      "replies": [
        {
          "id": 569220,
          "author_name": "phucco",
          "author_url": "",
          "post_date": "07/06/2019 09:07:11",
          "content": "<p>That was a useful article that addresses OPs question... Since in this competition the types have different numbers of cases, MAE is more meaningful as RMSE would be bigger for the types with more cases.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 568984,
      "author_name": "stanislavblinov",
      "author_url": "",
      "post_date": "07/05/2019 19:44:53",
      "content": "<p>Huber also works good, gamma - a bit worse. For my features at least</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 569072,
      "author_name": "snovik1975",
      "author_url": "",
      "post_date": "07/06/2019 02:11:59",
      "content": "<p>Under the hood MAE, obviosuly, looks at the mean error while RMSE is more \"interested\" in the median.</p>",
      "votes": null,
      "replies": [
        {
          "id": 571312,
          "author_name": "cpmpml",
          "author_url": "",
          "post_date": "07/09/2019 13:39:47",
          "content": "<p>It is the opposite actually.  If your prediction is constant, then MAE is minimal when using median, and RMSE is minimal when using mean.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 571265,
      "author_name": "cpmpml",
      "author_url": "",
      "post_date": "07/09/2019 12:09:23",
      "content": "<p>Issue with MAE as objective (or loss function) is that it has a constant derivative except at 0 where the derivative is not continuous, and even undefined.  This makes gradient descent ineffective.  Using other objective or loss functions is often better.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "568833": "Starting a discussion here after seeing lots of successful kernels using RMSE/L2 as LightGBM objective function despite our evaluation being based on MAE/L1. Does anybody have any insight here as to a) whether RMSE is consistently producing better scores and b) why that might be?",
    "568836": "I read this article a couple of years ago and I found it really interesting.\nIt might provide you with particular insights.\nhttps://medium.com/human-in-a-machine-world/mae-and-rmse-which-metric-is-better-e60ac3bde13d",
    "568984": "Huber also works good, gamma - a bit worse. For my features at least",
    "569072": "Under the hood MAE, obviosuly, looks at the mean error while RMSE is more \"interested\" in the median.",
    "569220": "That was a useful article that addresses OPs question... Since in this competition the types have different numbers of cases, MAE is more meaningful as RMSE would be bigger for the types with more cases.",
    "571265": "Issue with MAE as objective (or loss function) is that it has a constant derivative except at 0 where the derivative is not continuous, and even undefined.  This makes gradient descent ineffective.  Using other objective or loss functions is often better.",
    "571312": "It is the opposite actually.  If your prediction is constant, then MAE is minimal when using median, and RMSE is minimal when using mean."
  },
  "source": "meta"
}