{
  "id": 298373,
  "title": "Why is the sample_submission score -1?",
  "url": "/competitions/g-research-crypto-forecasting/discussion/298373",
  "author_name": "",
  "post_date": "2022-01-02T15:05:25.879118Z",
  "votes": 1,
  "comment_count": 2,
  "views": 0,
  "content": "<p>This competition is evaluated using Pearson's correlation coefficient.</p>\n<p>A correlation of -1 means that as your predictions increase, the target decreases, and vice-versa.</p>\n<p>Having such a forecast would be incredibly valuable, as it would mean that you could just multiply your predictions by -1 and get a perfect correlation.</p>\n<p>The sample submission is evidently not useful, as it's just zeros. Yet it achieves a correlation of -1. Why is that?</p>",
  "messages": [
    {
      "id": "1636063",
      "postDate": "01/02/2022 15:05:25",
      "content": "<p>This competition is evaluated using Pearson's correlation coefficient.</p>\n<p>A correlation of -1 means that as your predictions increase, the target decreases, and vice-versa.</p>\n<p>Having such a forecast would be incredibly valuable, as it would mean that you could just multiply your predictions by -1 and get a perfect correlation.</p>\n<p>The sample submission is evidently not useful, as it's just zeros. Yet it achieves a correlation of -1. Why is that?</p>",
      "rawMarkdown": "This competition is evaluated using Pearson's correlation coefficient.\n\nA correlation of -1 means that as your predictions increase, the target decreases, and vice-versa.\n\nHaving such a forecast would be incredibly valuable, as it would mean that you could just multiply your predictions by -1 and get a perfect correlation.\n\nThe sample submission is evidently not useful, as it's just zeros. Yet it achieves a correlation of -1. Why is that?",
      "votes": null
    },
    {
      "id": "1636147",
      "postDate": "01/02/2022 16:41:07",
      "content": "<p>Evaluation does not exactly use the correlation coefficient. It is a weighted version <a href=\"https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/291845\" target=\"_blank\">as defined here</a>. If the variance of your prediction is (very close to) zero, then the metric is undefined. The way the hosts are dealing with this is to give a score of -1, presumably since this is the score corresponding to the bottom of the range [-1, 1]. A score of -1 should not be interpreted as a correlation of -1.*</p>\n<p>*unless you are making non-constant predictions - in which case you've hit the jackpot!</p>",
      "rawMarkdown": "Evaluation does not exactly use the correlation coefficient. It is a weighted version [as defined here](https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/291845). If the variance of your prediction is (very close to) zero, then the metric is undefined. The way the hosts are dealing with this is to give a score of -1, presumably since this is the score corresponding to the bottom of the range [-1, 1]. A score of -1 should not be interpreted as a correlation of -1.*\n\n\\*unless you are making non-constant predictions - in which case you've hit the jackpot!",
      "votes": null
    },
    {
      "id": "1636283",
      "postDate": "01/02/2022 18:19:30",
      "content": "<p>Makes sense, so <code>-1</code> is an indication of invalid predictions</p>",
      "rawMarkdown": "Makes sense, so `-1` is an indication of invalid predictions",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1636147,
      "author_name": "jagofc",
      "author_url": "",
      "post_date": "01/02/2022 16:41:07",
      "content": "<p>Evaluation does not exactly use the correlation coefficient. It is a weighted version <a href=\"https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/291845\" target=\"_blank\">as defined here</a>. If the variance of your prediction is (very close to) zero, then the metric is undefined. The way the hosts are dealing with this is to give a score of -1, presumably since this is the score corresponding to the bottom of the range [-1, 1]. A score of -1 should not be interpreted as a correlation of -1.*</p>\n<p>*unless you are making non-constant predictions - in which case you've hit the jackpot!</p>",
      "votes": null,
      "replies": [
        {
          "id": 1636283,
          "author_name": "marcogorelli",
          "author_url": "",
          "post_date": "01/02/2022 18:19:30",
          "content": "<p>Makes sense, so <code>-1</code> is an indication of invalid predictions</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "1636063": "This competition is evaluated using Pearson's correlation coefficient.\n\nA correlation of -1 means that as your predictions increase, the target decreases, and vice-versa.\n\nHaving such a forecast would be incredibly valuable, as it would mean that you could just multiply your predictions by -1 and get a perfect correlation.\n\nThe sample submission is evidently not useful, as it's just zeros. Yet it achieves a correlation of -1. Why is that?",
    "1636147": "Evaluation does not exactly use the correlation coefficient. It is a weighted version [as defined here](https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/291845). If the variance of your prediction is (very close to) zero, then the metric is undefined. The way the hosts are dealing with this is to give a score of -1, presumably since this is the score corresponding to the bottom of the range [-1, 1]. A score of -1 should not be interpreted as a correlation of -1.*\n\n\\*unless you are making non-constant predictions - in which case you've hit the jackpot!",
    "1636283": "Makes sense, so `-1` is an indication of invalid predictions"
  },
  "source": "meta"
}