{
  "id": 15689,
  "title": "Calculating KS and CVM with predictions only",
  "url": "/competitions/flavours-of-physics/discussion/15689",
  "author_name": "",
  "post_date": "2015-08-01T01:30:03.800Z",
  "votes": null,
  "comment_count": 2,
  "views": 460,
  "content": "<p>The current code available for testing KS and CVM scores require the classifier to make predictions on 'test' datasets.  Is there a way to calculate these scores without a classifier?  I.e. using only the predictions of the classifier, masses and real signals?  There must be a way since this is what is done when submitting, no?</p>\n\n<p>If we can have a way of calculating these scores without the classifier it opens up opportunities to have custom objective functions and other interesting things that can be used during model creation (not just afterwards for validation).</p>",
  "messages": [
    {
      "id": "87829",
      "postDate": "08/01/2015 01:30:03",
      "content": "<p>The current code available for testing KS and CVM scores require the classifier to make predictions on 'test' datasets.  Is there a way to calculate these scores without a classifier?  I.e. using only the predictions of the classifier, masses and real signals?  There must be a way since this is what is done when submitting, no?</p>\n\n<p>If we can have a way of calculating these scores without the classifier it opens up opportunities to have custom objective functions and other interesting things that can be used during model creation (not just afterwards for validation).</p>",
      "rawMarkdown": "The current code available for testing KS and CVM scores require the classifier to make predictions on 'test' datasets.  Is there a way to calculate these scores without a classifier?  I.e. using only the predictions of the classifier, masses and real signals?  There must be a way since this is what is done when submitting, no?\r\n\r\nIf we can have a way of calculating these scores without the classifier it opens up opportunities to have custom objective functions and other interesting things that can be used during model creation (not just afterwards for validation).",
      "votes": null
    },
    {
      "id": "87837",
      "postDate": "08/01/2015 04:58:55",
      "content": "<blockquote>\n  <p>There must be a way since this is what is done when submitting, no?</p>\n</blockquote>\n\n<p>The test data contains a number of control data points. When your run your classifier on the test data, you are simultaneously computing predictions for the 'real' test data and for these control data points. </p>\n\n<p>Then upon submission, Kaggle uses your predictions on these control data points to compute the KS score. </p>",
      "rawMarkdown": "> There must be a way since this is what is done when submitting, no?\r\n\r\nThe test data contains a number of control data points. When your run your classifier on the test data, you are simultaneously computing predictions for the 'real' test data and for these control data points. \r\n\r\nThen upon submission, Kaggle uses your predictions on these control data points to compute the KS score.",
      "votes": null
    },
    {
      "id": "87838",
      "postDate": "08/01/2015 05:21:15",
      "content": "<p>makes sense.  tnx for clarifying </p>",
      "rawMarkdown": "makes sense.  tnx for clarifying",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 87837,
      "author_name": "fchollet",
      "author_url": "",
      "post_date": "08/01/2015 04:58:55",
      "content": "<blockquote>\n  <p>There must be a way since this is what is done when submitting, no?</p>\n</blockquote>\n\n<p>The test data contains a number of control data points. When your run your classifier on the test data, you are simultaneously computing predictions for the 'real' test data and for these control data points. </p>\n\n<p>Then upon submission, Kaggle uses your predictions on these control data points to compute the KS score. </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 87838,
      "author_name": "guidotapia",
      "author_url": "",
      "post_date": "08/01/2015 05:21:15",
      "content": "<p>makes sense.  tnx for clarifying </p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "87829": "The current code available for testing KS and CVM scores require the classifier to make predictions on 'test' datasets.  Is there a way to calculate these scores without a classifier?  I.e. using only the predictions of the classifier, masses and real signals?  There must be a way since this is what is done when submitting, no?\r\n\r\nIf we can have a way of calculating these scores without the classifier it opens up opportunities to have custom objective functions and other interesting things that can be used during model creation (not just afterwards for validation).",
    "87837": "> There must be a way since this is what is done when submitting, no?\r\n\r\nThe test data contains a number of control data points. When your run your classifier on the test data, you are simultaneously computing predictions for the 'real' test data and for these control data points. \r\n\r\nThen upon submission, Kaggle uses your predictions on these control data points to compute the KS score.",
    "87838": "makes sense.  tnx for clarifying"
  },
  "source": "meta"
}