{
  "id": 9558,
  "title": "Submission scoring",
  "url": "/competitions/acquire-valued-shoppers-challenge/discussion/9558",
  "author_name": "",
  "post_date": "2014-06-25T11:40:32.880Z",
  "votes": null,
  "comment_count": 10,
  "views": 1836,
  "content": "<p>Can someone advise on submission scoring?</p>\n\n<p>Is the score for each submission reported back to the submitter, or only if it exceeds the previous submission score?</p>\n\n<p>My last 3 submissions have all received the same score, but they are substantively different from one another (and within each accross the different id's).</p>\n\n<p>I'm not sure whether this is simply because only the best score to date is reported back, or if they have genuinely received the same score (which I don't understand given their disimmilarity).</p>\n\n<p>Thanks,<br><br>Rob</p>",
  "messages": [
    {
      "id": "49550",
      "postDate": "06/25/2014 11:40:32",
      "content": "<p>Can someone advise on submission scoring?</p>\n\n<p>Is the score for each submission reported back to the submitter, or only if it exceeds the previous submission score?</p>\n\n<p>My last 3 submissions have all received the same score, but they are substantively different from one another (and within each accross the different id's).</p>\n\n<p>I'm not sure whether this is simply because only the best score to date is reported back, or if they have genuinely received the same score (which I don't understand given their disimmilarity).</p>\n\n<p>Thanks,<br><br>Rob</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "49553",
      "postDate": "06/25/2014 11:51:07",
      "content": "<p>Hi,</p>\n<p>you should get the score for every submission. If it's not better than the previous one should say something like: 'sorry you didn't improve...' &nbsp;</p>\n<p>what score are you having? maybe is the 0.5?&nbsp;</p>\n\n<p>Mike</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "49558",
      "postDate": "06/25/2014 12:37:38",
      "content": "<p>Thanks Mike, yes I got that message but no it's not 0.5, it's&nbsp;0.56822 for my last four submissions.&nbsp;3 of them were the same or very similar, but the last was&nbsp;substantively different (I've re-checked the&nbsp;on-line submissions to confirm it wasn't user error at my end). I guess the ROC will generate&nbsp;the same score for different submissions (it's clearly not going to be unique for each submission), but at 5 decimal places it's pretty unlikely. Any more help you can provide?</p>\n<p><br>Thanks,</p>\n\n<p>Rob</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "49559",
      "postDate": "06/25/2014 12:58:48",
      "content": "<p>It's possible you are changing rows in the private fold of the test set.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "49562",
      "postDate": "06/25/2014 13:13:29",
      "content": "<p>Yes, but every row has changed i.e. including public.&nbsp;</p>\n\n<p>Every row has a lower&nbsp;probability in my latest submission than the previous, which I thought would necessarily give a different score (given also&nbsp;that they are different within each submission&nbsp;e.g. I'm not going from probabilities of all 0.4 to all 0.3, which, if I understand the ROC correctly, would give the same score of 0.5).</p>\n\n<p>Thanks,<br><br>Rob</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "49563",
      "postDate": "06/25/2014 13:26:58",
      "content": "<p>It's evaluated by AUC so if you just reduced / increased probabilities by the same ratio then your score will be the same</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "49564",
      "postDate": "06/25/2014 13:29:17",
      "content": "<p>If I remember correctly you get the low score just for a few seconds and then it goes back to your best one again. maybe you're missing that few seconds.</p>\n\n<p>Mike</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "49567",
      "postDate": "06/25/2014 13:34:04",
      "content": "<p>I'll try again then, but wouldn't the &quot;My Submissions&quot; page, which shows&nbsp;every submission, show the score against each, rather than the best?</p>\n\n<p>Thanks,<br><br>Rob</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "49568",
      "postDate": "06/25/2014 13:38:18",
      "content": "<p>Thanks. It's not the same ratio for every row. Plot thickens.</p>\n\n<p>P.S. I'm still in mourning&nbsp;England's poor show!</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "49569",
      "postDate": "06/25/2014 13:56:23",
      "content": "<p>As ACS69 said, check that the rank ordering of the predictions is actually changing (sort them and compare the id order). If it's not then the AUC will be identical.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "49570",
      "postDate": "06/25/2014 14:17:58",
      "content": "<p>They are indeed the same, that's improved by understanding significantly.</p>\n\n<p>Thanks,</p>\n\n<p>Rob</p>",
      "rawMarkdown": "",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 49553,
      "author_name": "mikedemurtas",
      "author_url": "",
      "post_date": "06/25/2014 11:51:07",
      "content": "<p>Hi,</p>\n<p>you should get the score for every submission. If it's not better than the previous one should say something like: 'sorry you didn't improve...' &nbsp;</p>\n<p>what score are you having? maybe is the 0.5?&nbsp;</p>\n\n<p>Mike</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 49558,
      "author_name": "robfran",
      "author_url": "",
      "post_date": "06/25/2014 12:37:38",
      "content": "<p>Thanks Mike, yes I got that message but no it's not 0.5, it's&nbsp;0.56822 for my last four submissions.&nbsp;3 of them were the same or very similar, but the last was&nbsp;substantively different (I've re-checked the&nbsp;on-line submissions to confirm it wasn't user error at my end). I guess the ROC will generate&nbsp;the same score for different submissions (it's clearly not going to be unique for each submission), but at 5 decimal places it's pretty unlikely. Any more help you can provide?</p>\n<p><br>Thanks,</p>\n\n<p>Rob</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 49559,
      "author_name": "wcukierski",
      "author_url": "",
      "post_date": "06/25/2014 12:58:48",
      "content": "<p>It's possible you are changing rows in the private fold of the test set.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 49562,
      "author_name": "robfran",
      "author_url": "",
      "post_date": "06/25/2014 13:13:29",
      "content": "<p>Yes, but every row has changed i.e. including public.&nbsp;</p>\n\n<p>Every row has a lower&nbsp;probability in my latest submission than the previous, which I thought would necessarily give a different score (given also&nbsp;that they are different within each submission&nbsp;e.g. I'm not going from probabilities of all 0.4 to all 0.3, which, if I understand the ROC correctly, would give the same score of 0.5).</p>\n\n<p>Thanks,<br><br>Rob</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 49563,
      "author_name": "domcastro",
      "author_url": "",
      "post_date": "06/25/2014 13:26:58",
      "content": "<p>It's evaluated by AUC so if you just reduced / increased probabilities by the same ratio then your score will be the same</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 49564,
      "author_name": "mikedemurtas",
      "author_url": "",
      "post_date": "06/25/2014 13:29:17",
      "content": "<p>If I remember correctly you get the low score just for a few seconds and then it goes back to your best one again. maybe you're missing that few seconds.</p>\n\n<p>Mike</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 49567,
      "author_name": "robfran",
      "author_url": "",
      "post_date": "06/25/2014 13:34:04",
      "content": "<p>I'll try again then, but wouldn't the &quot;My Submissions&quot; page, which shows&nbsp;every submission, show the score against each, rather than the best?</p>\n\n<p>Thanks,<br><br>Rob</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 49568,
      "author_name": "robfran",
      "author_url": "",
      "post_date": "06/25/2014 13:38:18",
      "content": "<p>Thanks. It's not the same ratio for every row. Plot thickens.</p>\n\n<p>P.S. I'm still in mourning&nbsp;England's poor show!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 49569,
      "author_name": "wcukierski",
      "author_url": "",
      "post_date": "06/25/2014 13:56:23",
      "content": "<p>As ACS69 said, check that the rank ordering of the predictions is actually changing (sort them and compare the id order). If it's not then the AUC will be identical.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 49570,
      "author_name": "robfran",
      "author_url": "",
      "post_date": "06/25/2014 14:17:58",
      "content": "<p>They are indeed the same, that's improved by understanding significantly.</p>\n\n<p>Thanks,</p>\n\n<p>Rob</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "49550": "",
    "49553": "",
    "49558": "",
    "49559": "",
    "49562": "",
    "49563": "",
    "49564": "",
    "49567": "",
    "49568": "",
    "49569": "",
    "49570": ""
  },
  "source": "meta"
}