{
  "id": 11196,
  "title": "Use of test data",
  "url": "/competitions/avazu-ctr-prediction/discussion/11196",
  "author_name": "",
  "post_date": "2014-12-10T15:14:36.307Z",
  "votes": 7,
  "comment_count": 21,
  "views": 4598,
  "content": "<p>I am a bit confused about the competition rules. Is it ok to use test data to calibrate predictions?</p>",
  "messages": [
    {
      "id": "59950",
      "postDate": "12/10/2014 15:14:36",
      "content": "<p>I am a bit confused about the competition rules. Is it ok to use test data to calibrate predictions?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "59952",
      "postDate": "12/10/2014 15:57:00",
      "content": "<p>In general you can do whatever you want with all the data you're given. What part of the rules concerns you?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "59953",
      "postDate": "12/10/2014 16:02:03",
      "content": "<p>Well, for example in Epilepsy Seizure Challenge it had to be additionally confirmed that the use of test data to calibrate predictions is ok.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "59954",
      "postDate": "12/10/2014 16:03:51",
      "content": "<p>I don't see anything along those lines here. It just says you cannot use the old versions of the data. But I haven't read the rules line by line...</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "60003",
      "postDate": "12/11/2014 15:53:03",
      "content": "<p>I wish there was some clarity on this. Are people doing this already or not?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "60004",
      "postDate": "12/11/2014 16:04:23",
      "content": "<p>Hi I have a naive thought and I'd like some comment:</p>\n<p>is it possible to use unsupervised clustering to find a subset of training data that is &quot;quite similar&quot; to test data and use that subset as holdout validation?</p>\n<p>Is it stupid? unpractical? unnecessary?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "60005",
      "postDate": "12/11/2014 16:08:35",
      "content": "<p>I agree, we do need clarifications. If we look into the competition goal 'The winning models from this competition will be released under an open-source license' and the idea 'click prediction systems are essential and widely used for sponsored search and real-time bidding', the admins should prohibit the use of test data, since you never know test data parameters in advance.</p>\n<p>But if we're strict enough, we should ban all non online learning algorithms as well, since you can't use others for real-time bidding. It's going to take too much computational time otherwise. But that might be an overkill.</p>\n<p>What do you think?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "60006",
      "postDate": "12/11/2014 16:13:25",
      "content": "<p>[quote=rcarson;60004]</p>\n<p>Hi I have a naive thought and I'd like some comment:</p>\n<p>is it possible to use unsupervised clustering to find a subset of training data that is &quot;quite similar&quot; to test data and use that subset as holdout validation?</p>\n<p>Is it stupid? unpractical? unnecessary?</p>\n<p>[/quote]</p>\n<p>If you do that, how would you be able to make accurate predictions on CV if all the 'similar' data won't be available for learning? For example, if you use data with rare features for CV, how would your algorithm know how to predict on them?</p>\n<p>It makes sense to me only if you find similar data to test data, then split it into training and CV and then perform testing. But that's too much, I suppose.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "60007",
      "postDate": "12/11/2014 16:14:47",
      "content": "<p>[quote=Ivan Lobov;60005]</p>\n<p>we should ban all non online learning algorithms as well, since you can't use others for real-time bidding. It's going to take too much computational time otherwise. But that might be an overkill.</p>\n<p>What do you think?</p>\n<p>[/quote]</p>\n<p>Not necessarily so slow if you have a giant&nbsp;supercomputer. Feasibility is not&nbsp;a&nbsp;concern for a contest, in most cases. We just push it to the limit at all costs. :P</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "60009",
      "postDate": "12/11/2014 16:19:22",
      "content": "<p>[quote=Ivan Lobov;60006]</p>\n<p>If you do that, how would you be able to make accurate predictions on CV if all the 'similar' data won't be available for learning? For example, if you use data with rare features for CV, how would your algorithm know how to predict on them?</p>\n<p>It makes sense to me only if you find similar data to test data, then split it into training and CV and then perform testing. But that's too much, I suppose.</p>\n<p>[/quote]</p>\n<p>Thank you! I'm just worried about overfitting.&nbsp;</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "60011",
      "postDate": "12/11/2014 16:22:35",
      "content": "<p>[quote=rcarson;60007]</p>\n<p>[quote=Ivan Lobov;60005]</p>\n<p>we should ban all non online learning algorithms as well, since you can't use others for real-time bidding. It's going to take too much computational time otherwise. But that might be an overkill.</p>\n<p>What do you think?</p>\n<p>[/quote]</p>\n<p>Not necessarily so slow if you have a giant&nbsp;supercomputer. Feasibility is not&nbsp;a&nbsp;concern for a contest, in most cases. We just push it to the limit at all costs. :P</p>\n<p>[/quote]</p>\n<p>I honestly don't think Avazu needs to spend money and time to release an open-source code that nobody except for evil rich genius of advertising can afford to use (and even evil geniuses have better things to do with all their supercomputing power). Plus, they won't be able to use it themselves.</p>\n<p>It's going to be like <a href=\"https://www.techdirt.com/blog/innovation/articles/20120409/03412518422/why-netflix-never-implemented-algorithm-that-won-netflix-1-million-challenge.shtml\">Netflix</a>&nbsp;that paid 1 mln bucks for something they didn't implement.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "60014",
      "postDate": "12/11/2014 16:30:32",
      "content": "<p>[quote=Ivan Lobov;60011]</p>\n<p>I honestly don't think Avazu needs to spend money and time to release an open-source code that nobody except for evil rich genius of advertising can afford to use (and even evil geniuses have better things to do with all their supercomputing power). Plus, they won't be able to use it themselves.</p>\n<p>[/quote]</p>\n<p>lol, that's right! but honestly, I don't expect Avazu to use the winning algorithm at all. They may borrow some ideas.&nbsp;Actually, I'm curious about whether there is any&nbsp;machine learning contest with timing and memory constraint, just like an ACM?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "60015",
      "postDate": "12/11/2014 16:31:30",
      "content": "<p>I think these are good and legitimate questions. But I have the feeling most Kaggle competitions do produce models that are not practical in any production setting. So, I wound't worry&nbsp;too much about this competition either...</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "60019",
      "postDate": "12/11/2014 16:39:25",
      "content": "<p>[quote=Matfyzak;59950]</p>\n<p>I am a bit confused about the competition rules. Is it ok to use test data to calibrate predictions?</p>\n<p>[/quote]</p>\n<p>I saw a mention of this type of calibration in 'three idiots' solution to the Criteo contest. I'm curious - how can you calibrate without labels?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "60022",
      "postDate": "12/11/2014 16:54:30",
      "content": "<p>I wonder how many Kaggle competitions are truly (from a sponsor perspective) about finding the greatest model. IMHO, very few. I think it's more about the ideas people come up, different angles to look at the problem, and exposure as well. A nicely done competition like Higgs Boson drew lots of exposure. Plus, I don't know how much Kaggle gets paid, but prizes paid to contestants are a drop in the ocean of most companies' analytics budget. So, even if the sponsor&nbsp;only get ideas out of a Kaggle competition, given the minimal financial investment, that is plenty a good ROI in most cases.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "60024",
      "postDate": "12/11/2014 17:03:55",
      "content": "<p>[quote=Giulio;60022]</p>\n<p>I think it's more about the ideas people come up, different angles to look at the problem, and exposure as well.&nbsp;</p>\n<p>[/quote]</p>\n<p>In addition to these I think one of the biggest benefits is getting an estimation on the limit of your problem. Sometimes you just don't know what kind of error estimates to shoot for, and chances are that the people here will get close to or hit the cap for the given set of data. It's good to know when you've done enough to be able to move on to a new problem.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "60028",
      "postDate": "12/11/2014 17:42:13",
      "content": "<p>[quote=jwjohnson314;60019]</p>\n<p>[quote=Matfyzak;59950]</p>\n<p>I am a bit confused about the competition rules. Is it ok to use test data to calibrate predictions?</p>\n<p>[/quote]</p>\n<p>I saw a mention of this type of calibration in 'three idiots' solution to the Criteo contest. I'm curious - how can you calibrate without labels?</p>\n<p>[/quote]</p>\n\n<p>there's too many device_id's and device_ips. I'd like to set device_id and device_ip to null unless they appear in the test set.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "60029",
      "postDate": "12/11/2014 17:57:41",
      "content": "<p>[quote=laserwolf;60028]</p>\n<p>[quote=jwjohnson314;60019]</p>\n<p>[quote=Matfyzak;59950]</p>\n<p>I am a bit confused about the competition rules. Is it ok to use test data to calibrate predictions?</p>\n<p>[/quote]</p>\n<p>I saw a mention of this type of calibration in 'three idiots' solution to the Criteo contest. I'm curious - how can you calibrate without labels?</p>\n<p>[/quote]</p>\n<p>there's too many device_id's and device_ips. I'd like to set device_id and device_ip to null unless they appear in the test set.</p>\n<p>[/quote]</p>\n<p>I see. So what you're describing&nbsp;is different than the calibration in the Criteo solution, I think. There they reference the observed CTR on the public and private leaderboards, and scaled their predictions accordingly, but I'm not sure how CTR on the leaderboards was observed.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "60085",
      "postDate": "12/12/2014 09:11:07",
      "content": "<p>Although we can not see test data distributions in production environment, in a competition, using test data will be&nbsp;fine</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "60088",
      "postDate": "12/12/2014 09:47:59",
      "content": "<p>[quote=jwjohnson314;60029]</p>\n<p>[quote=laserwolf;60028]</p>\n<p>[quote=jwjohnson314;60019]</p>\n<p>[quote=Matfyzak;59950]</p>\n<p>I am a bit confused about the competition rules. Is it ok to use test data to calibrate predictions?</p>\n<p>[/quote]</p>\n<p>I saw a mention of this type of calibration in 'three idiots' solution to the Criteo contest. I'm curious - how can you calibrate without labels?</p>\n<p>[/quote]</p>\n<p>there's too many device_id's and device_ips. I'd like to set device_id and device_ip to null unless they appear in the test set.</p>\n<p>[/quote]</p>\n<p>I see. So what you're describing&nbsp;is different than the calibration in the Criteo solution, I think. There they reference the observed CTR on the public and private leaderboards, and scaled their predictions accordingly, but I'm not sure how CTR on the leaderboards was observed.</p>\n<p>[/quote]</p>\n<p>Making an all zeros submission or even taking a look at the&nbsp;exactly&nbsp;same scores on the leaderboard will give you the CTR on public set. :P</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "60091",
      "postDate": "12/12/2014 11:54:22",
      "content": "<p>[quote=infinitezxc;60088]</p>\n<p>Making an all zeros submission or even taking a look at the&nbsp;exactly&nbsp;same scores on the leaderboard will give you the CTR on public set. :P</p>\n<p>[/quote]</p>\n<p>Doh! So obvious once someone points it out to you. &nbsp;:-)</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "60103",
      "postDate": "12/12/2014 14:53:56",
      "content": "<p>[quote=inversion;60091]</p>\n<p>[quote=infinitezxc;60088]</p>\n<p>Making an all zeros submission or even taking a look at the&nbsp;exactly&nbsp;same scores on the leaderboard will give you the CTR on public set. :P</p>\n<p>[/quote]</p>\n<p>Doh! So obvious once someone points it out to you. &nbsp;:-)</p>\n<p>[/quote]</p>\n\n<p>No kidding. Thanks!</p>",
      "rawMarkdown": "",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 59952,
      "author_name": "adjgiulio",
      "author_url": "",
      "post_date": "12/10/2014 15:57:00",
      "content": "<p>In general you can do whatever you want with all the data you're given. What part of the rules concerns you?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 59953,
      "author_name": "lukasdrapal",
      "author_url": "",
      "post_date": "12/10/2014 16:02:03",
      "content": "<p>Well, for example in Epilepsy Seizure Challenge it had to be additionally confirmed that the use of test data to calibrate predictions is ok.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 59954,
      "author_name": "adjgiulio",
      "author_url": "",
      "post_date": "12/10/2014 16:03:51",
      "content": "<p>I don't see anything along those lines here. It just says you cannot use the old versions of the data. But I haven't read the rules line by line...</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 60003,
      "author_name": "vvarkey",
      "author_url": "",
      "post_date": "12/11/2014 15:53:03",
      "content": "<p>I wish there was some clarity on this. Are people doing this already or not?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 60004,
      "author_name": "jiweiliu",
      "author_url": "",
      "post_date": "12/11/2014 16:04:23",
      "content": "<p>Hi I have a naive thought and I'd like some comment:</p>\n<p>is it possible to use unsupervised clustering to find a subset of training data that is &quot;quite similar&quot; to test data and use that subset as holdout validation?</p>\n<p>Is it stupid? unpractical? unnecessary?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 60005,
      "author_name": "ivanlobov",
      "author_url": "",
      "post_date": "12/11/2014 16:08:35",
      "content": "<p>I agree, we do need clarifications. If we look into the competition goal 'The winning models from this competition will be released under an open-source license' and the idea 'click prediction systems are essential and widely used for sponsored search and real-time bidding', the admins should prohibit the use of test data, since you never know test data parameters in advance.</p>\n<p>But if we're strict enough, we should ban all non online learning algorithms as well, since you can't use others for real-time bidding. It's going to take too much computational time otherwise. But that might be an overkill.</p>\n<p>What do you think?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 60006,
      "author_name": "ivanlobov",
      "author_url": "",
      "post_date": "12/11/2014 16:13:25",
      "content": "<p>[quote=rcarson;60004]</p>\n<p>Hi I have a naive thought and I'd like some comment:</p>\n<p>is it possible to use unsupervised clustering to find a subset of training data that is &quot;quite similar&quot; to test data and use that subset as holdout validation?</p>\n<p>Is it stupid? unpractical? unnecessary?</p>\n<p>[/quote]</p>\n<p>If you do that, how would you be able to make accurate predictions on CV if all the 'similar' data won't be available for learning? For example, if you use data with rare features for CV, how would your algorithm know how to predict on them?</p>\n<p>It makes sense to me only if you find similar data to test data, then split it into training and CV and then perform testing. But that's too much, I suppose.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 60007,
      "author_name": "jiweiliu",
      "author_url": "",
      "post_date": "12/11/2014 16:14:47",
      "content": "<p>[quote=Ivan Lobov;60005]</p>\n<p>we should ban all non online learning algorithms as well, since you can't use others for real-time bidding. It's going to take too much computational time otherwise. But that might be an overkill.</p>\n<p>What do you think?</p>\n<p>[/quote]</p>\n<p>Not necessarily so slow if you have a giant&nbsp;supercomputer. Feasibility is not&nbsp;a&nbsp;concern for a contest, in most cases. We just push it to the limit at all costs. :P</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 60009,
      "author_name": "jiweiliu",
      "author_url": "",
      "post_date": "12/11/2014 16:19:22",
      "content": "<p>[quote=Ivan Lobov;60006]</p>\n<p>If you do that, how would you be able to make accurate predictions on CV if all the 'similar' data won't be available for learning? For example, if you use data with rare features for CV, how would your algorithm know how to predict on them?</p>\n<p>It makes sense to me only if you find similar data to test data, then split it into training and CV and then perform testing. But that's too much, I suppose.</p>\n<p>[/quote]</p>\n<p>Thank you! I'm just worried about overfitting.&nbsp;</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 60011,
      "author_name": "ivanlobov",
      "author_url": "",
      "post_date": "12/11/2014 16:22:35",
      "content": "<p>[quote=rcarson;60007]</p>\n<p>[quote=Ivan Lobov;60005]</p>\n<p>we should ban all non online learning algorithms as well, since you can't use others for real-time bidding. It's going to take too much computational time otherwise. But that might be an overkill.</p>\n<p>What do you think?</p>\n<p>[/quote]</p>\n<p>Not necessarily so slow if you have a giant&nbsp;supercomputer. Feasibility is not&nbsp;a&nbsp;concern for a contest, in most cases. We just push it to the limit at all costs. :P</p>\n<p>[/quote]</p>\n<p>I honestly don't think Avazu needs to spend money and time to release an open-source code that nobody except for evil rich genius of advertising can afford to use (and even evil geniuses have better things to do with all their supercomputing power). Plus, they won't be able to use it themselves.</p>\n<p>It's going to be like <a href=\"https://www.techdirt.com/blog/innovation/articles/20120409/03412518422/why-netflix-never-implemented-algorithm-that-won-netflix-1-million-challenge.shtml\">Netflix</a>&nbsp;that paid 1 mln bucks for something they didn't implement.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 60014,
      "author_name": "jiweiliu",
      "author_url": "",
      "post_date": "12/11/2014 16:30:32",
      "content": "<p>[quote=Ivan Lobov;60011]</p>\n<p>I honestly don't think Avazu needs to spend money and time to release an open-source code that nobody except for evil rich genius of advertising can afford to use (and even evil geniuses have better things to do with all their supercomputing power). Plus, they won't be able to use it themselves.</p>\n<p>[/quote]</p>\n<p>lol, that's right! but honestly, I don't expect Avazu to use the winning algorithm at all. They may borrow some ideas.&nbsp;Actually, I'm curious about whether there is any&nbsp;machine learning contest with timing and memory constraint, just like an ACM?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 60015,
      "author_name": "adjgiulio",
      "author_url": "",
      "post_date": "12/11/2014 16:31:30",
      "content": "<p>I think these are good and legitimate questions. But I have the feeling most Kaggle competitions do produce models that are not practical in any production setting. So, I wound't worry&nbsp;too much about this competition either...</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 60019,
      "author_name": "jwjohnson314",
      "author_url": "",
      "post_date": "12/11/2014 16:39:25",
      "content": "<p>[quote=Matfyzak;59950]</p>\n<p>I am a bit confused about the competition rules. Is it ok to use test data to calibrate predictions?</p>\n<p>[/quote]</p>\n<p>I saw a mention of this type of calibration in 'three idiots' solution to the Criteo contest. I'm curious - how can you calibrate without labels?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 60022,
      "author_name": "adjgiulio",
      "author_url": "",
      "post_date": "12/11/2014 16:54:30",
      "content": "<p>I wonder how many Kaggle competitions are truly (from a sponsor perspective) about finding the greatest model. IMHO, very few. I think it's more about the ideas people come up, different angles to look at the problem, and exposure as well. A nicely done competition like Higgs Boson drew lots of exposure. Plus, I don't know how much Kaggle gets paid, but prizes paid to contestants are a drop in the ocean of most companies' analytics budget. So, even if the sponsor&nbsp;only get ideas out of a Kaggle competition, given the minimal financial investment, that is plenty a good ROI in most cases.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 60024,
      "author_name": "dkaylor",
      "author_url": "",
      "post_date": "12/11/2014 17:03:55",
      "content": "<p>[quote=Giulio;60022]</p>\n<p>I think it's more about the ideas people come up, different angles to look at the problem, and exposure as well.&nbsp;</p>\n<p>[/quote]</p>\n<p>In addition to these I think one of the biggest benefits is getting an estimation on the limit of your problem. Sometimes you just don't know what kind of error estimates to shoot for, and chances are that the people here will get close to or hit the cap for the given set of data. It's good to know when you've done enough to be able to move on to a new problem.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 60028,
      "author_name": "vvarkey",
      "author_url": "",
      "post_date": "12/11/2014 17:42:13",
      "content": "<p>[quote=jwjohnson314;60019]</p>\n<p>[quote=Matfyzak;59950]</p>\n<p>I am a bit confused about the competition rules. Is it ok to use test data to calibrate predictions?</p>\n<p>[/quote]</p>\n<p>I saw a mention of this type of calibration in 'three idiots' solution to the Criteo contest. I'm curious - how can you calibrate without labels?</p>\n<p>[/quote]</p>\n\n<p>there's too many device_id's and device_ips. I'd like to set device_id and device_ip to null unless they appear in the test set.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 60029,
      "author_name": "jwjohnson314",
      "author_url": "",
      "post_date": "12/11/2014 17:57:41",
      "content": "<p>[quote=laserwolf;60028]</p>\n<p>[quote=jwjohnson314;60019]</p>\n<p>[quote=Matfyzak;59950]</p>\n<p>I am a bit confused about the competition rules. Is it ok to use test data to calibrate predictions?</p>\n<p>[/quote]</p>\n<p>I saw a mention of this type of calibration in 'three idiots' solution to the Criteo contest. I'm curious - how can you calibrate without labels?</p>\n<p>[/quote]</p>\n<p>there's too many device_id's and device_ips. I'd like to set device_id and device_ip to null unless they appear in the test set.</p>\n<p>[/quote]</p>\n<p>I see. So what you're describing&nbsp;is different than the calibration in the Criteo solution, I think. There they reference the observed CTR on the public and private leaderboards, and scaled their predictions accordingly, but I'm not sure how CTR on the leaderboards was observed.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 60085,
      "author_name": "stevewang0",
      "author_url": "",
      "post_date": "12/12/2014 09:11:07",
      "content": "<p>Although we can not see test data distributions in production environment, in a competition, using test data will be&nbsp;fine</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 60088,
      "author_name": "infinite",
      "author_url": "",
      "post_date": "12/12/2014 09:47:59",
      "content": "<p>[quote=jwjohnson314;60029]</p>\n<p>[quote=laserwolf;60028]</p>\n<p>[quote=jwjohnson314;60019]</p>\n<p>[quote=Matfyzak;59950]</p>\n<p>I am a bit confused about the competition rules. Is it ok to use test data to calibrate predictions?</p>\n<p>[/quote]</p>\n<p>I saw a mention of this type of calibration in 'three idiots' solution to the Criteo contest. I'm curious - how can you calibrate without labels?</p>\n<p>[/quote]</p>\n<p>there's too many device_id's and device_ips. I'd like to set device_id and device_ip to null unless they appear in the test set.</p>\n<p>[/quote]</p>\n<p>I see. So what you're describing&nbsp;is different than the calibration in the Criteo solution, I think. There they reference the observed CTR on the public and private leaderboards, and scaled their predictions accordingly, but I'm not sure how CTR on the leaderboards was observed.</p>\n<p>[/quote]</p>\n<p>Making an all zeros submission or even taking a look at the&nbsp;exactly&nbsp;same scores on the leaderboard will give you the CTR on public set. :P</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 60091,
      "author_name": "inversion",
      "author_url": "",
      "post_date": "12/12/2014 11:54:22",
      "content": "<p>[quote=infinitezxc;60088]</p>\n<p>Making an all zeros submission or even taking a look at the&nbsp;exactly&nbsp;same scores on the leaderboard will give you the CTR on public set. :P</p>\n<p>[/quote]</p>\n<p>Doh! So obvious once someone points it out to you. &nbsp;:-)</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 60103,
      "author_name": "jwjohnson314",
      "author_url": "",
      "post_date": "12/12/2014 14:53:56",
      "content": "<p>[quote=inversion;60091]</p>\n<p>[quote=infinitezxc;60088]</p>\n<p>Making an all zeros submission or even taking a look at the&nbsp;exactly&nbsp;same scores on the leaderboard will give you the CTR on public set. :P</p>\n<p>[/quote]</p>\n<p>Doh! So obvious once someone points it out to you. &nbsp;:-)</p>\n<p>[/quote]</p>\n\n<p>No kidding. Thanks!</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "59950": "",
    "59952": "",
    "59953": "",
    "59954": "",
    "60003": "",
    "60004": "",
    "60005": "",
    "60006": "",
    "60007": "",
    "60009": "",
    "60011": "",
    "60014": "",
    "60015": "",
    "60019": "",
    "60022": "",
    "60024": "",
    "60028": "",
    "60029": "",
    "60085": "",
    "60088": "",
    "60091": "",
    "60103": ""
  },
  "source": "meta"
}