{
  "id": 9520,
  "title": "Thanks for flooding the leaderboard top 50",
  "url": "/competitions/acquire-valued-shoppers-challenge/discussion/9520",
  "author_name": "",
  "post_date": "2014-06-23T07:14:30.913Z",
  "votes": 1,
  "comment_count": 18,
  "views": 6146,
  "content": "<p>I'm humbled by the greatness of these &quot;new&quot; entrants.</p>\n\n<p>I, for one, welcome our new &quot;new&quot; overlords.&nbsp;</p>",
  "messages": [
    {
      "id": "49395",
      "postDate": "06/23/2014 07:14:30",
      "content": "<p>I'm humbled by the greatness of these &quot;new&quot; entrants.</p>\n\n<p>I, for one, welcome our new &quot;new&quot; overlords.&nbsp;</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "49397",
      "postDate": "06/23/2014 08:24:05",
      "content": "<p>Don't worry Mike. Experience says, Kaggle admins will kick them when the competition ends.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "49402",
      "postDate": "06/23/2014 11:21:27",
      "content": "<p>I really think that these fake accounts should be removed as soon as possible, not until the end of the contest. they get what they want and it is possible to produce some statistically different solution in the end, I think. Horrible!!</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "49403",
      "postDate": "06/23/2014 11:31:13",
      "content": "<p>They never get what they deserve - only the sub-accounts get removed. The master account always stays as they're too clever to link them with the dummies. I was waiting for the obvious Dummy Master in another competition to get deleted - but they never did. And that was sooooo obvious as it was impossible for them to get that score with only 6 submissions. Howver, instead they finished in top 10 and became a Kaggle Master.</p>\n<p>And the cheaters don't care if you remove the accounts because, they've got the results and will now use them for their master acoount.</p>\n<p>People suck and don't believe that they will be dealt with because they won't!</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "49404",
      "postDate": "06/23/2014 11:54:56",
      "content": "<p>[quote=ACS69;49403]</p>\n<p>They never get what they deserve - only the sub-accounts get removed. The master account always stays as they're too clever to link them with the dummies. I was waiting for the obvious Dummy Master in the Walmart competition to get deleted - but they never did. And that was sooooo obvious as it was impossible for them to get that score with only 6 submissions. Howver, instead they finished in top 10 and became a Kaggle Master.</p>\n<p>And the cheaters don't care if you remove the accounts because, they've got the results and will now use them for their master acoount.</p>\n<p>People suck and don't believe that they will be dealt with because they won't!</p>\n<p>[/quote]</p>\n\n<p>That's exactly what I'm worried about. It's so unfair. Kaggle should be much much more aggressive, just as cheaters.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "49405",
      "postDate": "06/23/2014 12:09:09",
      "content": "<p>[quote=ACS69;49403]</p>\n<p>They never get what they deserve - only the sub-accounts get removed. The master account always stays as they're too clever to link them with the dummies. I was waiting for the obvious Dummy Master in the Walmart competition to get deleted - but they never did. And that was sooooo obvious as it was impossible for them to get that score with only 6 submissions. Howver, instead they finished in top 10 and became a Kaggle Master.</p>\n<p>And the cheaters don't care if you remove the accounts because, they've got the results and will now use them for their master acoount.</p>\n<p>People suck and don't believe that they will be dealt with because they won't!</p>\n<p>[/quote]</p>\n<p>I was in&nbsp;Walmart comp, but I wonder who...</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "49406",
      "postDate": "06/23/2014 12:14:26",
      "content": "<p>How would be&nbsp;possible to cheat in this competition?</p>\n<p>Mike</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "49408",
      "postDate": "06/23/2014 12:41:07",
      "content": "<p>What should they do? Link kaggle accounts with bank accounts ? My experience with&nbsp;online lending&nbsp;tells me&nbsp;that no matter how complicated and demanding you make the application process (with SMS&nbsp;verification, email verification, car-identification), cheaters (e.g. fraudsters) will keep coming through, when money is at stake...I guess there should be a &quot;refer a cheater&quot; element in kaggle . Also the top 10 (not just the winners) should&nbsp;provide the code they used to achieve their score (since it counts for master's status) , including how they derived their optimum solution (e.g. how they cross-validated, how they removed//added features // tuned hyper-parameters etc ). &nbsp;</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "49409",
      "postDate": "06/23/2014 13:05:19",
      "content": "<p>[quote=Mike Demurtas;49406]</p>\n<p>How would be&nbsp;possible to cheat in this competition?</p>\n<p>Mike</p>\n<p>[/quote]</p>\n\n<p>I think what everybody is referring to is the 5 submission per day limit that can be 'breached' this way.<br>generally speaking, I didn't really see the need for more then 5 subs per day in any kaggle competition I've competed so far (not a lot, mind you). In this competition I've actually found myself struggling to 'scrape up' submissions over the last couple of days, and I don't see myself get anywhere near to 'filling up' the 100 or so submissions we all have left till the end of the competition.<br><br></p>\n<p>I mean, I guess that for this competition, in which proper validation is not straight forward, someone might gain something out of more submissions, but from the fact that everybody gets so much submission to begin with, and since it's also a big risk relying on feedback from the leaderboard so much I don't really see the huge benefit in this type of cheating (the allstate competition that just ended a while ago is a very good example of significant over-fitting).<br><br></p>\n<p>perhaps the more experienced kagglers might share some stories about competitions in which the amount of submission actually mattered, it would be interesting to hear.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "49413",
      "postDate": "06/23/2014 14:41:35",
      "content": "<p>Two points:</p>\n<ul>\n<li><span style=\"line-height: 1.4\">We do our best to use the info&nbsp;we have to catch cheaters. For obvious reasons, we cannot be transparent about what we do and when we do it. Do people slip through? Absolutely. Do they win or finish in top places? We'd like to hope&nbsp;not. Do not interpret&nbsp;Kaggle's silence in the forums as us ignoring the issue.</span></li>\n<li><span style=\"line-height: 1.4\">Cheating on large problems via multiple accounts doesn't buy you much. You don't get any help&nbsp;on the private test set and you don't gain a proportional/material increase in your training data (because you already have so much of it).</span></li>\n</ul>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "49414",
      "postDate": "06/23/2014 14:43:13",
      "content": "<p>[quote=KazAnova;49408]</p>\n<p>I guess there should be a &quot;refer a cheater&quot; element in kaggle .</p>\n<p>[/quote]</p>\n\n<p>Above the leaderboard, on the leaderboard page, there's a link for reporting folks who you think might be using multiple accounts. &nbsp;I've just been assuming that folks who jump into the top 50 with a small number of submissions are MUCH better at feature engineering than I am (since all of my advances have seemed to hinge on new features), or are using some newfangled technology that I'm not. &nbsp;:-)</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "49417",
      "postDate": "06/23/2014 15:19:04",
      "content": "<p>I don't think it matters that much in this competition. However, it does matter when the competition has a 2 or 1 submission limit, the competition is forecasting, and non stationary.&nbsp;</p>\n\n<p>To my knowledge, Kaggle admins seem to do a fairly good job clearing the leaderboard. My initial post was regarding the somewhat obvious nature of these new overlords who seem to be doing so well with similar scores with newly created accounts within the last 24 hours or so.&nbsp;</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "49427",
      "postDate": "06/23/2014 16:59:33",
      "content": "<p>[quote=Phil Culliton;49414]</p>\n<p>[quote=KazAnova;49408]</p>\n<p>[/quote]</p>\n<p>I've just been assuming that folks who jump into the top 50 with a small number of submissions are MUCH better at feature engineering than I am (since all of my advances have seemed to hinge on new features), or are using some newfangled technology that I'm not. &nbsp;:-)</p>\n<p>[/quote]</p>\n<p>I have seen highly ranked&nbsp;players pop up high on the leaderboard with relatively few submissions, so it can happen. But what are the chances that five people who just happened to all join 3 days ago are world class modeling wizards? Almost zero.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "49476",
      "postDate": "06/24/2014 11:53:29",
      "content": "<p>51 submissions and &nbsp;I could barely beat the benchmark .</p>\n<p>Hats-off to great modelers &nbsp;who made it to top 50 with &nbsp;just 5 -10 &nbsp;submissions.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "49591",
      "postDate": "06/25/2014 22:04:19",
      "content": "<p>What&#8217;s most suspicious about these new entrants is that they would be able to produce high quality models with this data in just one day. I have found this to be a very challenging dataset just to assemble. I have been working with this dataset for two weeks and have not made a single submission yet. Constructing a robust cross-validation set in this competition seems basically impossible given the inherent differences between the training set and the test set. A person using multiple accounts is basically treating the public test set as a cross-validation set and that does seem like it could be a significant advantage. I am assuming that Kaggle has the ability to track commonalities in submissions across users and that the low-lifes using multiple accounts are not sophisticated enough to outwit them given how obvious they have been about it!</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "49991",
      "postDate": "07/03/2014 08:44:37",
      "content": "<p>Not Only in top 50 but in many places multiple accounts are present</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "50485",
      "postDate": "07/14/2014 17:09:00",
      "content": "<p>The worrying thing is that some of the dummy accounts are currently being deleted. Havingfun at number 13 exists no more - does this mean someone in the Top 12 is a cheat?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "50488",
      "postDate": "07/14/2014 17:14:35",
      "content": "<p>[quote=ACS69;50485]</p>\n<p>The worrying thing is that some of the dummy accounts are currently being deleted. Havingfun at number 13 exists no more - does this mean someone in the Top 12 is a cheat?</p>\n<p>[/quote]</p>\n\n<p>Yes!</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "50489",
      "postDate": "07/14/2014 17:25:13",
      "content": "<p>Committing account seppuku does not exclude the team from&nbsp;being analyzed for patterns of cheating.</p>",
      "rawMarkdown": "",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 49397,
      "author_name": "kazanova",
      "author_url": "",
      "post_date": "06/23/2014 08:24:05",
      "content": "<p>Don't worry Mike. Experience says, Kaggle admins will kick them when the competition ends.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 49402,
      "author_name": "jiweiliu",
      "author_url": "",
      "post_date": "06/23/2014 11:21:27",
      "content": "<p>I really think that these fake accounts should be removed as soon as possible, not until the end of the contest. they get what they want and it is possible to produce some statistically different solution in the end, I think. Horrible!!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 49403,
      "author_name": "domcastro",
      "author_url": "",
      "post_date": "06/23/2014 11:31:13",
      "content": "<p>They never get what they deserve - only the sub-accounts get removed. The master account always stays as they're too clever to link them with the dummies. I was waiting for the obvious Dummy Master in another competition to get deleted - but they never did. And that was sooooo obvious as it was impossible for them to get that score with only 6 submissions. Howver, instead they finished in top 10 and became a Kaggle Master.</p>\n<p>And the cheaters don't care if you remove the accounts because, they've got the results and will now use them for their master acoount.</p>\n<p>People suck and don't believe that they will be dealt with because they won't!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 49404,
      "author_name": "jiweiliu",
      "author_url": "",
      "post_date": "06/23/2014 11:54:56",
      "content": "<p>[quote=ACS69;49403]</p>\n<p>They never get what they deserve - only the sub-accounts get removed. The master account always stays as they're too clever to link them with the dummies. I was waiting for the obvious Dummy Master in the Walmart competition to get deleted - but they never did. And that was sooooo obvious as it was impossible for them to get that score with only 6 submissions. Howver, instead they finished in top 10 and became a Kaggle Master.</p>\n<p>And the cheaters don't care if you remove the accounts because, they've got the results and will now use them for their master acoount.</p>\n<p>People suck and don't believe that they will be dealt with because they won't!</p>\n<p>[/quote]</p>\n\n<p>That's exactly what I'm worried about. It's so unfair. Kaggle should be much much more aggressive, just as cheaters.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 49405,
      "author_name": "chenglongchen",
      "author_url": "",
      "post_date": "06/23/2014 12:09:09",
      "content": "<p>[quote=ACS69;49403]</p>\n<p>They never get what they deserve - only the sub-accounts get removed. The master account always stays as they're too clever to link them with the dummies. I was waiting for the obvious Dummy Master in the Walmart competition to get deleted - but they never did. And that was sooooo obvious as it was impossible for them to get that score with only 6 submissions. Howver, instead they finished in top 10 and became a Kaggle Master.</p>\n<p>And the cheaters don't care if you remove the accounts because, they've got the results and will now use them for their master acoount.</p>\n<p>People suck and don't believe that they will be dealt with because they won't!</p>\n<p>[/quote]</p>\n<p>I was in&nbsp;Walmart comp, but I wonder who...</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 49406,
      "author_name": "mikedemurtas",
      "author_url": "",
      "post_date": "06/23/2014 12:14:26",
      "content": "<p>How would be&nbsp;possible to cheat in this competition?</p>\n<p>Mike</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 49408,
      "author_name": "kazanova",
      "author_url": "",
      "post_date": "06/23/2014 12:41:07",
      "content": "<p>What should they do? Link kaggle accounts with bank accounts ? My experience with&nbsp;online lending&nbsp;tells me&nbsp;that no matter how complicated and demanding you make the application process (with SMS&nbsp;verification, email verification, car-identification), cheaters (e.g. fraudsters) will keep coming through, when money is at stake...I guess there should be a &quot;refer a cheater&quot; element in kaggle . Also the top 10 (not just the winners) should&nbsp;provide the code they used to achieve their score (since it counts for master's status) , including how they derived their optimum solution (e.g. how they cross-validated, how they removed//added features // tuned hyper-parameters etc ). &nbsp;</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 49409,
      "author_name": "selfishgene",
      "author_url": "",
      "post_date": "06/23/2014 13:05:19",
      "content": "<p>[quote=Mike Demurtas;49406]</p>\n<p>How would be&nbsp;possible to cheat in this competition?</p>\n<p>Mike</p>\n<p>[/quote]</p>\n\n<p>I think what everybody is referring to is the 5 submission per day limit that can be 'breached' this way.<br>generally speaking, I didn't really see the need for more then 5 subs per day in any kaggle competition I've competed so far (not a lot, mind you). In this competition I've actually found myself struggling to 'scrape up' submissions over the last couple of days, and I don't see myself get anywhere near to 'filling up' the 100 or so submissions we all have left till the end of the competition.<br><br></p>\n<p>I mean, I guess that for this competition, in which proper validation is not straight forward, someone might gain something out of more submissions, but from the fact that everybody gets so much submission to begin with, and since it's also a big risk relying on feedback from the leaderboard so much I don't really see the huge benefit in this type of cheating (the allstate competition that just ended a while ago is a very good example of significant over-fitting).<br><br></p>\n<p>perhaps the more experienced kagglers might share some stories about competitions in which the amount of submission actually mattered, it would be interesting to hear.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 49413,
      "author_name": "wcukierski",
      "author_url": "",
      "post_date": "06/23/2014 14:41:35",
      "content": "<p>Two points:</p>\n<ul>\n<li><span style=\"line-height: 1.4\">We do our best to use the info&nbsp;we have to catch cheaters. For obvious reasons, we cannot be transparent about what we do and when we do it. Do people slip through? Absolutely. Do they win or finish in top places? We'd like to hope&nbsp;not. Do not interpret&nbsp;Kaggle's silence in the forums as us ignoring the issue.</span></li>\n<li><span style=\"line-height: 1.4\">Cheating on large problems via multiple accounts doesn't buy you much. You don't get any help&nbsp;on the private test set and you don't gain a proportional/material increase in your training data (because you already have so much of it).</span></li>\n</ul>",
      "votes": null,
      "replies": []
    },
    {
      "id": 49414,
      "author_name": "philculliton",
      "author_url": "",
      "post_date": "06/23/2014 14:43:13",
      "content": "<p>[quote=KazAnova;49408]</p>\n<p>I guess there should be a &quot;refer a cheater&quot; element in kaggle .</p>\n<p>[/quote]</p>\n\n<p>Above the leaderboard, on the leaderboard page, there's a link for reporting folks who you think might be using multiple accounts. &nbsp;I've just been assuming that folks who jump into the top 50 with a small number of submissions are MUCH better at feature engineering than I am (since all of my advances have seemed to hinge on new features), or are using some newfangled technology that I'm not. &nbsp;:-)</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 49417,
      "author_name": "mikeskim",
      "author_url": "",
      "post_date": "06/23/2014 15:19:04",
      "content": "<p>I don't think it matters that much in this competition. However, it does matter when the competition has a 2 or 1 submission limit, the competition is forecasting, and non stationary.&nbsp;</p>\n\n<p>To my knowledge, Kaggle admins seem to do a fairly good job clearing the leaderboard. My initial post was regarding the somewhat obvious nature of these new overlords who seem to be doing so well with similar scores with newly created accounts within the last 24 hours or so.&nbsp;</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 49427,
      "author_name": "jfkingiii",
      "author_url": "",
      "post_date": "06/23/2014 16:59:33",
      "content": "<p>[quote=Phil Culliton;49414]</p>\n<p>[quote=KazAnova;49408]</p>\n<p>[/quote]</p>\n<p>I've just been assuming that folks who jump into the top 50 with a small number of submissions are MUCH better at feature engineering than I am (since all of my advances have seemed to hinge on new features), or are using some newfangled technology that I'm not. &nbsp;:-)</p>\n<p>[/quote]</p>\n<p>I have seen highly ranked&nbsp;players pop up high on the leaderboard with relatively few submissions, so it can happen. But what are the chances that five people who just happened to all join 3 days ago are world class modeling wizards? Almost zero.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 49476,
      "author_name": "namesj",
      "author_url": "",
      "post_date": "06/24/2014 11:53:29",
      "content": "<p>51 submissions and &nbsp;I could barely beat the benchmark .</p>\n<p>Hats-off to great modelers &nbsp;who made it to top 50 with &nbsp;just 5 -10 &nbsp;submissions.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 49591,
      "author_name": "swhardy",
      "author_url": "",
      "post_date": "06/25/2014 22:04:19",
      "content": "<p>What&#8217;s most suspicious about these new entrants is that they would be able to produce high quality models with this data in just one day. I have found this to be a very challenging dataset just to assemble. I have been working with this dataset for two weeks and have not made a single submission yet. Constructing a robust cross-validation set in this competition seems basically impossible given the inherent differences between the training set and the test set. A person using multiple accounts is basically treating the public test set as a cross-validation set and that does seem like it could be a significant advantage. I am assuming that Kaggle has the ability to track commonalities in submissions across users and that the low-lifes using multiple accounts are not sophisticated enough to outwit them given how obvious they have been about it!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 49991,
      "author_name": "vivekaditya",
      "author_url": "",
      "post_date": "07/03/2014 08:44:37",
      "content": "<p>Not Only in top 50 but in many places multiple accounts are present</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 50485,
      "author_name": "domcastro",
      "author_url": "",
      "post_date": "07/14/2014 17:09:00",
      "content": "<p>The worrying thing is that some of the dummy accounts are currently being deleted. Havingfun at number 13 exists no more - does this mean someone in the Top 12 is a cheat?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 50488,
      "author_name": "abhishek",
      "author_url": "",
      "post_date": "07/14/2014 17:14:35",
      "content": "<p>[quote=ACS69;50485]</p>\n<p>The worrying thing is that some of the dummy accounts are currently being deleted. Havingfun at number 13 exists no more - does this mean someone in the Top 12 is a cheat?</p>\n<p>[/quote]</p>\n\n<p>Yes!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 50489,
      "author_name": "wcukierski",
      "author_url": "",
      "post_date": "07/14/2014 17:25:13",
      "content": "<p>Committing account seppuku does not exclude the team from&nbsp;being analyzed for patterns of cheating.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "49395": "",
    "49397": "",
    "49402": "",
    "49403": "",
    "49404": "",
    "49405": "",
    "49406": "",
    "49408": "",
    "49409": "",
    "49413": "",
    "49414": "",
    "49417": "",
    "49427": "",
    "49476": "",
    "49591": "",
    "49991": "",
    "50485": "",
    "50488": "",
    "50489": ""
  },
  "source": "meta"
}