Dear folks,
\nTo me, it doesn't make sense to share code that bring you in the top 10% in a Kaggle competition. I can understand if you do that after the competition ends, it would be a way to improve.
\nBut in this way, it's a good chance to let lazy people to copy and paste your code and get the first positions. And this is not fair for people who just want to compete without tricks.
\nI think it is useful to share strategies and some hints, a nice tutorial as well. Those are effective chances to learn and improve for me and others like me.
\nBut if it's all about a bunch of "professors" who share their amazing code and others who copy and paste this code, the competition (and the interest for such kind of competitions) stops.
\nThis is my opinion. What do you think about that?
\nHave a nice day!
\n\n(PS: I also have job and family, I spend weeks/days on doing that because I love ML and hope to find a job position where I can apply ML methods. It's frustrating to be worst of somebody who's only copying and pasting "professors"'code)
", "messages": [ { "id": "66333", "postDate": "03/16/2015 07:58:34", "content": "Dear folks,
\nTo me, it doesn't make sense to share code that bring you in the top 10% in a Kaggle competition. I can understand if you do that after the competition ends, it would be a way to improve.
\nBut in this way, it's a good chance to let lazy people to copy and paste your code and get the first positions. And this is not fair for people who just want to compete without tricks.
\nI think it is useful to share strategies and some hints, a nice tutorial as well. Those are effective chances to learn and improve for me and others like me.
\nBut if it's all about a bunch of "professors" who share their amazing code and others who copy and paste this code, the competition (and the interest for such kind of competitions) stops.
\nThis is my opinion. What do you think about that?
\nHave a nice day!
\n\n(PS: I also have job and family, I spend weeks/days on doing that because I love ML and hope to find a job position where I can apply ML methods. It's frustrating to be worst of somebody who's only copying and pasting "professors"'code)
", "rawMarkdown": "", "votes": null }, { "id": "66334", "postDate": "03/16/2015 08:34:35", "content": "After the competition these solutions will not be interesting because winners will describe better models. So, maybe, these "professors" hope to get votes and increase their kaggle's rank. I'm sure it's an example of "bad popularity". Same as adding a lot of "sampleSubmission" before entries deadline.
\nI think that kaggle's admins should create some rules in order to prevent these fraud. For example, exclude votes from overall rating in posts which was posted after entries deadline and before competition deadline.
", "rawMarkdown": "", "votes": null }, { "id": "66335", "postDate": "03/16/2015 08:38:37", "content": "@Buzz Lightyear, I think you are right, but at the same time it does not matter. And what is the most funny, it is that if someone download Cifar10-winner code and apply it to this dataset, she got score better than any of this receipts.
", "rawMarkdown": "", "votes": null }, { "id": "66336", "postDate": "03/16/2015 08:44:14", "content": "I completely agree with you. There was noise about similar things in earlier competitions. And the consensus was ONLY SHARE if the competition has still got a month+ to end. NOT after that.
\nI am sure I am the dumb guy here - with work and family it takes me days - sometimes weeks to keep trying little by little to inch up my score. And then when I log in after 7 days - and submit - I see someone else has shared code a week back which is 50 ranks better than me and lots of people have already tried it out.
\nIt is a competition. It is about what are your existing skill sets that you have grown before and while the competition - not how good are others skill sets are and how good you are in doing exactly as they have done.
", "rawMarkdown": "", "votes": null }, { "id": "66337", "postDate": "03/16/2015 08:55:31", "content": "Agreed. I am all for knowledge sharing but it does total injustice to people sweating it out to build a good model and learn something new. I feel your pain as I've put in more man-hours into this than I can imagine myself and it just put me off seeing these a day before the deadline.
\nTutorials are fine. Teach and let people code themselves, don't spoon feed. Not a day before it ends.
\nI hope the admins will take note.
", "rawMarkdown": "", "votes": null }, { "id": "66341", "postDate": "03/16/2015 09:09:36", "content": "Agreed, I think it's a bit disingenuous to post "getting started" guides and code less than a week before the deadline. If helping people get started was really the intention, they should have been posted a few months ago. Now all it will do is distort the ranking and discourage a lot of people.
", "rawMarkdown": "", "votes": null }, { "id": "66342", "postDate": "03/16/2015 09:13:21", "content": "I didn't think it would be such a big issue, but it seems that ranking points matters a lot to people, even if you can't win the competition. Admins can delete my 0.775 thread.
\nThe implementations shared by other users helped me a lot, and I just wanted to help as well. I thought a 0.85 wouldn't infringe too much on other peoples efforts, but after a small modification the implementation reached 0.775 unexpectedly.
\nRemember also that it'll take +4h and some effort to use the implementation, and thus far only 2 people reached the 0.775 mark after the fact and we can't be sure it was with the shared implementation. At best only a handful of people will move up in the ranks due to the shared implementation, and I'm risking my own ranking to boot. However, I do take notice of the concern and Admins can delete my 0.775 thread.
", "rawMarkdown": "", "votes": null }, { "id": "66343", "postDate": "03/16/2015 09:21:03", "content": "I will support who share their code or thoughts, no matter what rank they are.
\nLearn from community and give back to the community, that is what I saw.
\nIt is just a learning process, so kindly be open minded.
", "rawMarkdown": "", "votes": null }, { "id": "66349", "postDate": "03/16/2015 10:04:55", "content": "[quote=Steven Du;66343]
\nI will support who share their code or thoughts, no matter what rank they are.
\nLearn from community and give back to the community, that is what I saw.
\nIt is just a learning process, so kindly be open minded.
\n[/quote]
\nIt is not *just* a learning process, it is also a competition with goals to achieve for competitors and for the sponsor of having real effort applied, and merits rewarded. I would expect each competitor to have different balance of motivations, learning versus competing. So what is good for person X could be too much or too little for person Y.
\nA sensible middle ground needs to be found. I would hate to lose public code sharing and open discussion of approaches. But equally if I just scraped in to top 10%, I would find it infuriating to have a top 10% code published to all in last week. Much better at that stage to publish such code *after* the competition - it would be just as good for those who want to learn, and a whole lot less disruptive to the leaderboard for those who are competing.
\nEdit: The "beat the benchmark" threads usually strike the right balance in my opinion. I always learn something, they are available to all early in the competition, and they don't skew results in the top 25%.
", "rawMarkdown": "", "votes": null }, { "id": "66352", "postDate": "03/16/2015 10:20:46", "content": "I agree with you. Learning process shouldn't share code. You can share your thought or found in data. If you share your code, it hurt the one who really do the competetion.
", "rawMarkdown": "", "votes": null }, { "id": "66375", "postDate": "03/16/2015 14:55:34", "content": "Thanks for your thoughts and comments!
\nKaggle doesn't have a reliable way to stop public sharing. If we disallowed it in our forums, it will move off of the forums to other venues. I'm not even convinced we could stop it in our forums! It'd require hand moderating every post, and every line of code attached to every post. It's impossible Kaggle and too slow for you guys to have data scientists playing forum traffic police.
\nHaving rules against public sharing has its own enforcement issues as well. If somebody posts and we ban them for it, who's going to find and catch all the people that grabbed the code/idea? What about somebody who doesn't read the rules and shares innocently? They would get banned, but the people who use their idea won't get caught unless they win. While all forms of sharing can offset the balance of "fairness", we've allowed public because the alternative is, speaking frankly, much worse.
\nPeople come to Kaggle for a variety of reasons, so regardless of our ability to perfectly enforce this issue, limiting sharing too broadly could harm the community rather than help it. It's almost always better to create healthy social norms around good behavior, as opposed to reacting to bad behavior with punishments. We think our community benefits a lot from the learning that occurs when people share, and are actively discussing what we can do address these concerns but still allow for positive communal learning.
", "rawMarkdown": "", "votes": null }, { "id": "66380", "postDate": "03/16/2015 15:14:41", "content": "As mentioned before, sharing code publicly doe not obey with rules. I believe most people come kaggle with a positive attitude, which is to improve his machine learning skills and enjoy the competition. With that said, kaggle is an open community. We are learning in the community, or are inspired by the community(see other ones' code) and we will feedback to the community(share my own code) as well. Thus, we grow up. On the other side, sharing code before deadline may incur some "unfairness". Over all, I think the advantages may overweigh the disadvantages.
", "rawMarkdown": "", "votes": null }, { "id": "66388", "postDate": "03/16/2015 15:44:17", "content": "[quote=PengPai;66380]
\nI am sorry that I do NOT agree with @Buzz Lightyear. As mentioned before, sharing code publicly doe not obey with rules. I believe most people come kaggle with a positive attitude, which is to improve his machine learning skills and enjoy the competition. With that said, kaggle is an open community. We are learning in the community, or are inspired by the community(see other ones' code) and we will feedback to the community(share my own code) as well. Thus, we grow up.
\n[/quote]
\nIt's not against the rules, of course. But the behavior does ruin the competition for a lot of people.
\nYou might feel differently if, say, you were in the Top 10, and going to hit Master status if you could stay there for the next 24 hours, but someone publicly posts a solution that allows 50 people to jump ahead of you.
\nIf it's just about "learning" why not wait 2 days until the competition is over and then post the code?
", "rawMarkdown": "", "votes": null }, { "id": "66389", "postDate": "03/16/2015 15:50:27", "content": "@inversion, of course it can be waited after deadline. I am not arguing about the time before or after : )
", "rawMarkdown": "", "votes": null }, { "id": "66397", "postDate": "03/16/2015 16:16:21", "content": "Some additional reflection on this matter:
\nI personally find the competition is what drives my learning. During the competition, I'll stay up late, wake up early, spend all-day on the weekends, etc., working on making improvements in order to drive up my ranking.
\nWhen the competition is over, I'll take a look at the winning solutions, but the amount of time and effort I put into the project goes to almost zero. (There was one exception, out of the dozen contests I've participated in, where I continued to work on my solution after the competition deadline.) That's because it's more rewarding to work on the next competition.
\nSo, here's the effect of posting full solutions to the forum, from my personal perspective:
\nMonths/Weeks before contest end - I learn a ton. Everyone benefits. It's not detrimental to anyone.
\nDays before contest end - Whether I learn anything depends on the amount of available time I have. A much smaller set of people will benefit. It will, most likely, be detrimental to people.
\nAfter the contest - Generally much less valuable to me personally. But, it is not detrimental to anyone.
\nI personally tried to run the code, but got a GraphLab error I couldn't figure out. Since I'll be at work all day today, I'm pretty much done with the competition. Then only thing I have to look forward to is to see how many people are able to use the code to bump me down.
\nFortunately, this hasn't been an all-in contest for me (I was 100% on Avazu until that ended), so it's not a big deal. But, it does feel a bit irksome regardless.
", "rawMarkdown": "", "votes": null }, { "id": "66399", "postDate": "03/16/2015 16:20:40", "content": "All fair points. Moral policing might not be the best way to go with the sharing culture more vibrant than ever these days. At the end of the day it's for the overall good.
\nPerhaps add some verbiage to the rules to "recommend" not post the entire solution x days before a competition ends, as a courtesy to people who have worked hard for x**n days prior to that? Someone sharing a Top 10% code knows what he/she is doing and I am sure most people like Unas will happily abide by the rules. We can't control the outliers.
\nIt's just a thin line between fostering learning and not demotivating folks who're trying hard and use Ranking as a source of inspiration.
\n[quote=William Cukierski;66375]
\nThanks for your thoughts and comments!
\nKaggle doesn't have a reliable way to stop public sharing. If we disallowed it in our forums, it will move off of the forums to other venues. I'm not even convinced we could stop it in our forums! It'd require hand moderating every post, and every line of code attached to every post. It's impossible Kaggle and too slow for you guys to have data scientists playing forum traffic police.
\nHaving rules against public sharing has its own enforcement issues as well. If somebody posts and we ban them for it, who's going to find and catch all the people that grabbed the code/idea? What about somebody who doesn't read the rules and shares innocently? They would get banned, but the people who use their idea won't get caught unless they win. While all forms of sharing can offset the balance of "fairness", we've allowed public because the alternative is, speaking frankly, much worse.
\nPeople come to Kaggle for a variety of reasons, so regardless of our ability to perfectly enforce this issue, limiting sharing too broadly could harm the community rather than help it. It's almost always better to create healthy social norms around good behavior, as opposed to reacting to bad behavior with punishments. We think our community benefits a lot from the learning that occurs when people share, and are actively discussing what we can do address these concerns but still allow for positive communal learning.
\n[/quote]
", "rawMarkdown": "", "votes": null }, { "id": "66407", "postDate": "03/16/2015 17:14:55", "content": "Code sharing is excellent for the community, but I think we can expect people to have common sense.
\nHow on Earth do people think it's OK to swoop in during the last week and post code that basically ruins the bottom 90% of the leaderboard??
\nI know there are different opinions about sharing etc, but think of it this way:
\nif you share such code in the final week:
50% will like it
50% will feel like their entire efforts were invalidated, as they are competing for points and ranking, maybe even depend on Kaggle for credentials ( as I do ).
If you do NOT share such code in the final week:
nobody will know but you
Many people feel as if weeks or months of hard labor have been thrown in the trash. So, given many many people feel that way - why would you choose to do something that makes these people feel like that? If you know something is going to hurt a lot of people, why would you do it, ever?
\n\nYou can always post it at the end of the competition. Unlike inversion, I do go through code and try things out post-competition. I'm sure many others do as well.
\n\nThe bottom line is this: if you release such code so near the end, you are KNOWINGLY causing a ruckus, shaking things, up, pissing 50% of the people of, invalidating most of the LB scores etc. So why would you do it??? Example - it's not illegal to go into the middle of a room of people taking a test and go "NNNNNNNAHHHHHHH!!!!!" - but it just kinda sucks and pisses people off. So: let's not.
", "rawMarkdown": "", "votes": null }, { "id": "66416", "postDate": "03/16/2015 17:45:21", "content": "FWIW, I rather like the idea of allowing unrestricted sharing. I can understand the opposing position, but my point of view is different. I never really play to win -- I'd LOVE to; I hold out hope that I'll come up with a clever solution that in its simplicity will win an event, but I'm more here to socialize, learn, and enjoy the friendly competition.
\nThat said, the events that I've gotten the most out of have been strong in "coopetition"; particularly among the very large competitions with million dollar prizes and months of preparation the regular sharing of information had the effect of driving the entire leaderboard ahead, sparking discussions, and making the events more pleasant, to me at least.
\nWhile I admit that it takes some time to do that, and so I understand the desire to limit the "last minute" sharing, I'm not sure I agree there either. If someone has a high ranking algorithm that is easy enough to implement and run that they're willing to share, it doesn't bother me. If it's trivial enough to implement in the time remaining, then the competition will benefit those who can quickly adapt and make use of the new tool -- ensembling it, tuning it, or simply having crafted a better solution. If it's difficult to implement, that will weed out teams who can't commit the time or expertise to make use of the tool (which, admittedly, is what is upsetting some people). But managing time and resources in this way is a competitive decision. We're talking about the sprint to the finish; learning from the forums seems like a valid competitive tool.
\nLastly, it seems that the group most likely to be hurt by sharing code is the team that shared it. Rather than condemn, I personally will choose to praise a team willing to risk a competitive advantage like that and contribute to the community.
", "rawMarkdown": "", "votes": null }, { "id": "66425", "postDate": "03/16/2015 17:59:35", "content": "I generally support public sharing. However, some code sharing would be regarded as a commercial promotion. E.g.
\nhttps://www.kaggle.com/c/datasciencebowl/forums/t/12750/code-sample-for-score-of-0-93
\nAlthough GraphLab-Create uses CXXNet as a deep learning engine and CXXnet is having the Apache license, GraphLab-Create is not, and they are making money for commercial licensing. Therefore, they don't care about Kagglers' thoughts by posting such a post in the last week.
", "rawMarkdown": "", "votes": null }, { "id": "66446", "postDate": "03/16/2015 19:09:38", "content": "So, the originator of this post is voted some 20 times. Does this post (complaining) count as a data science skill if we are measuring skill with number of votes?
\nI am neutral about sharing any code at any point in time in kaggle competitions. For anyone who is competing to learn, that shouldn't be a problem although I understand that some people would like to finish top 10% or so to build their kaggle 'reputation'. But, if you are not in the money, I think, you should just compete to learn not necessarily competing with people for the sake of competing.
\nI personally learned from the controversial code shared here. I didn't know about 'graphlab create' before and it helped me to explore it further. I thought it was a well documented much more flexible package (to train, to test any arbitrary network, and their utility functions) than the other deep learning packages I have come across so far. So, to me, learning about 'graphlab create' was worth the try. Just a thought :)
", "rawMarkdown": "", "votes": null }, { "id": "66451", "postDate": "03/16/2015 19:14:51", "content": "@Deep
\nI learned about graphlab too - I think it's excellent.
\nHowever, we're not arguing against the sharing of such information. Its the timing that's the problem.
\nYou would have learned about graphlab just fine if it were posted a month ago, or today post-contest in the solutions forum.
\nAnd neither of those cases would have upset anybody.
", "rawMarkdown": "", "votes": null }, { "id": "66457", "postDate": "03/16/2015 19:27:20", "content": "[quote=Deep;66446]
\nBut, if you are not in the money, I think, you should just compete to learn not necessarily competing with people for the sake of competing.
\n[/quote]
\nFor those who might be looking to get data science jobs, which would sound better to an interviewer:
\nBONUS - (although you'll never hear it) - "I placed in the top 10% in a very high-profile kaggle competition because fortunately someone posted a high-scoring solution at the last minute and all I had to do was compile it!"
", "rawMarkdown": "", "votes": null }, { "id": "66458", "postDate": "03/16/2015 19:29:17", "content": "@Phillip I got the point of the discussion. Thanks for making it more pronounced.
\nI just thought people shouldn't fight like this for something that might not mean anything after the competition is over in 4.6 hrs to be exact :)
\nPersonally, the only reason I will get back to this competition after it is finished is to see the winning solutions posts. Cos we all would move on to minding our business or to the next competition. The space-time of the competition creates a good competition spirit to learn more. So, we should compete in a cheerful spirit and not bicker with each other before the competition expires :) that's my point.
\n@inversion so you think the +20 votes to this post originator will also sound good on interviews? Cos apparently these votes also add up to your profile and make you say that I am rank #sth sometime? You should compete to learn not necessarily to show your kaggle profile to interviewer. I am not even sure interviewers will care much about your positions on a kaggle competition if you cannot demonstrate yourself on the spot.
", "rawMarkdown": "", "votes": null }, { "id": "66462", "postDate": "03/16/2015 19:36:52", "content": "As an example of good sharing - this post of @yr https://www.kaggle.com/c/datasciencebowl/forums/t/11360/data-augmentation/65117#post65117
Not ready recipe, but nice-tuned version of tool with lots of hand-picked useful pull requests.
[quote=Deep;66458]
\n@inversion so you think the +20 votes to this post originator will also sound good on interviews? Cos apparently these votes also add up to your profile and make you say that I am rank #sth sometime? You should compete to learn not necessarily to show your kaggle profile to interviewer. I am not even sure interviewers will care much about your positions on a kaggle competition if you cannot demonstrate yourself on the spot.
\n[/quote]
\nPlacing high on a kaggle competition may be the difference between even getting the interview or not.
\nI've had 2 companies reach out to me because of my results on kaggle, plus two other opportunities that were directly related to my results on kaggle.
\nKaggle is a better place if your results/rank correlate with your skill, not whether your ran code that was handed to you 2 days before the competition end.
", "rawMarkdown": "", "votes": null }, { "id": "66470", "postDate": "03/16/2015 19:54:06", "content": "[quote=Deep;66458]
\nCos apparently these votes also add up to your profile and make you say that I am rank #sth sometime?
\n[/quote]
\nThis is not correct. The forum votes system is a much simpler add on, and does not affect your ranking.
\n[quote=Deep;66458]
\nYou should compete to learn not necessarily to show your kaggle profile to interviewer.
\n[/quote]
\nIsn't this the difference of opinion, right here? People here compete for different reasons. Declaring what someone else's motivation is, or should be, does not really go anywhere. People have the motivations they have. We're seeing plenty of variation just on this thread, and none of it seems extreme. So compromise is needed for fair play and keep the largest number of people involved.
", "rawMarkdown": "", "votes": null }, { "id": "66472", "postDate": "03/16/2015 19:55:16", "content": "@Deep
\nI don't think you have any business telling anybody why they should compete.
\nI personally compete BOTH to learn and to have something to show people to prove that I've got some machine learning and data analysis skills.
\nPersonally - after I won first prize in the Seizure Prediction competition, my life changed. I'm like a totally different person in people's eyes. My overall rank has a lot to do with that.
\nYou can't tell people that this factor should just be ignored just because you don't care. For me, in my life, and for many others, their performance here may be the only way that they can prove to an employer that they're worth a damn.
\n"I am not even sure interviewers will care much about your positions on a kaggle competition if you cannot demonstrate yourself on the spot."
\nYou're basically saying here "if you are complete crap in your interview, they won't care about Kaggle". But, what if
\n- you're pretty good, but you don't have a degree in machine learning?
- you're on the edge and competing with another person - and could show your top 100 ranking as a reason to select you over your competition
- you have much higher than average skill, but nothing on your CV to prove it officially?
Don't tell me or anyone else that they "shouldn't" use Kaggle in that way. You might have the luxury of not needing to, but many of us have to work hard to prove ourselves amongst many competitors. Don't tell us what we "should" do.
", "rawMarkdown": "", "votes": null }, { "id": "66476", "postDate": "03/16/2015 20:08:46", "content": "[quote=Phillip Chilton Adkins;66472]
\n@Deep
\nFor me, in my life, and for many others, their performance here may be the only way that they can prove to an employer that they're worth a damn.
\n[/quote]
\n\nSo true. I started competing in Kaggle partly just to prove to a new slow-witted boss that I knew what I was talking about. Once I put up some decent results he started taking what I was saying seriously.
\nBTW I ran the code last night and submitted it and got a score of 0.976, no idea what went wrong.
", "rawMarkdown": "", "votes": null }, { "id": "66484", "postDate": "03/16/2015 20:25:45", "content": "@a few people -- If you really are trying to compete, then you need to be paying attention to the forums and the code people publish just as much as your competition. Intent seems to matter here, to me at least -- someone sharing their code probably isn't out to ruin your day; they're probably intending to be helpful and constructive. What's the intent of people complaining about people trying to be helpful? That seems to be what adds the negativity, and it smacks of a sore loser mentality.
\nIf you've tied your career to your Kaggle rank then you've picked a benchmark that you have to stay on your game to keep up, and I think if you're diligent then you can do that no matter what happens in the forums (short of actual behind the scenes cheating, which is already banned by the rules). The top Kagglers are in the top repeatedly, and that can't be by accident; they're not all wholesale jumping in at the last minute. If your score (and marketability) can be brought down by someone running a publicly available piece of software written by someone else then I'd say that may correlate well with how much we have to work to stay ahead in the real world.
\nAll of data science is topsy turvy right now with new 3rd party tools and libraries and algorithms coming out all the time. Part of winning these competitions is knowing what's available and how to use it -- I lost ground to Vowpal Wabbit code when it came out, so I had to learn how to leverage it, but if some people were able to benefit from it earlier than I did then great for them -- they beat me fair and square. If you need to monitor the forums to keep ahead then that's what you need to do; the fact that the winning entry changes frequently late in the game can drive competition. The only way to be the best is to write and tune the best algorithm. Do that, and you'll have nothing to worry about.
", "rawMarkdown": "", "votes": null }, { "id": "66487", "postDate": "03/16/2015 20:31:08", "content": "I think questioning others' motives for competing is the wrong approach here. Some might care about votes or ranks on Kaggle, others might not. To each their own.
\nThe main issue imho is sharing too much, too close to the deadline.
\nI'm all in for sharing ideas. I see no harm in that, if the original poster is fine with sharing it.
\nI'm all in for sharing fragments of code. Same as above, and also most times, code is more succinct in explaining what you mean.
\nBut I do have a problem with sharing ready-to-run code. Especially near the deadline. I don't think anyone would be annoyed if the #1 ranking person/team shared their full code when there's still a month to go.
\nBut sharing a top 25% ranking code (ready-to-run) a week before deadline would annoy some. Sharing top 10% code on the last few days would annoy even more.
\nI fully agree with William the great admin on his post above. There's no way to enforce rules about this, it would only make it worse. But I do think maybe we could discuss and vote as a community on some etiquette about sharing code. Some common courtesy on when it's too late to publish ready-to-run code.
\nThe exacts of how much and when could be discussed, and then be posted as a gentlemen's agreement, possibly on every competition forum. No enforcements, but only stating it's not favorable.
\nJust my two cents.
", "rawMarkdown": "", "votes": null }, { "id": "66490", "postDate": "03/16/2015 20:38:37", "content": "@ChipMonkey
\nThis is a reasonable argument, and in the end - I buy it. There's tons of code out there, and part of making good models is being aware of it, and being diligent in the search for it. If your code isn't way above average, someone could just stumble on the right package and beat you out. Yep, that's totally true. Making models in real life is just like that - you borrow and steal, and try to make the best thing you can make from what's available.
\nI also don't like the negativity of complaining.
\nThe only way I can fully subscribe to what you say is if it's stated upfront. I understand that what you're saying is correct, but these competitions don't say as much themselves. On their own, it's implied to be a private competition: you develop an algorithm, compete with others, try for prize and rank. It's assumed to have some sort of guidelines typical to a contest.
\nAn example would be SATs - you assume that everybody studies, everybody shows up, and we're all scored on our performance and our performance alone. It's not assumed that part of performing well on SATs is keeping an eye out for an answer sheet that may or may not appear during the course of the test, causing 75% of students to go from crap to a competitive score.
\nPeople have the same expectations of these contests up front - that they're like a "test" - like the SAT.
\nIf it's really a free-for-all code-sharing LB-destroying dog-eat-dog prove-yourself chaos-fest, then people should be made aware of the fact; then, they wouldn't be upset when their expectations get trampled on.
", "rawMarkdown": "", "votes": null }, { "id": "66491", "postDate": "03/16/2015 20:42:38", "content": "@barisumog
\nI agree.
\nIt's simply the case that this is a huge annoyance for a lot of the community. You can say what you want, but if you walk into a room and do something that pisses off 1/2 the people in it, maybe you should reconsider doing that in the future?
\nI don't think the people who posted this solution meant wrong - and I think it would be reasonable to assume that most Kagglers would abide by a gentleman's agreement; that, say, when the deadline is ~2weeks away that it'd be polite not to post high ranking solutions so as not to undermine the efforts of many who have tried so hard to get to where they are.
\nI don't think that this will undermine the spirit of sharing - share all you want, but let's be respectful to feelings that are shared among a very very large portion of the Kaggle community. Wouldn't that be the most civil thing to do? Wouldn't it be nice to have contests and sharing in which NO ONE get's upset? That sounds ideal IMHO.
", "rawMarkdown": "", "votes": null }, { "id": "66493", "postDate": "03/16/2015 20:48:47", "content": "[quote=ChipMonkey;66484]
\nWhat's the intent of people complaining about people trying to be helpful? That seems to be what adds the negativity, and it smacks of a sore loser mentality.
\n[/quote]
\nBecause it helps some at the expense of others.
\nIf someone posted Top 10 code five hours before the final submission deadline, it would help only those that (a) happened to see it, and (b) happened to have submissions left. It doesn't matter if they were just trying to help.
\nPointing this out is not complaining - it's helping people understand the effect their last-minute sharing has on others.
\nAs for sore loser mentality . . . I've got very little to gain or lose from this competition. I don't expect to be in the Top 10%. (That is, of course, unless I can get home from work early and figure out the dependency issue that is keeping me from running the GraphLab code. That is the only thing standing in the way between me placing 300th or me placing in the Top 10%. Kinda silly, don't you think?) But, imagine if every competition someone came in and posted a Top 10% solution 2 days before the contest end. Do you think Kaggle would be better or worse? How about 1-day before? I'd wager to say most people would find that detrimental.
", "rawMarkdown": "", "votes": null }, { "id": "66494", "postDate": "03/16/2015 20:50:13", "content": "Sharing code/systems in the last days can be very unfair. The 0.775 example is clearly unfair. If you saw it last night and don't have access to a good graphics card, then it will take 50 hours to achieve the result and you are screwed. That turns the overall rankings competition into dumb luck and money.
\nSharing ideas without complete code is great even late in the competition. Early it is also good.
\nIf you want to learn, great. Read papers and information people share. When a complete system is shared, learning doesn't necessarily happen. Many will just copy/paste/run and have no clue as to why it works. I suppose it is different in other countries, but here kids are taught that memorization is not as important as understanding. There is no understanding in copy/paste.
", "rawMarkdown": "", "votes": null }, { "id": "66500", "postDate": "03/16/2015 21:02:49", "content": "How about this for an approach to mitigate last minute sharing: lock the competition forum for the final week.
\nNo more team mergers, no more new entrants, no more sharing code/ideas.
", "rawMarkdown": "", "votes": null }, { "id": "66503", "postDate": "03/16/2015 21:10:45", "content": "[quote=emolson;66500]
\nHow about this for an approach to mitigate last minute sharing: lock the competition forum for the final week.
\nNo more team mergers, no more new entrants, no more sharing code/ideas.
\n[/quote]
\nSounds good, but as William already mentioned earlier - people will just go elsewhere to share code/ideas during the last week of the competition.
", "rawMarkdown": "", "votes": null }, { "id": "66506", "postDate": "03/16/2015 21:14:25", "content": "@Dmitry
\ni believe that if it were completely forbidden to share code / ideas in the forums that a forum could spring up elsewhere for sharing.
\nit seems much less likely that there would be a special place for sharing code / ideas in the final minutes of the competition.
\nmost people go to the forums.
", "rawMarkdown": "", "votes": null }, { "id": "66507", "postDate": "03/16/2015 21:15:01", "content": "[quote=Dmitry Larko;66503]
\n[quote=emolson;66500]
\nHow about this for an approach to mitigate last minute sharing: lock the competition forum for the final week.
\nNo more team mergers, no more new entrants, no more sharing code/ideas.
\n[/quote]
\nSounds good, but as William already mentioned earlier - people will just go elsewhere to share code/ideas during the last week of the competition.
\n[/quote]
\nThe rules would have to be updated to clarify that public sharing must be documented on the forum. I think this is a bit of a loophole at the moment, you could create a "public" github repo but only tell a few friends.
\nIf nothing else it would limit the effect on the leaderboard since fewer people would find and use the code.
", "rawMarkdown": "", "votes": null }, { "id": "66514", "postDate": "03/16/2015 21:26:46", "content": "@Phillip, @emolson,
\nGood point. You are right - less people will be looking for the solution outside of Kaggle's forum. At least for now. But Kaggle admins cannot prevent anonymous twitters with #Kaggle_soultion tag from appearing, for example. I'm just trying to say, if you block people from doing something - they find the other way.
", "rawMarkdown": "", "votes": null }, { "id": "66516", "postDate": "03/16/2015 21:28:19", "content": "[quote=emolson;66507]
\n[quote=Dmitry Larko;66503]
\n[quote=emolson;66500]
\nHow about this for an approach to mitigate last minute sharing: lock the competition forum for the final week.
\nNo more team mergers, no more new entrants, no more sharing code/ideas.
\n[/quote]
\nSounds good, but as William already mentioned earlier - people will just go elsewhere to share code/ideas during the last week of the competition.
\n[/quote]
\nThe rules would have to be updated to clarify that public sharing must be documented on the forum. I think this is a bit of a loophole at the moment, you could create a "public" github repo but only tell a few friends.
\nIf nothing else it would limit the effect on the leaderboard since fewer people would find and use the code.
\n[/quote]
\n\nAgreed. People can already form tiny cabals and cheat their butts off behind closed doors - there's not much to be done about that.
\nAnd, that's not even the problem being complained about.
\nPeople are concerned with MASS sharing of a solution - a totally different problem. Sharing that completely destroys the meaning of the leaderboard.
\nThe only way some independent non-Kaggle-hosted sharing source could have such an impact is if everyone knew about it - and I'm not sure such a place would be likely to even exist. If it was, it would be found out easily - since, by definition of a sharing spot capable of mass leaderboard exodus, many people would have to know about it.
\nThe forum is the place that a massive leaderboard invalidation might occur, which is a big problem compared to a few people cheating behind closed doors - a smaller problem. Let's focus on solving the big problem first.
", "rawMarkdown": "", "votes": null }, { "id": "66517", "postDate": "03/16/2015 21:29:25", "content": "If you note that most people probably aren't trying to be malicous when posting these solutions, you might expect them not to try to get around the rules with black market kaggle solutions
", "rawMarkdown": "", "votes": null }, { "id": "66520", "postDate": "03/16/2015 21:31:30", "content": "I submit that Data Science could be used to handle external sharing. When suddenly a group of competitors jumps 50 or more places and clusters together where they were not clustered before then there is likely a leak.
\nI checked the leaderboard one night and saw that I had dropped 50+ positions in about 36 hours. After looking at the leaderboard, I saw a huge cluster that didn't exist before. I suspected leaked code and checked the forums: there it was.
", "rawMarkdown": "", "votes": null }, { "id": "66532", "postDate": "03/16/2015 21:49:29", "content": "@philip and @inversion I didn't mean to offend anyone. That was not my intent. I shared my thought thinking it would make the discussion better. Now, that it made it worse, apologies to all the parties concerned!
", "rawMarkdown": "", "votes": null }, { "id": "66534", "postDate": "03/16/2015 21:51:28", "content": "@Deep
\nok, no prob. Sorry if I / we misinterpreted you
", "rawMarkdown": "", "votes": null }, { "id": "66538", "postDate": "03/16/2015 21:59:42", "content": "[quote=Deep;66532]
\n@philip and @inversion I didn't mean to offend anyone. That was not my intent. I shared my thought thinking it would make the discussion better. Now, that it made it worse, apologies to all the parties concerned!
\n[/quote]
\nI took your comments in good faith, and was not offended at all. People always sound more serious than they usually are in forums. Maybe kaggle should enable emoticons? :-)
\nIt's a topic that has come up before. And I'm sure will come up again in the future.
\nAs for right now . . . I have to decide how to spend my last submission. :-D
", "rawMarkdown": "", "votes": null }, { "id": "66554", "postDate": "03/16/2015 22:34:51", "content": "+1 For the idea of locking the forum a week so before the end of the competition. Even though there will always be a way around it, I think that it would help establish a norm and most people would honor it for that reason alone.
\nAlso, like someone else said, none of these code sharing posts are malicious, so just letting people know that it's frowned upon after a certain time period, and a weak attempt at enforcement like locking the forums I suspect would be enough. I agree that most people aren't going to go out of their way to get around such a system.
", "rawMarkdown": "", "votes": null }, { "id": "66559", "postDate": "03/16/2015 22:47:18", "content": "[quote=Matt Krump;66554]
\n+1 For the idea of locking the forum a week so before the end of the competition. Even though there will always be a way around it, I think that it would help establish a norm and most people would honor it for that reason alone.
\n[/quote]
\nPerhaps that could be part of the rules you accept when you download the data.
\nHave a box you must click: "I agree not to share complete code/solutions to the forum within 7 days of the competition close."
\n\nI'm not sure if I like that, but it's an idea.
", "rawMarkdown": "", "votes": null }, { "id": "66610", "postDate": "03/17/2015 01:48:48", "content": "Such last days complete solutions could be malicious in nature. A person could post just to give many a chance to move ahead of a particular person. Such a poster could know of a reason or reasons that would keep the intended victim from utilizing the solution.
", "rawMarkdown": "", "votes": null }, { "id": "66611", "postDate": "03/17/2015 01:50:31", "content": ""Could" does not mean "likely".
", "rawMarkdown": "", "votes": null }, { "id": "66635", "postDate": "03/17/2015 05:04:58", "content": "It some situations could matters more than likely. In other situations the opposite is true. Likely times 1049 participants may be more likely than you think.
\nThe real issue is which is more important: could or likely. Sometimes could is far more important. Is this one of them? Combine that with the other issues and late sharing of full systems is just wrong.
", "rawMarkdown": "", "votes": null }, { "id": "66674", "postDate": "03/17/2015 09:48:10", "content": "Dear All,
I'm more than happy to have open this topic. As most of you have understood, I was clearly speaking about sharing top 10% ranking (and ready-to-run) code 2-3 days before a competition ends.
In general, I believe that sharing (code, strategies, ideas, a bottle of wine) is a beautiful value, because it lets you know and meet people and improve yourself. It fosters creativity and so on.
\nI'm sorry about some friends, who seem not to have understood what I wrote but it probably depends on the fact that I'm Italian and my English is not so good. Or perhaps they were just disappointed and a bit angry not to have found some ready-to-run code to improve their score (the title of this topic was really misleading, sorry about that). I'm kidding ... :-)
\nSincerely, many thanks to Kaggle for hosting such beautiful competitions and to all Kagglers who share their knowledge, ideas and code (not 2-3 days before a contest ends please :-)) and let other people improve!
\nHave a nice day you all!
", "rawMarkdown": "", "votes": null }, { "id": "66732", "postDate": "03/17/2015 16:09:31", "content": "Unless the private LB score was better, the .775 score wouldn't have put you in the top 10% by itself, although ensembling it with my current best would likely have pushed me at least the few slots I needed for the cutoff, which would likely have made former mr. 104 unhappy. Personally, I think that releasing anything that would drop someone in the top quarter within the last month is in poor taste, and +1 the idea of locking the forums in the last week/putting something about this in the rules.
", "rawMarkdown": "", "votes": null }, { "id": "66744", "postDate": "03/17/2015 17:07:45", "content": "My initial reaction to the idea of locking the forums was to agree, but after thinking about it I'm less sure.
\nMy main worry about locking the forum is that there's often a flurry of conversation around the last week, particularly in competitions where a final dataset is held out and there may be questions or issues with the new dataset that weren't in the original. There seem to be a growing number of subtle differences in how Kaggle runs the last week -- some competitions require you to use the same code with new data, so closing the week before the algorithm must be entered and re-opening the forum when new data arrives may work. But not all competitions work this way. Kaggle would have to be careful, and the rules may be different for each competition.
\nAlso, someone who really had to ask a critical question near the end of the competition would feel pretty left out to dry; they'd just be powerless to voice it, so you could simply be shifting who the people are that become upset while at the same time silencing them. I can't say how many that would be, but I've certainly seen a lot of last-week "why is my submission failing" posts over the years that people helpfully reply to.
\nLastly, I'm curious what the people putting up the money would think. Some of the best last minute pushes may come about due to forum postings (though I've no direct evidence of this); the promoters may justifiably have a different point of view, and that could flavor Kaggle competitions. Most likely they'd need to be part of the conversation, and have the rules tailored for each competition.
\nI'm not exactly casting a nay vote -- I'm just saying there are pros and cons to think about.
", "rawMarkdown": "", "votes": null }, { "id": "66772", "postDate": "03/17/2015 18:22:11", "content": "It comes down to this: fakers and frauds should not be rewarded!
\nIf fakers and frauds are rewarded then the integrity of the Kaggle ratings and the integrity of the quality of the average Data Scientist at Kaggle is lowered. Companies that come here looking for employees will eventually see that and turn away. This is nothing new, it has happened in other fields before and will happen here if allowed.
\nThe implementation of that concept requires discussion. Sure the forum should be open during the last week, but there comes a point where sharing complete solutions or any code is wrong. Maybe there is a completely different place to share code run and moderated by Kaggle. The forum could be open, but carefully moderated during the "last period" of the contests.
\nLots of possibilities exist, we just need one that works.
", "rawMarkdown": "", "votes": null }, { "id": "66780", "postDate": "03/17/2015 18:50:47", "content": "IMO, we don't need any new official rules or technical fixes for the issues raised in this thread.
\nWe just need a good conversation, like this one, every now and then. It helps create a consensus on what the forums are for, and a little common understanding.
\nI think that is the case here, because in order to be disruptive, shared code would need to be relatively competitive and well-prepared. Someone with enough skill and conscientiousness to put that together is going to find it simple enough to understand and follow social conventions on the site.
", "rawMarkdown": "", "votes": null }, { "id": "66784", "postDate": "03/17/2015 18:57:47", "content": "[quote=Neil Slater;66780]
\nIMO, we don't need any new official rules or technical fixes.
\nWe just need a good conversation, like this one, every now and then. It helps create a consensus on what the forums are for, and a little common understanding.
\n[/quote]
\nI've seen some people, a day before the last day you can submit your first entry, send a reminder to the forum.
\nPerhaps that will be sufficient - a forum post that says something like: "Just a reminder . . . while it is not against the rules, it is generally frowned upon to submit a high-scoring turn-key solution to the forum within two weeks of the deadline."
", "rawMarkdown": "", "votes": null }, { "id": "66786", "postDate": "03/17/2015 19:00:20", "content": "[quote=inversion;66559]
\n[quote=Matt Krump;66554]
\n+1 For the idea of locking the forum a week so before the end of the competition. Even though there will always be a way around it, I think that it would help establish a norm and most people would honor it for that reason alone.
\n[/quote]
\nPerhaps that could be part of the rules you accept when you download the data.
\nHave a box you must click: "I agree not to share complete code/solutions to the forum within 7 days of the competition close."
\nI'm not sure if I like that, but it's an idea.
\n[/quote]
\nLooks like the perfect Idea, I think Kaggle Organizers can have a thought about it.
", "rawMarkdown": "", "votes": null }, { "id": "66790", "postDate": "03/17/2015 19:11:56", "content": "[quote=inversion;66784]
\n[quote=Neil Slater;66780]
\nIMO, we don't need any new official rules or technical fixes.
\nWe just need a good conversation, like this one, every now and then. It helps create a consensus on what the forums are for, and a little common understanding.
\n[/quote]
\nI've seen some people, a day before the last day you can submit your first entry, send a reminder to the forum.
\nPerhaps that will be sufficient - a forum post that says something like: "Just a reminder . . . while it is not against the rules, it is generally frowned upon to submit a high-scoring turn-key solution to the forum within two weeks of the deadline."
\n[/quote]
\n\nThis idea seems like a winner.
\nWe wouldn't need Kaggle to make any changes. We can't expect Kaggle to alter their rule structure because of a conversation 5 users had in a forum.
\n\nIt should suffice to just agree as a community.
\n\nWhen the next competition starts, someone should make a post explaining this agreement, or inviting debate about it.
\n\nWhen the deadline comes, someone will make a new post to remind everybody and set the official start of the deadline.
", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 66334, "author_name": "obaiev", "author_url": "", "post_date": "03/16/2015 08:34:35", "content": "After the competition these solutions will not be interesting because winners will describe better models. So, maybe, these "professors" hope to get votes and increase their kaggle's rank. I'm sure it's an example of "bad popularity". Same as adding a lot of "sampleSubmission" before entries deadline.
\nI think that kaggle's admins should create some rules in order to prevent these fraud. For example, exclude votes from overall rating in posts which was posted after entries deadline and before competition deadline.
", "votes": null, "replies": [] }, { "id": 66335, "author_name": "oldufo", "author_url": "", "post_date": "03/16/2015 08:38:37", "content": "@Buzz Lightyear, I think you are right, but at the same time it does not matter. And what is the most funny, it is that if someone download Cifar10-winner code and apply it to this dataset, she got score better than any of this receipts.
", "votes": null, "replies": [] }, { "id": 66336, "author_name": "rightfit", "author_url": "", "post_date": "03/16/2015 08:44:14", "content": "I completely agree with you. There was noise about similar things in earlier competitions. And the consensus was ONLY SHARE if the competition has still got a month+ to end. NOT after that.
\nI am sure I am the dumb guy here - with work and family it takes me days - sometimes weeks to keep trying little by little to inch up my score. And then when I log in after 7 days - and submit - I see someone else has shared code a week back which is 50 ranks better than me and lots of people have already tried it out.
\nIt is a competition. It is about what are your existing skill sets that you have grown before and while the competition - not how good are others skill sets are and how good you are in doing exactly as they have done.
", "votes": null, "replies": [] }, { "id": 66337, "author_name": "shankarmsy", "author_url": "", "post_date": "03/16/2015 08:55:31", "content": "Agreed. I am all for knowledge sharing but it does total injustice to people sweating it out to build a good model and learn something new. I feel your pain as I've put in more man-hours into this than I can imagine myself and it just put me off seeing these a day before the deadline.
\nTutorials are fine. Teach and let people code themselves, don't spoon feed. Not a day before it ends.
\nI hope the admins will take note.
", "votes": null, "replies": [] }, { "id": 66341, "author_name": "sedielem", "author_url": "", "post_date": "03/16/2015 09:09:36", "content": "Agreed, I think it's a bit disingenuous to post "getting started" guides and code less than a week before the deadline. If helping people get started was really the intention, they should have been posted a few months ago. Now all it will do is distort the ranking and discourage a lot of people.
", "votes": null, "replies": [] }, { "id": 66342, "author_name": "unas252547", "author_url": "", "post_date": "03/16/2015 09:13:21", "content": "I didn't think it would be such a big issue, but it seems that ranking points matters a lot to people, even if you can't win the competition. Admins can delete my 0.775 thread.
\nThe implementations shared by other users helped me a lot, and I just wanted to help as well. I thought a 0.85 wouldn't infringe too much on other peoples efforts, but after a small modification the implementation reached 0.775 unexpectedly.
\nRemember also that it'll take +4h and some effort to use the implementation, and thus far only 2 people reached the 0.775 mark after the fact and we can't be sure it was with the shared implementation. At best only a handful of people will move up in the ranks due to the shared implementation, and I'm risking my own ranking to boot. However, I do take notice of the concern and Admins can delete my 0.775 thread.
", "votes": null, "replies": [] }, { "id": 66343, "author_name": "stevendu", "author_url": "", "post_date": "03/16/2015 09:21:03", "content": "I will support who share their code or thoughts, no matter what rank they are.
\nLearn from community and give back to the community, that is what I saw.
\nIt is just a learning process, so kindly be open minded.
", "votes": null, "replies": [] }, { "id": 66349, "author_name": "slobo777", "author_url": "", "post_date": "03/16/2015 10:04:55", "content": "[quote=Steven Du;66343]
\nI will support who share their code or thoughts, no matter what rank they are.
\nLearn from community and give back to the community, that is what I saw.
\nIt is just a learning process, so kindly be open minded.
\n[/quote]
\nIt is not *just* a learning process, it is also a competition with goals to achieve for competitors and for the sponsor of having real effort applied, and merits rewarded. I would expect each competitor to have different balance of motivations, learning versus competing. So what is good for person X could be too much or too little for person Y.
\nA sensible middle ground needs to be found. I would hate to lose public code sharing and open discussion of approaches. But equally if I just scraped in to top 10%, I would find it infuriating to have a top 10% code published to all in last week. Much better at that stage to publish such code *after* the competition - it would be just as good for those who want to learn, and a whole lot less disruptive to the leaderboard for those who are competing.
\nEdit: The "beat the benchmark" threads usually strike the right balance in my opinion. I always learn something, they are available to all early in the competition, and they don't skew results in the top 25%.
", "votes": null, "replies": [] }, { "id": 66352, "author_name": "liangpang", "author_url": "", "post_date": "03/16/2015 10:20:46", "content": "I agree with you. Learning process shouldn't share code. You can share your thought or found in data. If you share your code, it hurt the one who really do the competetion.
", "votes": null, "replies": [] }, { "id": 66375, "author_name": "wcukierski", "author_url": "", "post_date": "03/16/2015 14:55:34", "content": "Thanks for your thoughts and comments!
\nKaggle doesn't have a reliable way to stop public sharing. If we disallowed it in our forums, it will move off of the forums to other venues. I'm not even convinced we could stop it in our forums! It'd require hand moderating every post, and every line of code attached to every post. It's impossible Kaggle and too slow for you guys to have data scientists playing forum traffic police.
\nHaving rules against public sharing has its own enforcement issues as well. If somebody posts and we ban them for it, who's going to find and catch all the people that grabbed the code/idea? What about somebody who doesn't read the rules and shares innocently? They would get banned, but the people who use their idea won't get caught unless they win. While all forms of sharing can offset the balance of "fairness", we've allowed public because the alternative is, speaking frankly, much worse.
\nPeople come to Kaggle for a variety of reasons, so regardless of our ability to perfectly enforce this issue, limiting sharing too broadly could harm the community rather than help it. It's almost always better to create healthy social norms around good behavior, as opposed to reacting to bad behavior with punishments. We think our community benefits a lot from the learning that occurs when people share, and are actively discussing what we can do address these concerns but still allow for positive communal learning.
", "votes": null, "replies": [] }, { "id": 66380, "author_name": "pengpai", "author_url": "", "post_date": "03/16/2015 15:14:41", "content": "As mentioned before, sharing code publicly doe not obey with rules. I believe most people come kaggle with a positive attitude, which is to improve his machine learning skills and enjoy the competition. With that said, kaggle is an open community. We are learning in the community, or are inspired by the community(see other ones' code) and we will feedback to the community(share my own code) as well. Thus, we grow up. On the other side, sharing code before deadline may incur some "unfairness". Over all, I think the advantages may overweigh the disadvantages.
", "votes": null, "replies": [] }, { "id": 66388, "author_name": "inversion", "author_url": "", "post_date": "03/16/2015 15:44:17", "content": "[quote=PengPai;66380]
\nI am sorry that I do NOT agree with @Buzz Lightyear. As mentioned before, sharing code publicly doe not obey with rules. I believe most people come kaggle with a positive attitude, which is to improve his machine learning skills and enjoy the competition. With that said, kaggle is an open community. We are learning in the community, or are inspired by the community(see other ones' code) and we will feedback to the community(share my own code) as well. Thus, we grow up.
\n[/quote]
\nIt's not against the rules, of course. But the behavior does ruin the competition for a lot of people.
\nYou might feel differently if, say, you were in the Top 10, and going to hit Master status if you could stay there for the next 24 hours, but someone publicly posts a solution that allows 50 people to jump ahead of you.
\nIf it's just about "learning" why not wait 2 days until the competition is over and then post the code?
", "votes": null, "replies": [] }, { "id": 66389, "author_name": "pengpai", "author_url": "", "post_date": "03/16/2015 15:50:27", "content": "@inversion, of course it can be waited after deadline. I am not arguing about the time before or after : )
", "votes": null, "replies": [] }, { "id": 66397, "author_name": "inversion", "author_url": "", "post_date": "03/16/2015 16:16:21", "content": "Some additional reflection on this matter:
\nI personally find the competition is what drives my learning. During the competition, I'll stay up late, wake up early, spend all-day on the weekends, etc., working on making improvements in order to drive up my ranking.
\nWhen the competition is over, I'll take a look at the winning solutions, but the amount of time and effort I put into the project goes to almost zero. (There was one exception, out of the dozen contests I've participated in, where I continued to work on my solution after the competition deadline.) That's because it's more rewarding to work on the next competition.
\nSo, here's the effect of posting full solutions to the forum, from my personal perspective:
\nMonths/Weeks before contest end - I learn a ton. Everyone benefits. It's not detrimental to anyone.
\nDays before contest end - Whether I learn anything depends on the amount of available time I have. A much smaller set of people will benefit. It will, most likely, be detrimental to people.
\nAfter the contest - Generally much less valuable to me personally. But, it is not detrimental to anyone.
\nI personally tried to run the code, but got a GraphLab error I couldn't figure out. Since I'll be at work all day today, I'm pretty much done with the competition. Then only thing I have to look forward to is to see how many people are able to use the code to bump me down.
\nFortunately, this hasn't been an all-in contest for me (I was 100% on Avazu until that ended), so it's not a big deal. But, it does feel a bit irksome regardless.
", "votes": null, "replies": [] }, { "id": 66399, "author_name": "shankarmsy", "author_url": "", "post_date": "03/16/2015 16:20:40", "content": "All fair points. Moral policing might not be the best way to go with the sharing culture more vibrant than ever these days. At the end of the day it's for the overall good.
\nPerhaps add some verbiage to the rules to "recommend" not post the entire solution x days before a competition ends, as a courtesy to people who have worked hard for x**n days prior to that? Someone sharing a Top 10% code knows what he/she is doing and I am sure most people like Unas will happily abide by the rules. We can't control the outliers.
\nIt's just a thin line between fostering learning and not demotivating folks who're trying hard and use Ranking as a source of inspiration.
\n[quote=William Cukierski;66375]
\nThanks for your thoughts and comments!
\nKaggle doesn't have a reliable way to stop public sharing. If we disallowed it in our forums, it will move off of the forums to other venues. I'm not even convinced we could stop it in our forums! It'd require hand moderating every post, and every line of code attached to every post. It's impossible Kaggle and too slow for you guys to have data scientists playing forum traffic police.
\nHaving rules against public sharing has its own enforcement issues as well. If somebody posts and we ban them for it, who's going to find and catch all the people that grabbed the code/idea? What about somebody who doesn't read the rules and shares innocently? They would get banned, but the people who use their idea won't get caught unless they win. While all forms of sharing can offset the balance of "fairness", we've allowed public because the alternative is, speaking frankly, much worse.
\nPeople come to Kaggle for a variety of reasons, so regardless of our ability to perfectly enforce this issue, limiting sharing too broadly could harm the community rather than help it. It's almost always better to create healthy social norms around good behavior, as opposed to reacting to bad behavior with punishments. We think our community benefits a lot from the learning that occurs when people share, and are actively discussing what we can do address these concerns but still allow for positive communal learning.
\n[/quote]
", "votes": null, "replies": [] }, { "id": 66407, "author_name": "phillipadkins", "author_url": "", "post_date": "03/16/2015 17:14:55", "content": "Code sharing is excellent for the community, but I think we can expect people to have common sense.
\nHow on Earth do people think it's OK to swoop in during the last week and post code that basically ruins the bottom 90% of the leaderboard??
\nI know there are different opinions about sharing etc, but think of it this way:
\nif you share such code in the final week:
50% will like it
50% will feel like their entire efforts were invalidated, as they are competing for points and ranking, maybe even depend on Kaggle for credentials ( as I do ).
If you do NOT share such code in the final week:
nobody will know but you
Many people feel as if weeks or months of hard labor have been thrown in the trash. So, given many many people feel that way - why would you choose to do something that makes these people feel like that? If you know something is going to hurt a lot of people, why would you do it, ever?
\n\nYou can always post it at the end of the competition. Unlike inversion, I do go through code and try things out post-competition. I'm sure many others do as well.
\n\nThe bottom line is this: if you release such code so near the end, you are KNOWINGLY causing a ruckus, shaking things, up, pissing 50% of the people of, invalidating most of the LB scores etc. So why would you do it??? Example - it's not illegal to go into the middle of a room of people taking a test and go "NNNNNNNAHHHHHHH!!!!!" - but it just kinda sucks and pisses people off. So: let's not.
", "votes": null, "replies": [] }, { "id": 66416, "author_name": "chipmonkey", "author_url": "", "post_date": "03/16/2015 17:45:21", "content": "FWIW, I rather like the idea of allowing unrestricted sharing. I can understand the opposing position, but my point of view is different. I never really play to win -- I'd LOVE to; I hold out hope that I'll come up with a clever solution that in its simplicity will win an event, but I'm more here to socialize, learn, and enjoy the friendly competition.
\nThat said, the events that I've gotten the most out of have been strong in "coopetition"; particularly among the very large competitions with million dollar prizes and months of preparation the regular sharing of information had the effect of driving the entire leaderboard ahead, sparking discussions, and making the events more pleasant, to me at least.
\nWhile I admit that it takes some time to do that, and so I understand the desire to limit the "last minute" sharing, I'm not sure I agree there either. If someone has a high ranking algorithm that is easy enough to implement and run that they're willing to share, it doesn't bother me. If it's trivial enough to implement in the time remaining, then the competition will benefit those who can quickly adapt and make use of the new tool -- ensembling it, tuning it, or simply having crafted a better solution. If it's difficult to implement, that will weed out teams who can't commit the time or expertise to make use of the tool (which, admittedly, is what is upsetting some people). But managing time and resources in this way is a competitive decision. We're talking about the sprint to the finish; learning from the forums seems like a valid competitive tool.
\nLastly, it seems that the group most likely to be hurt by sharing code is the team that shared it. Rather than condemn, I personally will choose to praise a team willing to risk a competitive advantage like that and contribute to the community.
", "votes": null, "replies": [] }, { "id": 66425, "author_name": "songgc", "author_url": "", "post_date": "03/16/2015 17:59:35", "content": "I generally support public sharing. However, some code sharing would be regarded as a commercial promotion. E.g.
\nhttps://www.kaggle.com/c/datasciencebowl/forums/t/12750/code-sample-for-score-of-0-93
\nAlthough GraphLab-Create uses CXXNet as a deep learning engine and CXXnet is having the Apache license, GraphLab-Create is not, and they are making money for commercial licensing. Therefore, they don't care about Kagglers' thoughts by posting such a post in the last week.
", "votes": null, "replies": [] }, { "id": 66446, "author_name": "deepcnn", "author_url": "", "post_date": "03/16/2015 19:09:38", "content": "So, the originator of this post is voted some 20 times. Does this post (complaining) count as a data science skill if we are measuring skill with number of votes?
\nI am neutral about sharing any code at any point in time in kaggle competitions. For anyone who is competing to learn, that shouldn't be a problem although I understand that some people would like to finish top 10% or so to build their kaggle 'reputation'. But, if you are not in the money, I think, you should just compete to learn not necessarily competing with people for the sake of competing.
\nI personally learned from the controversial code shared here. I didn't know about 'graphlab create' before and it helped me to explore it further. I thought it was a well documented much more flexible package (to train, to test any arbitrary network, and their utility functions) than the other deep learning packages I have come across so far. So, to me, learning about 'graphlab create' was worth the try. Just a thought :)
", "votes": null, "replies": [] }, { "id": 66451, "author_name": "phillipadkins", "author_url": "", "post_date": "03/16/2015 19:14:51", "content": "@Deep
\nI learned about graphlab too - I think it's excellent.
\nHowever, we're not arguing against the sharing of such information. Its the timing that's the problem.
\nYou would have learned about graphlab just fine if it were posted a month ago, or today post-contest in the solutions forum.
\nAnd neither of those cases would have upset anybody.
", "votes": null, "replies": [] }, { "id": 66457, "author_name": "inversion", "author_url": "", "post_date": "03/16/2015 19:27:20", "content": "[quote=Deep;66446]
\nBut, if you are not in the money, I think, you should just compete to learn not necessarily competing with people for the sake of competing.
\n[/quote]
\nFor those who might be looking to get data science jobs, which would sound better to an interviewer:
\nBONUS - (although you'll never hear it) - "I placed in the top 10% in a very high-profile kaggle competition because fortunately someone posted a high-scoring solution at the last minute and all I had to do was compile it!"
", "votes": null, "replies": [] }, { "id": 66458, "author_name": "deepcnn", "author_url": "", "post_date": "03/16/2015 19:29:17", "content": "@Phillip I got the point of the discussion. Thanks for making it more pronounced.
\nI just thought people shouldn't fight like this for something that might not mean anything after the competition is over in 4.6 hrs to be exact :)
\nPersonally, the only reason I will get back to this competition after it is finished is to see the winning solutions posts. Cos we all would move on to minding our business or to the next competition. The space-time of the competition creates a good competition spirit to learn more. So, we should compete in a cheerful spirit and not bicker with each other before the competition expires :) that's my point.
\n@inversion so you think the +20 votes to this post originator will also sound good on interviews? Cos apparently these votes also add up to your profile and make you say that I am rank #sth sometime? You should compete to learn not necessarily to show your kaggle profile to interviewer. I am not even sure interviewers will care much about your positions on a kaggle competition if you cannot demonstrate yourself on the spot.
", "votes": null, "replies": [] }, { "id": 66462, "author_name": "oldufo", "author_url": "", "post_date": "03/16/2015 19:36:52", "content": "As an example of good sharing - this post of @yr https://www.kaggle.com/c/datasciencebowl/forums/t/11360/data-augmentation/65117#post65117
Not ready recipe, but nice-tuned version of tool with lots of hand-picked useful pull requests.
[quote=Deep;66458]
\n@inversion so you think the +20 votes to this post originator will also sound good on interviews? Cos apparently these votes also add up to your profile and make you say that I am rank #sth sometime? You should compete to learn not necessarily to show your kaggle profile to interviewer. I am not even sure interviewers will care much about your positions on a kaggle competition if you cannot demonstrate yourself on the spot.
\n[/quote]
\nPlacing high on a kaggle competition may be the difference between even getting the interview or not.
\nI've had 2 companies reach out to me because of my results on kaggle, plus two other opportunities that were directly related to my results on kaggle.
\nKaggle is a better place if your results/rank correlate with your skill, not whether your ran code that was handed to you 2 days before the competition end.
", "votes": null, "replies": [] }, { "id": 66470, "author_name": "slobo777", "author_url": "", "post_date": "03/16/2015 19:54:06", "content": "[quote=Deep;66458]
\nCos apparently these votes also add up to your profile and make you say that I am rank #sth sometime?
\n[/quote]
\nThis is not correct. The forum votes system is a much simpler add on, and does not affect your ranking.
\n[quote=Deep;66458]
\nYou should compete to learn not necessarily to show your kaggle profile to interviewer.
\n[/quote]
\nIsn't this the difference of opinion, right here? People here compete for different reasons. Declaring what someone else's motivation is, or should be, does not really go anywhere. People have the motivations they have. We're seeing plenty of variation just on this thread, and none of it seems extreme. So compromise is needed for fair play and keep the largest number of people involved.
", "votes": null, "replies": [] }, { "id": 66472, "author_name": "phillipadkins", "author_url": "", "post_date": "03/16/2015 19:55:16", "content": "@Deep
\nI don't think you have any business telling anybody why they should compete.
\nI personally compete BOTH to learn and to have something to show people to prove that I've got some machine learning and data analysis skills.
\nPersonally - after I won first prize in the Seizure Prediction competition, my life changed. I'm like a totally different person in people's eyes. My overall rank has a lot to do with that.
\nYou can't tell people that this factor should just be ignored just because you don't care. For me, in my life, and for many others, their performance here may be the only way that they can prove to an employer that they're worth a damn.
\n"I am not even sure interviewers will care much about your positions on a kaggle competition if you cannot demonstrate yourself on the spot."
\nYou're basically saying here "if you are complete crap in your interview, they won't care about Kaggle". But, what if
\n- you're pretty good, but you don't have a degree in machine learning?
- you're on the edge and competing with another person - and could show your top 100 ranking as a reason to select you over your competition
- you have much higher than average skill, but nothing on your CV to prove it officially?
Don't tell me or anyone else that they "shouldn't" use Kaggle in that way. You might have the luxury of not needing to, but many of us have to work hard to prove ourselves amongst many competitors. Don't tell us what we "should" do.
", "votes": null, "replies": [] }, { "id": 66476, "author_name": "jfkingiii", "author_url": "", "post_date": "03/16/2015 20:08:46", "content": "[quote=Phillip Chilton Adkins;66472]
\n@Deep
\nFor me, in my life, and for many others, their performance here may be the only way that they can prove to an employer that they're worth a damn.
\n[/quote]
\n\nSo true. I started competing in Kaggle partly just to prove to a new slow-witted boss that I knew what I was talking about. Once I put up some decent results he started taking what I was saying seriously.
\nBTW I ran the code last night and submitted it and got a score of 0.976, no idea what went wrong.
", "votes": null, "replies": [] }, { "id": 66484, "author_name": "chipmonkey", "author_url": "", "post_date": "03/16/2015 20:25:45", "content": "@a few people -- If you really are trying to compete, then you need to be paying attention to the forums and the code people publish just as much as your competition. Intent seems to matter here, to me at least -- someone sharing their code probably isn't out to ruin your day; they're probably intending to be helpful and constructive. What's the intent of people complaining about people trying to be helpful? That seems to be what adds the negativity, and it smacks of a sore loser mentality.
\nIf you've tied your career to your Kaggle rank then you've picked a benchmark that you have to stay on your game to keep up, and I think if you're diligent then you can do that no matter what happens in the forums (short of actual behind the scenes cheating, which is already banned by the rules). The top Kagglers are in the top repeatedly, and that can't be by accident; they're not all wholesale jumping in at the last minute. If your score (and marketability) can be brought down by someone running a publicly available piece of software written by someone else then I'd say that may correlate well with how much we have to work to stay ahead in the real world.
\nAll of data science is topsy turvy right now with new 3rd party tools and libraries and algorithms coming out all the time. Part of winning these competitions is knowing what's available and how to use it -- I lost ground to Vowpal Wabbit code when it came out, so I had to learn how to leverage it, but if some people were able to benefit from it earlier than I did then great for them -- they beat me fair and square. If you need to monitor the forums to keep ahead then that's what you need to do; the fact that the winning entry changes frequently late in the game can drive competition. The only way to be the best is to write and tune the best algorithm. Do that, and you'll have nothing to worry about.
", "votes": null, "replies": [] }, { "id": 66487, "author_name": "barisumog", "author_url": "", "post_date": "03/16/2015 20:31:08", "content": "I think questioning others' motives for competing is the wrong approach here. Some might care about votes or ranks on Kaggle, others might not. To each their own.
\nThe main issue imho is sharing too much, too close to the deadline.
\nI'm all in for sharing ideas. I see no harm in that, if the original poster is fine with sharing it.
\nI'm all in for sharing fragments of code. Same as above, and also most times, code is more succinct in explaining what you mean.
\nBut I do have a problem with sharing ready-to-run code. Especially near the deadline. I don't think anyone would be annoyed if the #1 ranking person/team shared their full code when there's still a month to go.
\nBut sharing a top 25% ranking code (ready-to-run) a week before deadline would annoy some. Sharing top 10% code on the last few days would annoy even more.
\nI fully agree with William the great admin on his post above. There's no way to enforce rules about this, it would only make it worse. But I do think maybe we could discuss and vote as a community on some etiquette about sharing code. Some common courtesy on when it's too late to publish ready-to-run code.
\nThe exacts of how much and when could be discussed, and then be posted as a gentlemen's agreement, possibly on every competition forum. No enforcements, but only stating it's not favorable.
\nJust my two cents.
", "votes": null, "replies": [] }, { "id": 66490, "author_name": "phillipadkins", "author_url": "", "post_date": "03/16/2015 20:38:37", "content": "@ChipMonkey
\nThis is a reasonable argument, and in the end - I buy it. There's tons of code out there, and part of making good models is being aware of it, and being diligent in the search for it. If your code isn't way above average, someone could just stumble on the right package and beat you out. Yep, that's totally true. Making models in real life is just like that - you borrow and steal, and try to make the best thing you can make from what's available.
\nI also don't like the negativity of complaining.
\nThe only way I can fully subscribe to what you say is if it's stated upfront. I understand that what you're saying is correct, but these competitions don't say as much themselves. On their own, it's implied to be a private competition: you develop an algorithm, compete with others, try for prize and rank. It's assumed to have some sort of guidelines typical to a contest.
\nAn example would be SATs - you assume that everybody studies, everybody shows up, and we're all scored on our performance and our performance alone. It's not assumed that part of performing well on SATs is keeping an eye out for an answer sheet that may or may not appear during the course of the test, causing 75% of students to go from crap to a competitive score.
\nPeople have the same expectations of these contests up front - that they're like a "test" - like the SAT.
\nIf it's really a free-for-all code-sharing LB-destroying dog-eat-dog prove-yourself chaos-fest, then people should be made aware of the fact; then, they wouldn't be upset when their expectations get trampled on.
", "votes": null, "replies": [] }, { "id": 66491, "author_name": "phillipadkins", "author_url": "", "post_date": "03/16/2015 20:42:38", "content": "@barisumog
\nI agree.
\nIt's simply the case that this is a huge annoyance for a lot of the community. You can say what you want, but if you walk into a room and do something that pisses off 1/2 the people in it, maybe you should reconsider doing that in the future?
\nI don't think the people who posted this solution meant wrong - and I think it would be reasonable to assume that most Kagglers would abide by a gentleman's agreement; that, say, when the deadline is ~2weeks away that it'd be polite not to post high ranking solutions so as not to undermine the efforts of many who have tried so hard to get to where they are.
\nI don't think that this will undermine the spirit of sharing - share all you want, but let's be respectful to feelings that are shared among a very very large portion of the Kaggle community. Wouldn't that be the most civil thing to do? Wouldn't it be nice to have contests and sharing in which NO ONE get's upset? That sounds ideal IMHO.
", "votes": null, "replies": [] }, { "id": 66493, "author_name": "inversion", "author_url": "", "post_date": "03/16/2015 20:48:47", "content": "[quote=ChipMonkey;66484]
\nWhat's the intent of people complaining about people trying to be helpful? That seems to be what adds the negativity, and it smacks of a sore loser mentality.
\n[/quote]
\nBecause it helps some at the expense of others.
\nIf someone posted Top 10 code five hours before the final submission deadline, it would help only those that (a) happened to see it, and (b) happened to have submissions left. It doesn't matter if they were just trying to help.
\nPointing this out is not complaining - it's helping people understand the effect their last-minute sharing has on others.
\nAs for sore loser mentality . . . I've got very little to gain or lose from this competition. I don't expect to be in the Top 10%. (That is, of course, unless I can get home from work early and figure out the dependency issue that is keeping me from running the GraphLab code. That is the only thing standing in the way between me placing 300th or me placing in the Top 10%. Kinda silly, don't you think?) But, imagine if every competition someone came in and posted a Top 10% solution 2 days before the contest end. Do you think Kaggle would be better or worse? How about 1-day before? I'd wager to say most people would find that detrimental.
", "votes": null, "replies": [] }, { "id": 66494, "author_name": "charlesroberson", "author_url": "", "post_date": "03/16/2015 20:50:13", "content": "Sharing code/systems in the last days can be very unfair. The 0.775 example is clearly unfair. If you saw it last night and don't have access to a good graphics card, then it will take 50 hours to achieve the result and you are screwed. That turns the overall rankings competition into dumb luck and money.
\nSharing ideas without complete code is great even late in the competition. Early it is also good.
\nIf you want to learn, great. Read papers and information people share. When a complete system is shared, learning doesn't necessarily happen. Many will just copy/paste/run and have no clue as to why it works. I suppose it is different in other countries, but here kids are taught that memorization is not as important as understanding. There is no understanding in copy/paste.
", "votes": null, "replies": [] }, { "id": 66500, "author_name": "emolson", "author_url": "", "post_date": "03/16/2015 21:02:49", "content": "How about this for an approach to mitigate last minute sharing: lock the competition forum for the final week.
\nNo more team mergers, no more new entrants, no more sharing code/ideas.
", "votes": null, "replies": [] }, { "id": 66503, "author_name": "dmitrylarko", "author_url": "", "post_date": "03/16/2015 21:10:45", "content": "[quote=emolson;66500]
\nHow about this for an approach to mitigate last minute sharing: lock the competition forum for the final week.
\nNo more team mergers, no more new entrants, no more sharing code/ideas.
\n[/quote]
\nSounds good, but as William already mentioned earlier - people will just go elsewhere to share code/ideas during the last week of the competition.
", "votes": null, "replies": [] }, { "id": 66506, "author_name": "phillipadkins", "author_url": "", "post_date": "03/16/2015 21:14:25", "content": "@Dmitry
\ni believe that if it were completely forbidden to share code / ideas in the forums that a forum could spring up elsewhere for sharing.
\nit seems much less likely that there would be a special place for sharing code / ideas in the final minutes of the competition.
\nmost people go to the forums.
", "votes": null, "replies": [] }, { "id": 66507, "author_name": "emolson", "author_url": "", "post_date": "03/16/2015 21:15:01", "content": "[quote=Dmitry Larko;66503]
\n[quote=emolson;66500]
\nHow about this for an approach to mitigate last minute sharing: lock the competition forum for the final week.
\nNo more team mergers, no more new entrants, no more sharing code/ideas.
\n[/quote]
\nSounds good, but as William already mentioned earlier - people will just go elsewhere to share code/ideas during the last week of the competition.
\n[/quote]
\nThe rules would have to be updated to clarify that public sharing must be documented on the forum. I think this is a bit of a loophole at the moment, you could create a "public" github repo but only tell a few friends.
\nIf nothing else it would limit the effect on the leaderboard since fewer people would find and use the code.
", "votes": null, "replies": [] }, { "id": 66514, "author_name": "dmitrylarko", "author_url": "", "post_date": "03/16/2015 21:26:46", "content": "@Phillip, @emolson,
\nGood point. You are right - less people will be looking for the solution outside of Kaggle's forum. At least for now. But Kaggle admins cannot prevent anonymous twitters with #Kaggle_soultion tag from appearing, for example. I'm just trying to say, if you block people from doing something - they find the other way.
", "votes": null, "replies": [] }, { "id": 66516, "author_name": "phillipadkins", "author_url": "", "post_date": "03/16/2015 21:28:19", "content": "[quote=emolson;66507]
\n[quote=Dmitry Larko;66503]
\n[quote=emolson;66500]
\nHow about this for an approach to mitigate last minute sharing: lock the competition forum for the final week.
\nNo more team mergers, no more new entrants, no more sharing code/ideas.
\n[/quote]
\nSounds good, but as William already mentioned earlier - people will just go elsewhere to share code/ideas during the last week of the competition.
\n[/quote]
\nThe rules would have to be updated to clarify that public sharing must be documented on the forum. I think this is a bit of a loophole at the moment, you could create a "public" github repo but only tell a few friends.
\nIf nothing else it would limit the effect on the leaderboard since fewer people would find and use the code.
\n[/quote]
\n\nAgreed. People can already form tiny cabals and cheat their butts off behind closed doors - there's not much to be done about that.
\nAnd, that's not even the problem being complained about.
\nPeople are concerned with MASS sharing of a solution - a totally different problem. Sharing that completely destroys the meaning of the leaderboard.
\nThe only way some independent non-Kaggle-hosted sharing source could have such an impact is if everyone knew about it - and I'm not sure such a place would be likely to even exist. If it was, it would be found out easily - since, by definition of a sharing spot capable of mass leaderboard exodus, many people would have to know about it.
\nThe forum is the place that a massive leaderboard invalidation might occur, which is a big problem compared to a few people cheating behind closed doors - a smaller problem. Let's focus on solving the big problem first.
", "votes": null, "replies": [] }, { "id": 66517, "author_name": "phillipadkins", "author_url": "", "post_date": "03/16/2015 21:29:25", "content": "If you note that most people probably aren't trying to be malicous when posting these solutions, you might expect them not to try to get around the rules with black market kaggle solutions
", "votes": null, "replies": [] }, { "id": 66520, "author_name": "charlesroberson", "author_url": "", "post_date": "03/16/2015 21:31:30", "content": "I submit that Data Science could be used to handle external sharing. When suddenly a group of competitors jumps 50 or more places and clusters together where they were not clustered before then there is likely a leak.
\nI checked the leaderboard one night and saw that I had dropped 50+ positions in about 36 hours. After looking at the leaderboard, I saw a huge cluster that didn't exist before. I suspected leaked code and checked the forums: there it was.
", "votes": null, "replies": [] }, { "id": 66532, "author_name": "deepcnn", "author_url": "", "post_date": "03/16/2015 21:49:29", "content": "@philip and @inversion I didn't mean to offend anyone. That was not my intent. I shared my thought thinking it would make the discussion better. Now, that it made it worse, apologies to all the parties concerned!
", "votes": null, "replies": [] }, { "id": 66534, "author_name": "phillipadkins", "author_url": "", "post_date": "03/16/2015 21:51:28", "content": "@Deep
\nok, no prob. Sorry if I / we misinterpreted you
", "votes": null, "replies": [] }, { "id": 66538, "author_name": "inversion", "author_url": "", "post_date": "03/16/2015 21:59:42", "content": "[quote=Deep;66532]
\n@philip and @inversion I didn't mean to offend anyone. That was not my intent. I shared my thought thinking it would make the discussion better. Now, that it made it worse, apologies to all the parties concerned!
\n[/quote]
\nI took your comments in good faith, and was not offended at all. People always sound more serious than they usually are in forums. Maybe kaggle should enable emoticons? :-)
\nIt's a topic that has come up before. And I'm sure will come up again in the future.
\nAs for right now . . . I have to decide how to spend my last submission. :-D
", "votes": null, "replies": [] }, { "id": 66554, "author_name": "matthewkrump", "author_url": "", "post_date": "03/16/2015 22:34:51", "content": "+1 For the idea of locking the forum a week so before the end of the competition. Even though there will always be a way around it, I think that it would help establish a norm and most people would honor it for that reason alone.
\nAlso, like someone else said, none of these code sharing posts are malicious, so just letting people know that it's frowned upon after a certain time period, and a weak attempt at enforcement like locking the forums I suspect would be enough. I agree that most people aren't going to go out of their way to get around such a system.
", "votes": null, "replies": [] }, { "id": 66559, "author_name": "inversion", "author_url": "", "post_date": "03/16/2015 22:47:18", "content": "[quote=Matt Krump;66554]
\n+1 For the idea of locking the forum a week so before the end of the competition. Even though there will always be a way around it, I think that it would help establish a norm and most people would honor it for that reason alone.
\n[/quote]
\nPerhaps that could be part of the rules you accept when you download the data.
\nHave a box you must click: "I agree not to share complete code/solutions to the forum within 7 days of the competition close."
\n\nI'm not sure if I like that, but it's an idea.
", "votes": null, "replies": [] }, { "id": 66610, "author_name": "charlesroberson", "author_url": "", "post_date": "03/17/2015 01:48:48", "content": "Such last days complete solutions could be malicious in nature. A person could post just to give many a chance to move ahead of a particular person. Such a poster could know of a reason or reasons that would keep the intended victim from utilizing the solution.
", "votes": null, "replies": [] }, { "id": 66611, "author_name": "phillipadkins", "author_url": "", "post_date": "03/17/2015 01:50:31", "content": ""Could" does not mean "likely".
", "votes": null, "replies": [] }, { "id": 66635, "author_name": "charlesroberson", "author_url": "", "post_date": "03/17/2015 05:04:58", "content": "It some situations could matters more than likely. In other situations the opposite is true. Likely times 1049 participants may be more likely than you think.
\nThe real issue is which is more important: could or likely. Sometimes could is far more important. Is this one of them? Combine that with the other issues and late sharing of full systems is just wrong.
", "votes": null, "replies": [] }, { "id": 66674, "author_name": "buzzlightyear", "author_url": "", "post_date": "03/17/2015 09:48:10", "content": "Dear All,
I'm more than happy to have open this topic. As most of you have understood, I was clearly speaking about sharing top 10% ranking (and ready-to-run) code 2-3 days before a competition ends.
In general, I believe that sharing (code, strategies, ideas, a bottle of wine) is a beautiful value, because it lets you know and meet people and improve yourself. It fosters creativity and so on.
\nI'm sorry about some friends, who seem not to have understood what I wrote but it probably depends on the fact that I'm Italian and my English is not so good. Or perhaps they were just disappointed and a bit angry not to have found some ready-to-run code to improve their score (the title of this topic was really misleading, sorry about that). I'm kidding ... :-)
\nSincerely, many thanks to Kaggle for hosting such beautiful competitions and to all Kagglers who share their knowledge, ideas and code (not 2-3 days before a contest ends please :-)) and let other people improve!
\nHave a nice day you all!
", "votes": null, "replies": [] }, { "id": 66732, "author_name": "telser", "author_url": "", "post_date": "03/17/2015 16:09:31", "content": "Unless the private LB score was better, the .775 score wouldn't have put you in the top 10% by itself, although ensembling it with my current best would likely have pushed me at least the few slots I needed for the cutoff, which would likely have made former mr. 104 unhappy. Personally, I think that releasing anything that would drop someone in the top quarter within the last month is in poor taste, and +1 the idea of locking the forums in the last week/putting something about this in the rules.
", "votes": null, "replies": [] }, { "id": 66744, "author_name": "chipmonkey", "author_url": "", "post_date": "03/17/2015 17:07:45", "content": "My initial reaction to the idea of locking the forums was to agree, but after thinking about it I'm less sure.
\nMy main worry about locking the forum is that there's often a flurry of conversation around the last week, particularly in competitions where a final dataset is held out and there may be questions or issues with the new dataset that weren't in the original. There seem to be a growing number of subtle differences in how Kaggle runs the last week -- some competitions require you to use the same code with new data, so closing the week before the algorithm must be entered and re-opening the forum when new data arrives may work. But not all competitions work this way. Kaggle would have to be careful, and the rules may be different for each competition.
\nAlso, someone who really had to ask a critical question near the end of the competition would feel pretty left out to dry; they'd just be powerless to voice it, so you could simply be shifting who the people are that become upset while at the same time silencing them. I can't say how many that would be, but I've certainly seen a lot of last-week "why is my submission failing" posts over the years that people helpfully reply to.
\nLastly, I'm curious what the people putting up the money would think. Some of the best last minute pushes may come about due to forum postings (though I've no direct evidence of this); the promoters may justifiably have a different point of view, and that could flavor Kaggle competitions. Most likely they'd need to be part of the conversation, and have the rules tailored for each competition.
\nI'm not exactly casting a nay vote -- I'm just saying there are pros and cons to think about.
", "votes": null, "replies": [] }, { "id": 66772, "author_name": "charlesroberson", "author_url": "", "post_date": "03/17/2015 18:22:11", "content": "It comes down to this: fakers and frauds should not be rewarded!
\nIf fakers and frauds are rewarded then the integrity of the Kaggle ratings and the integrity of the quality of the average Data Scientist at Kaggle is lowered. Companies that come here looking for employees will eventually see that and turn away. This is nothing new, it has happened in other fields before and will happen here if allowed.
\nThe implementation of that concept requires discussion. Sure the forum should be open during the last week, but there comes a point where sharing complete solutions or any code is wrong. Maybe there is a completely different place to share code run and moderated by Kaggle. The forum could be open, but carefully moderated during the "last period" of the contests.
\nLots of possibilities exist, we just need one that works.
", "votes": null, "replies": [] }, { "id": 66780, "author_name": "slobo777", "author_url": "", "post_date": "03/17/2015 18:50:47", "content": "IMO, we don't need any new official rules or technical fixes for the issues raised in this thread.
\nWe just need a good conversation, like this one, every now and then. It helps create a consensus on what the forums are for, and a little common understanding.
\nI think that is the case here, because in order to be disruptive, shared code would need to be relatively competitive and well-prepared. Someone with enough skill and conscientiousness to put that together is going to find it simple enough to understand and follow social conventions on the site.
", "votes": null, "replies": [] }, { "id": 66784, "author_name": "inversion", "author_url": "", "post_date": "03/17/2015 18:57:47", "content": "[quote=Neil Slater;66780]
\nIMO, we don't need any new official rules or technical fixes.
\nWe just need a good conversation, like this one, every now and then. It helps create a consensus on what the forums are for, and a little common understanding.
\n[/quote]
\nI've seen some people, a day before the last day you can submit your first entry, send a reminder to the forum.
\nPerhaps that will be sufficient - a forum post that says something like: "Just a reminder . . . while it is not against the rules, it is generally frowned upon to submit a high-scoring turn-key solution to the forum within two weeks of the deadline."
", "votes": null, "replies": [] }, { "id": 66786, "author_name": "sachinvernekar", "author_url": "", "post_date": "03/17/2015 19:00:20", "content": "[quote=inversion;66559]
\n[quote=Matt Krump;66554]
\n+1 For the idea of locking the forum a week so before the end of the competition. Even though there will always be a way around it, I think that it would help establish a norm and most people would honor it for that reason alone.
\n[/quote]
\nPerhaps that could be part of the rules you accept when you download the data.
\nHave a box you must click: "I agree not to share complete code/solutions to the forum within 7 days of the competition close."
\nI'm not sure if I like that, but it's an idea.
\n[/quote]
\nLooks like the perfect Idea, I think Kaggle Organizers can have a thought about it.
", "votes": null, "replies": [] }, { "id": 66790, "author_name": "phillipadkins", "author_url": "", "post_date": "03/17/2015 19:11:56", "content": "[quote=inversion;66784]
\n[quote=Neil Slater;66780]
\nIMO, we don't need any new official rules or technical fixes.
\nWe just need a good conversation, like this one, every now and then. It helps create a consensus on what the forums are for, and a little common understanding.
\n[/quote]
\nI've seen some people, a day before the last day you can submit your first entry, send a reminder to the forum.
\nPerhaps that will be sufficient - a forum post that says something like: "Just a reminder . . . while it is not against the rules, it is generally frowned upon to submit a high-scoring turn-key solution to the forum within two weeks of the deadline."
\n[/quote]
\n\nThis idea seems like a winner.
\nWe wouldn't need Kaggle to make any changes. We can't expect Kaggle to alter their rule structure because of a conversation 5 users had in a forum.
\n\nIt should suffice to just agree as a community.
\n\nWhen the next competition starts, someone should make a post explaining this agreement, or inviting debate about it.
\n\nWhen the deadline comes, someone will make a new post to remind everybody and set the official start of the deadline.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "66333": "", "66334": "", "66335": "", "66336": "", "66337": "", "66341": "", "66342": "", "66343": "", "66349": "", "66352": "", "66375": "", "66380": "", "66388": "", "66389": "", "66397": "", "66399": "", "66407": "", "66416": "", "66425": "", "66446": "", "66451": "", "66457": "", "66458": "", "66462": "", "66468": "", "66470": "", "66472": "", "66476": "", "66484": "", "66487": "", "66490": "", "66491": "", "66493": "", "66494": "", "66500": "", "66503": "", "66506": "", "66507": "", "66514": "", "66516": "", "66517": "", "66520": "", "66532": "", "66534": "", "66538": "", "66554": "", "66559": "", "66610": "", "66611": "", "66635": "", "66674": "", "66732": "", "66744": "", "66772": "", "66780": "", "66784": "", "66786": "", "66790": "" }, "source": "meta" }