After discovering today that the some of the test data is in the train data, I made a submission with the training data tags for which the training and test data matched up. In the case where I couldn't find the test data in the training data, I made the tags to be the top five tags. I got an F1 score of 0.60873. Which ranked me at no. 20.
", "messages": [ { "id": 35241, "postDate": "2013-11-26T00:13:42.607Z", "content": "After discovering today that the some of the test data is in the train data, I made a submission with the training data tags for which the training and test data matched up. In the case where I couldn't find the test data in the training data, I made the tags to be the top five tags. I got an F1 score of 0.60873. Which ranked me at no. 20.
", "votes": 18 }, { "id": 35319, "postDate": "2013-11-26T21:32:35.623Z", "content": "[quote=James King;35318]
\nI guess I assumed that someone who knew what they were doing (usually it's me) prepared the data and would not erroneously create non-disjoint train and test sets. Will we be getting a clean test set (one with observations we haven't seen yet) at some point in the future?
\n[/quote]
\nNo need to attack the competency of the people who prepared this, nor do you need to know the motivations for why the problem is structured the way it is. Want a job at Facebook? Welcome, please submit and do your best to maximize that F-score! Want to complain about the problem? You're still welcome, but you are also wasting your time.
\nThere won't be a new test set. The problem is meant to test more than just blind random foresting, and is still valid for relative comparisons. Also, to the complaints that duplicates spoil the academic purity, so do a hundred other factors (data source, question composition, sampling, moderation, etc.) This is a recruiting competition, not a research benchmark problem. If you're here, it's for a job, the glory, the experience, or the sweet sweet Kaggle points.
", "votes": 5 }, { "id": 35321, "postDate": "2013-11-26T21:37:14.337Z", "content": "Yeah, but this takes all the fun out of it. There is legitimate challenge, then there are tricks and gimmicks. We all have to make certain assumptions to function in life--if we second guess everthing, there is no end. Assuming a competition is based on legitimate challenges, not goofy gotcha's is one of those assumptions.
\nBesides, not scores could come down to a noise level. If 60% of the signal is so strong--because it's the same data...can't get much stronger than that, then legitimate signal, as a ratio, is essentially dwarfed.
", "votes": 3 }, { "id": 35284, "postDate": "2013-11-26T15:18:00.373Z", "content": "You are allowed to use the fact there are duplicates. Consider it a screen on people who care enough to check for these sorts of things ;)
", "votes": 4 }, { "id": 35333, "postDate": "2013-11-27T03:33:19.450Z", "content": "I didn't read James King's post as attacking. I can see how it was taken that way, but I took it as sincere.
\nI think an issue now exists between those people who read the forums and those that don't. Those that do now can get a minimum baseline that is quite high.
\nIf the duplicates were intentional, what else is there that hasn't been revealed? Is it altogether possible that a score of ~100% accuracy is possible and it is just a case of finding out how? I can't help but wonder what differentiates the duplicates from the non-duplicates. If intentional, then I believe that something could be derived from those that are not duplicate. That is an interesting problem, but I wonder if it is the problem.
\nFacebook created the contest with the apparent intention to hire people from it. Could Facebook afford to interview everyone that enters a submission? Well, duh, of course they do. They can use this contest as a common ground for discussing in the interview. Not everyone will submit an entry so it is a way to discern those that can walk the walk.
\nI presumed that this contest was to measure real world performance that could translate to what would be possible if the person was hired at Facebook. Leaking of data from training set to test set isn't something that would happen in the real world the way it has here. There's no way to leverage anomalies like these on real world data.
\nThe sweet kaggle points will only be forthcoming to those who leverage the duplicate data while the recruiting effort will likely be unaffected.
", "votes": 1 }, { "id": 35318, "postDate": "2013-11-26T21:13:37.597Z", "content": "I guess I assumed that someone who knew what they were doing (usually it's me) prepared the data and would not erroneously create non-disjoint train and test sets. Will we be getting a clean test set (one with observations we haven't seen yet) at some point in the future?
\n\n
There are 3 reasons this could happen:
\n\n
1. Person preparing the data made a mistake.
\n2. Peron preparing the data did not understand what train vs. test meant.
\n3. Trick question.
\n\n
Which one was it?
", "votes": 1 }, { "id": 35308, "postDate": "2013-11-26T19:27:15.943Z", "content": "The way I see it is quite simple. This is a recruiting competition, and FB's goal is to find people with certain technical skills and other non technical traits. What that combination exactly looks like, they know. Perhaps for them it's more important for a potential hire to bring the type of skepticism that is so important in data science (which, in our case, meant being willing to challenge the hypothesis that test and train did not have overlapping observation), rather than super advanced machine learning skills. Forget a second about Kaggle and the type of 0.00001 improvements that characterizes most competitions. Perhaps for FB being able to quickly identify the low hanging fruit is more important. In that sense, regardless of how this finishes, the folks who got the 0.7 scores the earliest are already on FB's call list.
\nWhile f1 is the competition metric, FB's actual evaluation metric of potential candidates might be very different.
\nG
\n", "votes": 1 }, { "id": 35289, "postDate": "2013-11-26T15:40:39.433Z", "content": "
You mean people like me ? :-) I'm sure lots of people on Kaggle are careful, but we all use, in a way or another, methods and papers published by scientists that would be annoyed to see that it's not mentioned that the performance on this competition is obtained thanks to a particular bias... It just hurts very scientific minds like Damien and I... ;-)
", "votes": 1 }, { "id": 36165, "postDate": "2013-12-14T05:51:28.270Z", "content": "Jason Z:
\nWell, you can either submit 1 entry with duplicates, 1 without, or you can quickly establish a baseline for the duplicates. With the baseline established, make an estimated guess as to what the accuracy of the duplicates is and extrapolate the rest in a weighted average:
\nFor instance, I'll assume that duplicates make up 60% of the posts (weighting of 0.6), and return an total score of about 0.57 when submitted only by themselves. That leaves the rest of the score to be made up with the actual model, with a weighting of 0.4. Lets say that we get about 0.45 for our actual model. The number of posts is relatively irrelevant as opposed to the rates, so lets just choose 100 because its nice and round.
\nFor these figures, it gives us an estimated score of 57 + (actual model rate) * (0.4 x 100) Plug in numbers and come up with an estimate. Here if we plug in our estimated rate of 0.45, we get 57 + 0.45*40 = 75. Which says that your score of 0.75 equates to an actual model performance of about 0.45 without the duplicates. Its a back of the envelope calculation, but its not bad, if i've written it down right.
\nml_learner:
\nLet me rephrase. Getting that last 1% or so can be A LOT of work. When I say its splitting hairs, what I mean is that:
\na) its usually about cleaning data/optimizing model parameters by this point, which personally, I find easy but tedious compared to creating a successful model in the first place.
\nb) without additional data or metrics, its often impossible to tell whether you're in fact generating a worse/overfitted model that doesn't generalise well in practice vs actually making improvements for all that work.
\nFrom my personal experience, its almost always the case that people will tend to put all this work into marginally overfitting...especially when there is no other empirical metric for how well their model actually performs. Maybe its a cultural thing about chasing grades or something :P
", "votes": 2 }, { "id": 35262, "postDate": "2013-11-26T12:03:20.190Z", "content": "I ended up submitting mine since I need a test case to check that I could actually make a submission and had already put it together.
\nAnyway, I think checking for dupes within the training data is within the bounds of reasonable activity. But dupes ACROSS training/test seems like error, and exploiting it seems more about gaming the system to me than actual machine learning.
\nIt could be argued that the algorithms of the "top ranked" players actually could be producing worse algorithms/techniques in the sense of machine learning, as this is a form of strongly overfitting the data and producing a very un-generalisable algorithm. That such algorithms happen to be score the highest suggests errors in specification and application of the competition, in my honest opinion.
", "votes": 2 }, { "id": 35261, "postDate": "2013-11-26T11:45:33.820Z", "content": "[quote=Damien Melksham;35254]I was also concerned that, given this, some of the better machine learning algorithms were almost certainly in the 0.25 - 0.5 range of the leaderboard and might miss out on consideration by facebook.[/quote]
\nChecking for dupes is part of the challenge (part of messy data sets, like missing values). Sharing this data on duplicates and posting pseudo-code benchmarks might be counter to the rules which state:
\nNo sharing outside teams
\nAny sharing code or data outside of teams is not permitted. This includes making code or data made available to all players, such as on the forums.
\nIt is clear that the top ranked players either already knew this (by properly studying the data sets themselves) or that their algorithms automatically picked up on this (making use of the duplicate data to perform better).
", "votes": 2 }, { "id": 36163, "postDate": "2013-12-14T04:16:42.170Z", "content": "I think I probably share this dilemma with other people, but for someone who just relatively recently started this competition and have now read the forums, what would be the best way to proceed? On one hand, I would like to get a high ranking, but on the other hand, I'd love to really know how well my methods work on the test set without using the duplicates issue. And I'm pretty sure creating two accounts to test both cases would be against the rules....any thoughts?
" }, { "id": 36162, "postDate": "2013-12-14T02:02:23.053Z", "content": "Interesting...my latest fix actually bumped me up more than I thought it would and was actually quite arbitrary, so i might be able to eek out a few more points from it.
\nThere's a small number of things i know people could do to beat my entry at the moment that could result in some marginal gains. If they're using established libraries (or i suppose if they coded some decent extra abilities themselves), had access to more processing power than myself, and were willing to let some of these algorithms run for longer, and had more time, I can see how they could do better than my entries.
\nAt this stage though I view it as kind of splitting hairs. 0.77/0.78/0.79 is pretty darn good. There's also a fine line between gaming the system/overfitting/actual best generalisable model, but such is the scoreboard and nature of the competition i guess.
\nI also have one or two more theories to test before the deadline if i get round to it, but they're closer to the over-fitting side of things...
\n\n
edit: although I'm still seeing some imbalances in the relative predictions my model is making, so maybe there's room for improvement yet....
" }, { "id": 36161, "postDate": "2013-12-14T01:38:59.913Z", "content": "I'm at 0.48 actually using 20,000 tags. I'm working on a new submission that should get me higher, but I can't achieve the 0.55 or so implied by the highest submissions with my current methods. The leaders are either very good at text mining or they found another leak.
" }, { "id": 36133, "postDate": "2013-12-12T21:00:04.533Z", "content": "[QUOTE]I interpret the response to mean that the duplicates were deliberate. I don't want a job at facebook, I want to know how well the response can be predicted on fresh data. Has anyone beat 50% by fair means (not using the dupes?) I am at slightly below 50% with models and also by hand tagging samples.[/QUOTE]
\nI'm slowly creeping up the leader-board. Always wishing for more time. Damn work :P
\nI'm working under the following restrictions:
\n- I won't manually tag anything.
\n- I can't get information from stackoverflow (barring my usual use of it for non-related work :P)
\n- I'm not using other people's models. I have to code it all myself.
\n- I'm not using libraries I consider non-trivial (i believe I've used a csv and an arbitrary sequence splitter off the top of my head, I consider stemming, specific parsers, distance metrics, n-gram libraries, etc non-trivial)
\n- I debated on whether it was ok to use the internet and external resources to reference what was valid html/code/etc from different sources. I've decided against it.
\nSince we're running out of time, I don't know if i'll get round to submitting another entry without the duplicates just for kicks, but you can get a decent estimate of what the other-wise average f1 score of entries aware of the duplicates must be at the top of the leaderboard.
\nI'm estimating mine at just about/above 0.50 currently predicting the first 10000 tags. I believe, however they're doing it, that the top entries at the time of writing can be deduced to be around 0.55-0.6 if we make some basic loose assumptions about the duplicates.
" }, { "id": 35946, "postDate": "2013-12-07T15:27:14.473Z", "content": "[quote=James King;35318]
\nThere are 3 reasons this could happen:
\n\n
1. Person preparing the data made a mistake.
\n2. Peron preparing the data did not understand what train vs. test meant.
\n3. Trick question.
\n\n
Which one was it?
\n[/quote]
\n\n
There is a more logical fourth reason. A perfectly valid model can be built without using the training set at all by extracting tags from the title and body in the test data set. A better approximation would be to weight words by looking at the tags in the training set, still ignoring the title and body. This would be a ~42 000 rows dataset that perfectly fits in memory. To go further you will need, at some point to start parsing the >6 million rows dataset which has to be dealt with as an out-of-core problem in most consumer grade machines. Facebook knows that, and most probably is using this fact as a screening factor.
" }, { "id": 35349, "postDate": "2013-11-27T13:21:19.480Z", "content": "I think from now on all participants must be made to know about duplicates for a fairer competition
" }, { "id": 35336, "postDate": "2013-11-27T04:39:47.230Z", "content": ""There are duplicates in the test set", I see this as "there are still unknown data in the test set to be predicted", means scope beating the 'duplicate+top 5 words' benchmark. It's like 'Is the glass half empty or half full ?'
" }, { "id": 35334, "postDate": "2013-11-27T04:19:32.087Z", "content": "Don't be a dilettante mate. This is about data science. It isn't about tricks, it is about sussing out the core information from the data presented. It isn't about Hadoop, R, whatever. It is about trying to figure out how the brain works, by whatever means possible. Facebook likes graphs, myself I think it is all multi-layered, i.e. figure out X% on pass 1, figure out Y% given another algorithm. How many cortical regions handle visual input? When Hubel & Wiesel won the Nobel N years ago it was 10 or so. Now, X times that. Same with this data. Just look for the different ways to parse it, hence the challenge.
" }, { "id": 35322, "postDate": "2013-11-26T22:25:47.667Z", "content": "William, admit this is quite hard to believe that this was done on purpose, and if it was, then we are some guys that misunderstood the point. You can't blame people obsessed with understanding hidden patterns to be obsessed with understanding a hidden pattern ! (which I must confess is close to insanity :-))
" }, { "id": 35320, "postDate": "2013-11-26T21:35:59.553Z", "content": "I interpret the response to mean that the duplicates were deliberate. I don't want a job at facebook, I want to know how well the response can be predicted on fresh data. Has anyone beat 50% by fair means (not using the dupes?) I am at slightly below 50% with models and also by hand tagging samples.
" }, { "id": 35316, "postDate": "2013-11-26T21:06:20.100Z", "content": "I remain skeptical about a competition that favors machine learning techniques that are prone to overfit... To be honest, I would have prefer the same competition without this particular bias ! Because this is a cool competition anyway...
" }, { "id": 35312, "postDate": "2013-11-26T20:23:42.187Z", "content": "I think that a F1 score of 0.7 doesn't necessarily means that someone figured out or cared to check for duplicates between the training and testing data sets. There are certain algorithms (e.g. KNN) which will automatically handle duplicates very well.
" }, { "id": 35278, "postDate": "2013-11-26T14:40:50.010Z", "content": "I may have a philosophical disagreement which Facebook then :-) I'm sorry if I broke the rule about sharing information, but to be fair, I think nobody complains about those who unveil the duplicate entries in the training set. So where is the limit ?
" }, { "id": 35274, "postDate": "2013-11-26T14:09:35.540Z", "content": "I agree with Triskelion.
\n[quote=Triskelion;35261]
\nChecking for dupes is part of the challenge (part of messy data sets, like missing values). Sharing this data on duplicates and posting pseudo-code benchmarks might be counter to the rules which state:
\nNo sharing outside teams
\nAny sharing code or data outside of teams is not permitted. This includes making code or data made available to all players, such as on the forums.
\nIt is clear that the top ranked players either already knew this (by properly studying the data sets themselves) or that their algorithms automatically picked up on this (making use of the duplicate data to perform better).
\n[/quote]
\nIn other competitions there is a rule that says that you can share data on the forums but in this one you can not.
\nObviously, everyone in the leaderboard with a score higher than 0.7 knew about the duplicates.
\nI suppose facebook is looking for people capable of discover information like this by themselves.
\nTwo years ago in the first facebook competition something similar happened and kaggle had to do something about it. And one year ago kaggle had to disable the forum in the second competition to avoid sharing information.
\n\n
" }, { "id": 35267, "postDate": "2013-11-26T13:05:17.157Z", "content": "
A machine learning competition that encourages overfitting looks like a tennis game where you make points when the ball is out... Funny but the winner may not be Roger Federer :-)
" }, { "id": 35264, "postDate": "2013-11-26T12:31:29.427Z", "content": "[quote=Damien Melksham;35262]
\nAnyway, I think checking for dupes within the training data is within the bounds of reasonable activity. But dupes ACROSS training/test seems like error, and exploiting it seems more about gaming the system to me than actual machine learning.
\n[/quote]
\nI can agree with your view, but I just don't think that a Kaggle competition is actual machine learning. It is a bit more than that. It is a competition with game elements. Your model does not even need to learn or use machine learning algo's. It just needs to perform well.
\nIn actual machine learning you would probably not run TFIDF over both the test and train set. In a Kaggle competition that will give you a higher leader board score. You would not deep learn/pre-train your networks on the test set. In a Kaggle competition that can give you a few improvements. In actual machine learning you would be concerned with the practicality, scale and the prediction speed. In a Kaggle competition you just care to create a predicting file in a reasonable time frame (which may be far too wide for predicting at a Facebook scale).
\nSo there are a few differences. Of course they are similar and we would love to join competitions where results actually have a real-life ML application. But in the end it is a game, a competition. Everything within the rules (including making use of duplicate data and other inter-data leaks) is allowed, and useful, provided it gives you a higher game score. If you do not "exploit" that, someone else will.
" }, { "id": 35254, "postDate": "2013-11-26T09:46:04.297Z", "content": "Thanks Chester,
\nYou beat me to it. After my other thread, I came home from work today, compared the titles from the two data sets, and was going to submit the exact same entry as you just to see what would happen.
\nYou've both helped solved the mystery of my other thread, and saved me putting together such a submission. So much thanks.
\nI was also wondering about a mysterious 0.05 gap that appeared in scores on the the leader boards at about the 0.60 mark, which suggested something different about everyone's entries above that given the relatively smooth gradations up to that point, and now its been found.
\nI was also concerned that, given this, some of the better machine learning algorithms were almost certainly in the 0.25 - 0.5 range of the leaderboard and might miss out on consideration by facebook.
" }, { "id": 35252, "postDate": "2013-11-26T08:28:55.940Z", "content": "I imagine that the duplicates common to both training and testing data was not intended, and exploiting this would be likely to be considered to be not in the spirit of the competition.
\nIn any case, the leaderboard is going to get misleading, quite quickly as people use this information. I suppose Kaggle may consider re-evaluating submissions based on the subset of non-duplicated test data. Alternatively, the playing field could be levelled by sanctioning the exploitation of the duplicate data.
\nCould an admin comment on how they'd like us to proceed with this?
" }, { "id": 35250, "postDate": "2013-11-26T07:21:26.177Z", "content": "This is a really good point. I wasn't aware of the duplicates between the training/testing data sets before, but after taking that into consideration, it improved my score dramatically.
" }, { "id": 36164, "postDate": "2013-12-14T05:23:26.727Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 35319, "author_name": "Will Cukierski", "author_url": "", "post_date": "2013-11-26T21:32:35.623000", "content": "[quote=James King;35318]
\nI guess I assumed that someone who knew what they were doing (usually it's me) prepared the data and would not erroneously create non-disjoint train and test sets. Will we be getting a clean test set (one with observations we haven't seen yet) at some point in the future?
\n[/quote]
\nNo need to attack the competency of the people who prepared this, nor do you need to know the motivations for why the problem is structured the way it is. Want a job at Facebook? Welcome, please submit and do your best to maximize that F-score! Want to complain about the problem? You're still welcome, but you are also wasting your time.
\nThere won't be a new test set. The problem is meant to test more than just blind random foresting, and is still valid for relative comparisons. Also, to the complaints that duplicates spoil the academic purity, so do a hundred other factors (data source, question composition, sampling, moderation, etc.) This is a recruiting competition, not a research benchmark problem. If you're here, it's for a job, the glory, the experience, or the sweet sweet Kaggle points.
", "votes": 5, "replies": [] }, { "id": 35321, "author_name": "James Madison", "author_url": "", "post_date": "2013-11-26T21:37:14.337000", "content": "Yeah, but this takes all the fun out of it. There is legitimate challenge, then there are tricks and gimmicks. We all have to make certain assumptions to function in life--if we second guess everthing, there is no end. Assuming a competition is based on legitimate challenges, not goofy gotcha's is one of those assumptions.
\nBesides, not scores could come down to a noise level. If 60% of the signal is so strong--because it's the same data...can't get much stronger than that, then legitimate signal, as a ratio, is essentially dwarfed.
", "votes": 3, "replies": [] }, { "id": 35284, "author_name": "Will Cukierski", "author_url": "", "post_date": "2013-11-26T15:18:00.373000", "content": "You are allowed to use the fact there are duplicates. Consider it a screen on people who care enough to check for these sorts of things ;)
", "votes": 4, "replies": [] }, { "id": 35333, "author_name": "jgoalby", "author_url": "", "post_date": "2013-11-27T03:33:19.450000", "content": "I didn't read James King's post as attacking. I can see how it was taken that way, but I took it as sincere.
\nI think an issue now exists between those people who read the forums and those that don't. Those that do now can get a minimum baseline that is quite high.
\nIf the duplicates were intentional, what else is there that hasn't been revealed? Is it altogether possible that a score of ~100% accuracy is possible and it is just a case of finding out how? I can't help but wonder what differentiates the duplicates from the non-duplicates. If intentional, then I believe that something could be derived from those that are not duplicate. That is an interesting problem, but I wonder if it is the problem.
\nFacebook created the contest with the apparent intention to hire people from it. Could Facebook afford to interview everyone that enters a submission? Well, duh, of course they do. They can use this contest as a common ground for discussing in the interview. Not everyone will submit an entry so it is a way to discern those that can walk the walk.
\nI presumed that this contest was to measure real world performance that could translate to what would be possible if the person was hired at Facebook. Leaking of data from training set to test set isn't something that would happen in the real world the way it has here. There's no way to leverage anomalies like these on real world data.
\nThe sweet kaggle points will only be forthcoming to those who leverage the duplicate data while the recruiting effort will likely be unaffected.
", "votes": 1, "replies": [] }, { "id": 35318, "author_name": "James King", "author_url": "", "post_date": "2013-11-26T21:13:37.597000", "content": "I guess I assumed that someone who knew what they were doing (usually it's me) prepared the data and would not erroneously create non-disjoint train and test sets. Will we be getting a clean test set (one with observations we haven't seen yet) at some point in the future?
\n\n
There are 3 reasons this could happen:
\n\n
1. Person preparing the data made a mistake.
\n2. Peron preparing the data did not understand what train vs. test meant.
\n3. Trick question.
\n\n
Which one was it?
", "votes": 1, "replies": [] }, { "id": 35308, "author_name": "Giulio", "author_url": "", "post_date": "2013-11-26T19:27:15.943000", "content": "The way I see it is quite simple. This is a recruiting competition, and FB's goal is to find people with certain technical skills and other non technical traits. What that combination exactly looks like, they know. Perhaps for them it's more important for a potential hire to bring the type of skepticism that is so important in data science (which, in our case, meant being willing to challenge the hypothesis that test and train did not have overlapping observation), rather than super advanced machine learning skills. Forget a second about Kaggle and the type of 0.00001 improvements that characterizes most competitions. Perhaps for FB being able to quickly identify the low hanging fruit is more important. In that sense, regardless of how this finishes, the folks who got the 0.7 scores the earliest are already on FB's call list.
\nWhile f1 is the competition metric, FB's actual evaluation metric of potential candidates might be very different.
\nG
\n", "votes": 1, "replies": [] }, { "id": 35289, "author_name": "elrude", "author_url": "", "post_date": "2013-11-26T15:40:39.433000", "content": "
You mean people like me ? :-) I'm sure lots of people on Kaggle are careful, but we all use, in a way or another, methods and papers published by scientists that would be annoyed to see that it's not mentioned that the performance on this competition is obtained thanks to a particular bias... It just hurts very scientific minds like Damien and I... ;-)
", "votes": 1, "replies": [] }, { "id": 36165, "author_name": "Damien Melksham", "author_url": "", "post_date": "2013-12-14T05:51:28.270000", "content": "Jason Z:
\nWell, you can either submit 1 entry with duplicates, 1 without, or you can quickly establish a baseline for the duplicates. With the baseline established, make an estimated guess as to what the accuracy of the duplicates is and extrapolate the rest in a weighted average:
\nFor instance, I'll assume that duplicates make up 60% of the posts (weighting of 0.6), and return an total score of about 0.57 when submitted only by themselves. That leaves the rest of the score to be made up with the actual model, with a weighting of 0.4. Lets say that we get about 0.45 for our actual model. The number of posts is relatively irrelevant as opposed to the rates, so lets just choose 100 because its nice and round.
\nFor these figures, it gives us an estimated score of 57 + (actual model rate) * (0.4 x 100) Plug in numbers and come up with an estimate. Here if we plug in our estimated rate of 0.45, we get 57 + 0.45*40 = 75. Which says that your score of 0.75 equates to an actual model performance of about 0.45 without the duplicates. Its a back of the envelope calculation, but its not bad, if i've written it down right.
\nml_learner:
\nLet me rephrase. Getting that last 1% or so can be A LOT of work. When I say its splitting hairs, what I mean is that:
\na) its usually about cleaning data/optimizing model parameters by this point, which personally, I find easy but tedious compared to creating a successful model in the first place.
\nb) without additional data or metrics, its often impossible to tell whether you're in fact generating a worse/overfitted model that doesn't generalise well in practice vs actually making improvements for all that work.
\nFrom my personal experience, its almost always the case that people will tend to put all this work into marginally overfitting...especially when there is no other empirical metric for how well their model actually performs. Maybe its a cultural thing about chasing grades or something :P
", "votes": 2, "replies": [] }, { "id": 35262, "author_name": "Damien Melksham", "author_url": "", "post_date": "2013-11-26T12:03:20.190000", "content": "I ended up submitting mine since I need a test case to check that I could actually make a submission and had already put it together.
\nAnyway, I think checking for dupes within the training data is within the bounds of reasonable activity. But dupes ACROSS training/test seems like error, and exploiting it seems more about gaming the system to me than actual machine learning.
\nIt could be argued that the algorithms of the "top ranked" players actually could be producing worse algorithms/techniques in the sense of machine learning, as this is a form of strongly overfitting the data and producing a very un-generalisable algorithm. That such algorithms happen to be score the highest suggests errors in specification and application of the competition, in my honest opinion.
", "votes": 2, "replies": [] }, { "id": 35261, "author_name": "Triskelion", "author_url": "", "post_date": "2013-11-26T11:45:33.820000", "content": "[quote=Damien Melksham;35254]I was also concerned that, given this, some of the better machine learning algorithms were almost certainly in the 0.25 - 0.5 range of the leaderboard and might miss out on consideration by facebook.[/quote]
\nChecking for dupes is part of the challenge (part of messy data sets, like missing values). Sharing this data on duplicates and posting pseudo-code benchmarks might be counter to the rules which state:
\nNo sharing outside teams
\nAny sharing code or data outside of teams is not permitted. This includes making code or data made available to all players, such as on the forums.
\nIt is clear that the top ranked players either already knew this (by properly studying the data sets themselves) or that their algorithms automatically picked up on this (making use of the duplicate data to perform better).
", "votes": 2, "replies": [] }, { "id": 36163, "author_name": "Jason Z.", "author_url": "", "post_date": "2013-12-14T04:16:42.170000", "content": "I think I probably share this dilemma with other people, but for someone who just relatively recently started this competition and have now read the forums, what would be the best way to proceed? On one hand, I would like to get a high ranking, but on the other hand, I'd love to really know how well my methods work on the test set without using the duplicates issue. And I'm pretty sure creating two accounts to test both cases would be against the rules....any thoughts?
", "votes": 0, "replies": [] }, { "id": 36162, "author_name": "Damien Melksham", "author_url": "", "post_date": "2013-12-14T02:02:23.053000", "content": "Interesting...my latest fix actually bumped me up more than I thought it would and was actually quite arbitrary, so i might be able to eek out a few more points from it.
\nThere's a small number of things i know people could do to beat my entry at the moment that could result in some marginal gains. If they're using established libraries (or i suppose if they coded some decent extra abilities themselves), had access to more processing power than myself, and were willing to let some of these algorithms run for longer, and had more time, I can see how they could do better than my entries.
\nAt this stage though I view it as kind of splitting hairs. 0.77/0.78/0.79 is pretty darn good. There's also a fine line between gaming the system/overfitting/actual best generalisable model, but such is the scoreboard and nature of the competition i guess.
\nI also have one or two more theories to test before the deadline if i get round to it, but they're closer to the over-fitting side of things...
\n\n
edit: although I'm still seeing some imbalances in the relative predictions my model is making, so maybe there's room for improvement yet....
", "votes": 0, "replies": [] }, { "id": 36161, "author_name": "James King", "author_url": "", "post_date": "2013-12-14T01:38:59.913000", "content": "I'm at 0.48 actually using 20,000 tags. I'm working on a new submission that should get me higher, but I can't achieve the 0.55 or so implied by the highest submissions with my current methods. The leaders are either very good at text mining or they found another leak.
", "votes": 0, "replies": [] }, { "id": 36133, "author_name": "Damien Melksham", "author_url": "", "post_date": "2013-12-12T21:00:04.533000", "content": "[QUOTE]I interpret the response to mean that the duplicates were deliberate. I don't want a job at facebook, I want to know how well the response can be predicted on fresh data. Has anyone beat 50% by fair means (not using the dupes?) I am at slightly below 50% with models and also by hand tagging samples.[/QUOTE]
\nI'm slowly creeping up the leader-board. Always wishing for more time. Damn work :P
\nI'm working under the following restrictions:
\n- I won't manually tag anything.
\n- I can't get information from stackoverflow (barring my usual use of it for non-related work :P)
\n- I'm not using other people's models. I have to code it all myself.
\n- I'm not using libraries I consider non-trivial (i believe I've used a csv and an arbitrary sequence splitter off the top of my head, I consider stemming, specific parsers, distance metrics, n-gram libraries, etc non-trivial)
\n- I debated on whether it was ok to use the internet and external resources to reference what was valid html/code/etc from different sources. I've decided against it.
\nSince we're running out of time, I don't know if i'll get round to submitting another entry without the duplicates just for kicks, but you can get a decent estimate of what the other-wise average f1 score of entries aware of the duplicates must be at the top of the leaderboard.
\nI'm estimating mine at just about/above 0.50 currently predicting the first 10000 tags. I believe, however they're doing it, that the top entries at the time of writing can be deduced to be around 0.55-0.6 if we make some basic loose assumptions about the duplicates.
", "votes": 0, "replies": [] }, { "id": 35946, "author_name": "elyase", "author_url": "", "post_date": "2013-12-07T15:27:14.473000", "content": "[quote=James King;35318]
\nThere are 3 reasons this could happen:
\n\n
1. Person preparing the data made a mistake.
\n2. Peron preparing the data did not understand what train vs. test meant.
\n3. Trick question.
\n\n
Which one was it?
\n[/quote]
\n\n
There is a more logical fourth reason. A perfectly valid model can be built without using the training set at all by extracting tags from the title and body in the test data set. A better approximation would be to weight words by looking at the tags in the training set, still ignoring the title and body. This would be a ~42 000 rows dataset that perfectly fits in memory. To go further you will need, at some point to start parsing the >6 million rows dataset which has to be dealt with as an out-of-core problem in most consumer grade machines. Facebook knows that, and most probably is using this fact as a screening factor.
", "votes": 0, "replies": [] }, { "id": 35349, "author_name": "Güngör Yılmaz", "author_url": "", "post_date": "2013-11-27T13:21:19.480000", "content": "I think from now on all participants must be made to know about duplicates for a fairer competition
", "votes": 0, "replies": [] }, { "id": 35336, "author_name": "Chitrasen", "author_url": "", "post_date": "2013-11-27T04:39:47.230000", "content": ""There are duplicates in the test set", I see this as "there are still unknown data in the test set to be predicted", means scope beating the 'duplicate+top 5 words' benchmark. It's like 'Is the glass half empty or half full ?'
", "votes": 0, "replies": [] }, { "id": 35334, "author_name": "", "author_url": "", "post_date": "2013-11-27T04:19:32.087000", "content": "Don't be a dilettante mate. This is about data science. It isn't about tricks, it is about sussing out the core information from the data presented. It isn't about Hadoop, R, whatever. It is about trying to figure out how the brain works, by whatever means possible. Facebook likes graphs, myself I think it is all multi-layered, i.e. figure out X% on pass 1, figure out Y% given another algorithm. How many cortical regions handle visual input? When Hubel & Wiesel won the Nobel N years ago it was 10 or so. Now, X times that. Same with this data. Just look for the different ways to parse it, hence the challenge.
", "votes": 0, "replies": [] }, { "id": 35322, "author_name": "elrude", "author_url": "", "post_date": "2013-11-26T22:25:47.667000", "content": "William, admit this is quite hard to believe that this was done on purpose, and if it was, then we are some guys that misunderstood the point. You can't blame people obsessed with understanding hidden patterns to be obsessed with understanding a hidden pattern ! (which I must confess is close to insanity :-))
", "votes": 0, "replies": [] }, { "id": 35320, "author_name": "James King", "author_url": "", "post_date": "2013-11-26T21:35:59.553000", "content": "I interpret the response to mean that the duplicates were deliberate. I don't want a job at facebook, I want to know how well the response can be predicted on fresh data. Has anyone beat 50% by fair means (not using the dupes?) I am at slightly below 50% with models and also by hand tagging samples.
", "votes": 0, "replies": [] }, { "id": 35316, "author_name": "elrude", "author_url": "", "post_date": "2013-11-26T21:06:20.100000", "content": "I remain skeptical about a competition that favors machine learning techniques that are prone to overfit... To be honest, I would have prefer the same competition without this particular bias ! Because this is a cool competition anyway...
", "votes": 0, "replies": [] }, { "id": 35312, "author_name": "Andrei Bozantan", "author_url": "", "post_date": "2013-11-26T20:23:42.187000", "content": "I think that a F1 score of 0.7 doesn't necessarily means that someone figured out or cared to check for duplicates between the training and testing data sets. There are certain algorithms (e.g. KNN) which will automatically handle duplicates very well.
", "votes": 0, "replies": [] }, { "id": 35278, "author_name": "elrude", "author_url": "", "post_date": "2013-11-26T14:40:50.010000", "content": "I may have a philosophical disagreement which Facebook then :-) I'm sorry if I broke the rule about sharing information, but to be fair, I think nobody complains about those who unveil the duplicate entries in the training set. So where is the limit ?
", "votes": 0, "replies": [] }, { "id": 35274, "author_name": "Roberto", "author_url": "", "post_date": "2013-11-26T14:09:35.540000", "content": "I agree with Triskelion.
\n[quote=Triskelion;35261]
\nChecking for dupes is part of the challenge (part of messy data sets, like missing values). Sharing this data on duplicates and posting pseudo-code benchmarks might be counter to the rules which state:
\nNo sharing outside teams
\nAny sharing code or data outside of teams is not permitted. This includes making code or data made available to all players, such as on the forums.
\nIt is clear that the top ranked players either already knew this (by properly studying the data sets themselves) or that their algorithms automatically picked up on this (making use of the duplicate data to perform better).
\n[/quote]
\nIn other competitions there is a rule that says that you can share data on the forums but in this one you can not.
\nObviously, everyone in the leaderboard with a score higher than 0.7 knew about the duplicates.
\nI suppose facebook is looking for people capable of discover information like this by themselves.
\nTwo years ago in the first facebook competition something similar happened and kaggle had to do something about it. And one year ago kaggle had to disable the forum in the second competition to avoid sharing information.
\n\n
", "votes": 0, "replies": [] }, { "id": 35267, "author_name": "elrude", "author_url": "", "post_date": "2013-11-26T13:05:17.157000", "content": "
A machine learning competition that encourages overfitting looks like a tennis game where you make points when the ball is out... Funny but the winner may not be Roger Federer :-)
", "votes": 0, "replies": [] }, { "id": 35264, "author_name": "Triskelion", "author_url": "", "post_date": "2013-11-26T12:31:29.427000", "content": "[quote=Damien Melksham;35262]
\nAnyway, I think checking for dupes within the training data is within the bounds of reasonable activity. But dupes ACROSS training/test seems like error, and exploiting it seems more about gaming the system to me than actual machine learning.
\n[/quote]
\nI can agree with your view, but I just don't think that a Kaggle competition is actual machine learning. It is a bit more than that. It is a competition with game elements. Your model does not even need to learn or use machine learning algo's. It just needs to perform well.
\nIn actual machine learning you would probably not run TFIDF over both the test and train set. In a Kaggle competition that will give you a higher leader board score. You would not deep learn/pre-train your networks on the test set. In a Kaggle competition that can give you a few improvements. In actual machine learning you would be concerned with the practicality, scale and the prediction speed. In a Kaggle competition you just care to create a predicting file in a reasonable time frame (which may be far too wide for predicting at a Facebook scale).
\nSo there are a few differences. Of course they are similar and we would love to join competitions where results actually have a real-life ML application. But in the end it is a game, a competition. Everything within the rules (including making use of duplicate data and other inter-data leaks) is allowed, and useful, provided it gives you a higher game score. If you do not "exploit" that, someone else will.
", "votes": 0, "replies": [] }, { "id": 35254, "author_name": "Damien Melksham", "author_url": "", "post_date": "2013-11-26T09:46:04.297000", "content": "Thanks Chester,
\nYou beat me to it. After my other thread, I came home from work today, compared the titles from the two data sets, and was going to submit the exact same entry as you just to see what would happen.
\nYou've both helped solved the mystery of my other thread, and saved me putting together such a submission. So much thanks.
\nI was also wondering about a mysterious 0.05 gap that appeared in scores on the the leader boards at about the 0.60 mark, which suggested something different about everyone's entries above that given the relatively smooth gradations up to that point, and now its been found.
\nI was also concerned that, given this, some of the better machine learning algorithms were almost certainly in the 0.25 - 0.5 range of the leaderboard and might miss out on consideration by facebook.
", "votes": 0, "replies": [] }, { "id": 35252, "author_name": "Philip Wahrlich", "author_url": "", "post_date": "2013-11-26T08:28:55.940000", "content": "I imagine that the duplicates common to both training and testing data was not intended, and exploiting this would be likely to be considered to be not in the spirit of the competition.
\nIn any case, the leaderboard is going to get misleading, quite quickly as people use this information. I suppose Kaggle may consider re-evaluating submissions based on the subset of non-duplicated test data. Alternatively, the playing field could be levelled by sanctioning the exploitation of the duplicate data.
\nCould an admin comment on how they'd like us to proceed with this?
", "votes": 0, "replies": [] }, { "id": 35250, "author_name": "Omar O", "author_url": "", "post_date": "2013-11-26T07:21:26.177000", "content": "This is a really good point. I wasn't aware of the duplicates between the training/testing data sets before, but after taking that into consideration, it improved my score dramatically.
", "votes": 0, "replies": [] }, { "id": 36164, "author_name": "", "author_url": "", "post_date": "2013-12-14T05:23:26.727000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "35241": "", "35319": "", "35321": "", "35284": "", "35333": "", "35318": "", "35308": "", "35289": "", "36165": "", "35262": "", "35261": "", "36163": "", "36162": "", "36161": "", "36133": "", "35946": "", "35349": "", "35336": "", "35334": "", "35322": "", "35320": "", "35316": "", "35312": "", "35278": "", "35274": "", "35267": "", "35264": "", "35254": "", "35252": "", "35250": "", "36164": "" } }