Thought I should share what my areas of improvement are and what happened with the methods I employed, in case, some of you find it helpful for the next time we have a recruiting competition:
\n\n
In summary, I couldn't get into the .40s and I think it comes from my using Porter Stemming which pruned a lot of coding terms important in driving significant variation to the tags from the training sample. I could've have traced the stemmed terms back to the original, but that would've have made my computing load even worse, so all that traceability info got lost. In addition to that, I used R. Admittiingly, I started this with the primary intention of "trying" R out to understand better any limitations with large data sets. Quickly, that intention became secondary as I wished I used a compiled language as the R issues became pretty clear (i.e. in-memory limitations, little support for distributed computing, bad hash performance over iterations, etc.) and my time investment became more than I originally wanted, but it was alreadly too late to throw away what I had already done. My path eventually had a speed limit that I simply couldn't overcome.
\nNext time, I want to try using Erlang as I feel its native support for concurrency would have been much better and although its still interpreted, I believe, the concurrency of execution across several old laptops and logical cores would've have overcome the overhead of the interpreted layer.
\nMy computation was kept to basic rules and also the derivation of mutual information sample distributions between tags and terms which were clustered into 5 character clusters which provided the only method I used to score ranking. I didn't think using svms or even decision trees across 42K tags would be feasible with only a quad-core processor and only 8 G ram, although I had 3 laptops to use. Also, keeping the algo to basic rules allows me to better chunk out tasks to Erlang workers, for next time.
\nI'm really impressed with the many folks getting in the 60s and above. Any insights from you folks to the larger community would be very much appreciated and big THANKS to Facebook for continuing this trend of recruiting. This way of recruiting is a no-brainer.
\nPeace Out Folks.
", "messages": [ { "id": "36371", "postDate": "12/19/2013 19:17:34", "content": "Thought I should share what my areas of improvement are and what happened with the methods I employed, in case, some of you find it helpful for the next time we have a recruiting competition:
\n\n
In summary, I couldn't get into the .40s and I think it comes from my using Porter Stemming which pruned a lot of coding terms important in driving significant variation to the tags from the training sample. I could've have traced the stemmed terms back to the original, but that would've have made my computing load even worse, so all that traceability info got lost. In addition to that, I used R. Admittiingly, I started this with the primary intention of "trying" R out to understand better any limitations with large data sets. Quickly, that intention became secondary as I wished I used a compiled language as the R issues became pretty clear (i.e. in-memory limitations, little support for distributed computing, bad hash performance over iterations, etc.) and my time investment became more than I originally wanted, but it was alreadly too late to throw away what I had already done. My path eventually had a speed limit that I simply couldn't overcome.
\nNext time, I want to try using Erlang as I feel its native support for concurrency would have been much better and although its still interpreted, I believe, the concurrency of execution across several old laptops and logical cores would've have overcome the overhead of the interpreted layer.
\nMy computation was kept to basic rules and also the derivation of mutual information sample distributions between tags and terms which were clustered into 5 character clusters which provided the only method I used to score ranking. I didn't think using svms or even decision trees across 42K tags would be feasible with only a quad-core processor and only 8 G ram, although I had 3 laptops to use. Also, keeping the algo to basic rules allows me to better chunk out tasks to Erlang workers, for next time.
\nI'm really impressed with the many folks getting in the 60s and above. Any insights from you folks to the larger community would be very much appreciated and big THANKS to Facebook for continuing this trend of recruiting. This way of recruiting is a no-brainer.
\nPeace Out Folks.
", "rawMarkdown": "", "votes": null }, { "id": "36379", "postDate": "12/19/2013 20:18:36", "content": "The best advice I can give is to read the forum. I'm sure you'll find the "insight" you seek.
", "rawMarkdown": "", "votes": null }, { "id": "36394", "postDate": "12/20/2013 07:56:11", "content": "@David Cho,
\nDoes your score of 0.40 include the use of the duplicates in training\\test?
", "rawMarkdown": "", "votes": null }, { "id": "36403", "postDate": "12/20/2013 14:37:47", "content": "Hi Rudi,
\nThanks for the suggestion and yes, I did account for the duplicates of which I became aware after reading the forum. I counted over 320K duplicates, but my score only jumped up to 39 from 26. I read bigger improvements from other folks, so my rules appear to have worked better on the duplicate records than the non-duplicates. I did spot-check my submission.
\nGood luck to all.
\n", "rawMarkdown": "", "votes": null }, { "id": "36412", "postDate": "12/20/2013 16:27:03", "content": "
I started really basic by simply looking for popular(top 10 000 or so) tags in the title. I added some logic to join tokens like 'windows' and '8' and remove subtags where it seemed appropriate. This all got me a score of ~0.32. All done in C#. Runs in about an hour, no pre-processing required. I left it there.
\nAfter hearing about the free lunch aka the duplicates, I got 0.69
\nDid you rebuild\\retrain your model without using any of the duplicate data? And then at prediction time only use the model if the test case isn't part of the duplicate set?
", "rawMarkdown": "", "votes": null }, { "id": "36420", "postDate": "12/20/2013 17:59:49", "content": "Great job Rudi, hope you get a call from FB. My lessons learned were not to use R and not to use Porter Stemming. I think if I had used any other language even Visual Basic , I would've done much better, but I got invested in R, plus I felt I didn't have time. This kind of competition, mistakes are costly since my model did do some computation (i.e. mutual information distribution sampling between tags and 5 -char clusters) combined with basic rules, obviously different than yours. Once you chose a computation model, it's difficult to go back since R took forever to re-compute based on changes. Your predictions on the non-dups look pretty terrific based on your improvement using the dups. I'm not sure how I got such a dup bias before I knew about the dups, since I only had a fractional improvement below yours. So, I didn't re-train on a smaller training set, but I am with the side that argues more training data is never a bad thing. More data in general should always be more preferable than a more complicated data model, in my opinion.
\n\n
Good luck with your future data science goals.
", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 36379, "author_name": "pvkooten", "author_url": "", "post_date": "12/19/2013 20:18:36", "content": "The best advice I can give is to read the forum. I'm sure you'll find the "insight" you seek.
", "votes": null, "replies": [] }, { "id": 36394, "author_name": "rudikruger", "author_url": "", "post_date": "12/20/2013 07:56:11", "content": "@David Cho,
\nDoes your score of 0.40 include the use of the duplicates in training\\test?
", "votes": null, "replies": [] }, { "id": 36403, "author_name": "davidcho0", "author_url": "", "post_date": "12/20/2013 14:37:47", "content": "Hi Rudi,
\nThanks for the suggestion and yes, I did account for the duplicates of which I became aware after reading the forum. I counted over 320K duplicates, but my score only jumped up to 39 from 26. I read bigger improvements from other folks, so my rules appear to have worked better on the duplicate records than the non-duplicates. I did spot-check my submission.
\nGood luck to all.
\n", "votes": null, "replies": [] }, { "id": 36412, "author_name": "rudikruger", "author_url": "", "post_date": "12/20/2013 16:27:03", "content": "
I started really basic by simply looking for popular(top 10 000 or so) tags in the title. I added some logic to join tokens like 'windows' and '8' and remove subtags where it seemed appropriate. This all got me a score of ~0.32. All done in C#. Runs in about an hour, no pre-processing required. I left it there.
\nAfter hearing about the free lunch aka the duplicates, I got 0.69
\nDid you rebuild\\retrain your model without using any of the duplicate data? And then at prediction time only use the model if the test case isn't part of the duplicate set?
", "votes": null, "replies": [] }, { "id": 36420, "author_name": "davidcho0", "author_url": "", "post_date": "12/20/2013 17:59:49", "content": "Great job Rudi, hope you get a call from FB. My lessons learned were not to use R and not to use Porter Stemming. I think if I had used any other language even Visual Basic , I would've done much better, but I got invested in R, plus I felt I didn't have time. This kind of competition, mistakes are costly since my model did do some computation (i.e. mutual information distribution sampling between tags and 5 -char clusters) combined with basic rules, obviously different than yours. Once you chose a computation model, it's difficult to go back since R took forever to re-compute based on changes. Your predictions on the non-dups look pretty terrific based on your improvement using the dups. I'm not sure how I got such a dup bias before I knew about the dups, since I only had a fractional improvement below yours. So, I didn't re-train on a smaller training set, but I am with the side that argues more training data is never a bad thing. More data in general should always be more preferable than a more complicated data model, in my opinion.
\n\n
Good luck with your future data science goals.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "36371": "", "36379": "", "36394": "", "36403": "", "36412": "", "36420": "" }, "source": "meta" }