Assume forward link for a node is a link going out of the node. And backward link is the link going into the node. Ie the forwards of A is the people A is following. And backwards of A is the people that follow A.
\r\nAssume X is the node for which we want to find forward links for.
\r\nThe algorithm goes like this:
\r\nThe node X has 1,000,000 credits. It splits it to the number of connections(both forwards and backwards) it has and gives them to them.
\r\nEach node that gets credit:
\r\n1. Keeps half of the credits for itself.
\r\n2. Divides the credits it received by the number of links and distributes them. Like X did, but with one difference. Backward links do not get to keep the credits. Only to redistribute them.
\r\n\r\n
Repeat 3 times.
\r\n\r\n
Q & As:
\r\n\r\n
a .Why keep only half the credits?
\r\nFor the backwards links we only want to connect to the links that have a higher degree of being connected back. When a node Y keeps half of the credits and redistributes the rest to its links, it will get credits back for its backward connections that have\r\n them in its forward connections. I.e. the more "friends" it has the more credits it will get back.
\r\n\r\n
b. Why backward connections don't get credits?
\r\nBecause we are interested in forward connections. Also like this we are trying to discover forward only links of node X that where lost. I.e. Lets say nodes A, B, C and D are being followed as a group by a lot of people. Now lets say X is one of them and\r\n it lost its connection to node D.
\r\nNodes A,B,C will get credits, which they will pass to their followers, which in turn they will pass to D. Hence D is discovered!
\r\n\r\n
The code below gives a score of 0.711. So its the new 0!
\r\n\r\n
To run it use the train file as a first argument and the test file as the second. Please strip any headers from those files.
", "messages": [ { "id": "11883", "postDate": "06/26/2012 01:01:04", "content": "Assume forward link for a node is a link going out of the node. And backward link is the link going into the node. Ie the forwards of A is the people A is following. And backwards of A is the people that follow A.
\r\nAssume X is the node for which we want to find forward links for.
\r\nThe algorithm goes like this:
\r\nThe node X has 1,000,000 credits. It splits it to the number of connections(both forwards and backwards) it has and gives them to them.
\r\nEach node that gets credit:
\r\n1. Keeps half of the credits for itself.
\r\n2. Divides the credits it received by the number of links and distributes them. Like X did, but with one difference. Backward links do not get to keep the credits. Only to redistribute them.
\r\n\r\n
Repeat 3 times.
\r\n\r\n
Q & As:
\r\n\r\n
a .Why keep only half the credits?
\r\nFor the backwards links we only want to connect to the links that have a higher degree of being connected back. When a node Y keeps half of the credits and redistributes the rest to its links, it will get credits back for its backward connections that have\r\n them in its forward connections. I.e. the more "friends" it has the more credits it will get back.
\r\n\r\n
b. Why backward connections don't get credits?
\r\nBecause we are interested in forward connections. Also like this we are trying to discover forward only links of node X that where lost. I.e. Lets say nodes A, B, C and D are being followed as a group by a lot of people. Now lets say X is one of them and\r\n it lost its connection to node D.
\r\nNodes A,B,C will get credits, which they will pass to their followers, which in turn they will pass to D. Hence D is discovered!
\r\n\r\n
The code below gives a score of 0.711. So its the new 0!
\r\n\r\n
To run it use the train file as a first argument and the test file as the second. Please strip any headers from those files.
", "rawMarkdown": "", "votes": null }, { "id": "11884", "postDate": "06/26/2012 04:16:04", "content": "It sounds a lot like you just reinvented PageRank/EdgeRank.
", "rawMarkdown": "", "votes": null }, { "id": "11885", "postDate": "06/26/2012 04:34:57", "content": "Never heard of edgerank.. But yeah I guess it looks somewhat like pagerank. But needs quite a lot if tweaking. At the current stage a backward connection with a lot of "friends" will probable get better score than one with less friend but already connected\r\n to the "user's immediate network". Also discovering lost single forward connections may need to get a boost when the user has has high "follower" index or high peer index. Anyway. Lots of ways to tweak it and play around. By the way to compute predictions\r\n for the "test" takes about 15 mins on a 10 node cluster. So anybody trying it be patient..", "rawMarkdown": "", "votes": null }, { "id": "11886", "postDate": "06/26/2012 05:02:20", "content": "Looking at the code, it seems that on the first step you allow backward-connected nodes to retain credit, but on deeper steps you do not (contrast scorenetwork with propagatescore).
", "rawMarkdown": "", "votes": null }, { "id": "11887", "postDate": "06/26/2012 05:25:06", "content": "Yes. That's on purpose. We are interested in making forward connections only with the direct backward connections. 2nd line backwards (or forward->backward) are too of long shot to get connected to them.. (And if the are of interest they shall get some\r\n credit from forward connections.I am much more likely to follow somebody the people that follow me or that I follow, follow. Than to follow somebody who follows the one who follows me..). But they play critical role for transferring the credit to their forwards.\r\n I think I understand your concern of under-crediting a lost "friend". It's something that needs tweaking. Any suggestions?", "rawMarkdown": "", "votes": null }, { "id": "11888", "postDate": "06/26/2012 05:50:05", "content": "The approach makes sense, I was just checking if the description or the code was what you meant.
\r\nHere, at least, it's going to take 20 days to actually run that, so no 0.711 for me!
", "rawMarkdown": "", "votes": null }, { "id": "11891", "postDate": "06/26/2012 12:26:56", "content": "My first attemp is similar to this method, using a modified random walk with restart from each node that need to be predict following nodes.
\r\nThat is, assume that q is node to be predicted, we start random walk with restart with probability 0.15 to return to q, and 0.85 to fly to other nodes.
\r\nAny other nodes except q will receive a random walk score that indicates the probability of reaching this node from q in 20 iteration.
\r\nThis is similar to : instead of using 0.5 to keep credit, I use 0.15, and the repeated times is set to 20.
\r\nBut I only get map@10 around 0.004 and then I give up this method totally.
\r\nVery curious why you achive at 0.71, will try and see:)
", "rawMarkdown": "", "votes": null }, { "id": "11901", "postDate": "06/26/2012 17:00:01", "content": "If you are using python on a linux/mac machine with multiple cores and plenty of ram, this method is very easy to parallelize. See the python documentation for the multiprocessing module: http://docs.python.org/library/multiprocessing.html
\r\nThe attached code runs considerbly faster than 20 hours on 4 cores, also note additional functions added to utilities.py
", "rawMarkdown": "", "votes": null }, { "id": "11902", "postDate": "06/26/2012 17:18:48", "content": "We have removed the code posted to this thread by Den Souleo and others. We encourage contestants to use the competition forums to discuss and learn from each other, but posting full solution code has a direct impact on the integrity of the recruiting competition. \r\n If you receive an interview with Facebook, you will be asked to present code that only reflects your own work. Using code that was posted by another user will not help you, and you should be aware of this before selecting such work as your final submission.
", "rawMarkdown": "", "votes": null }, { "id": "11903", "postDate": "06/26/2012 17:27:06", "content": "You removed the code after 16 hours? Is that fair towards us who did not have a look at the code?
", "rawMarkdown": "", "votes": null }, { "id": "11904", "postDate": "06/26/2012 17:30:22", "content": "With respect, I've had equivalent code up on the forum for nineteen days already. At the time I posted it, this code yielded results not dissimilar in ranking from the current offers in this thread. The result, I believe, was to help kaggle users learn and\r\n advance more quickly, in effect raising the bar. Already with a score of 0.711, this code will scarcely get you into the top ten, and certainly will not by the end of the competition. I don't believe posting code here violates any agreement made by kaggle\r\n users. It is already the case that using code posted by other users will not help you to win. On what grounds can this selective censoring be truly justified?
", "rawMarkdown": "", "votes": null }, { "id": "11905", "postDate": "06/26/2012 17:37:37", "content": "", "rawMarkdown": "", "votes": null }, { "id": "11906", "postDate": "06/26/2012 17:37:41", "content": "[quote=SivaP;11903]
\r\nYou removed the code after 16 hours? Is that fair towards us who did not have a look at the code?
\r\n[/quote]
\r\nWe are discussing the situation with users who downloaded the code during the 16 hour time window
\r\n", "rawMarkdown": "", "votes": null }, { "id": "11908", "postDate": "06/26/2012 17:43:37", "content": "
[quote=Aaron Schumacher;11904]
\r\nWith respect, I've had equivalent code up on the forum for nineteen days already. At the time I posted it, this code yielded results not dissimilar in ranking from the current offers in this thread. The result, I believe, was to help kaggle users learn and\r\n advance more quickly, in effect raising the bar. Already with a score of 0.711, this code will scarcely get you into the top ten, and certainly will not by the end of the competition. I don't believe posting code here violates any agreement made by kaggle\r\n users. It is already the case that using code posted by other users will not help you to win. On what grounds can this selective censoring be truly justified?
\r\n[/quote]
\r\n\r\n
Hi Aaron. I agree with you that the purpose of the forums should be to help Kaggle users climb the [machine] learning curve, and I don't want to be in the position of censoring anybody. However, this is starting to get out of hand. It is one thing to\r\n post code snippets, but another to post full solutions. The purpose of a recruiting competition is to give participants a chance to demonstrate their skills to a potential employer. Posting full solution code undermines this for everybody, throwing the originality\r\n of everyone's work in doubt.
", "rawMarkdown": "", "votes": null }, { "id": "11910", "postDate": "06/26/2012 17:54:55", "content": "Anyway, the algorithm is written in the start of this post, and it is is based on pageRank/edgeRank. Anyone can implement it now. The users who downloaded the file will have to re-implement it by themselves anyway! And as aaron stated, 0.71130 won't be enough\r\n to make it to top 10...
", "rawMarkdown": "", "votes": null }, { "id": "11911", "postDate": "06/26/2012 18:01:25", "content": "Den's case is much different from Aaron since Den has NO ranking on this competition after posting the code! The purpose of doing that is kind of suspicious. On the other hand, Aaron is a very helpful person as we can see. BTW, Aaron's result could be also\r\n getton after obtaining stats of inbound and outbound links based on Bayes' theorem.
", "rawMarkdown": "", "votes": null }, { "id": "11912", "postDate": "06/26/2012 18:01:53", "content": "Yes, we could have deleted the entire post, but you can't put an idea back in the bottle (especially when its based on such a well known algo). We're not saying you can't use an approach based on PageRank/EdgeRank, but that you shouldn't just be recycling\r\n someone else's code to try to slip through an interview screen.
", "rawMarkdown": "", "votes": null }, { "id": "11913", "postDate": "06/26/2012 18:10:25", "content": "There is an issue. People who've seen the code will rewrite it, combine it with their own approaches and get better score than equally skilled people who haven't seen the code. That's pretty unfair.
", "rawMarkdown": "", "votes": null }, { "id": "11915", "postDate": "06/26/2012 18:36:39", "content": "a) Just google for the code
\r\nb) The code will *not* put you into final top 10
", "rawMarkdown": "", "votes": null }, { "id": "11916", "postDate": "06/26/2012 18:44:02", "content": "I just received this email:
\r\n"You are receiving this email because you downloaded the files that were posted to competition thread " 0.711 is the new 0" before they were removed by the competition admins. There will be no backlash for having\r\n downloaded the files, but we respectfully ask that you inform us of any submissions you have already made based on this code so that we can remove them from the leaderboard. If you have any questions about why the files were removed, please feel free to contact\r\n us to discuss the situation."
\r\nSo how does this work? I never ran that actual code, but I did look at it and integrated some of the ideas into my solution. Sorry, but I can't "forget" what I saw. Does that mean I can't submit any solutions based on this idea anymore? This\r\n makes no sense.
\r\nI think Kaggle put itself in a difficult situation. How can you possibly know that everybody is honest and tells you exactly which submissions they made based on that code?
\r\nMy suggestion would be to put the code back on the forum for everyone to see, and clarify the rules as of what can be posted on the forums from now on. I'm sure the guys at Facebook can figure out in an interview, if someone has no idea about\r\n the code they submitted.
\r\nBy the way, this is a quote from your own rules:
\r\n"privately sharing code or data is not permitted outside of teams (sharing data or code is permissible if made available to all players, such as on the forums)"
", "rawMarkdown": "", "votes": null }, { "id": "11917", "postDate": "06/26/2012 18:46:01", "content": "I did receive the same email and replied basically the same thing.
\r\nMost probably, the code will be up again soon, it is the most sensible solution.
", "rawMarkdown": "", "votes": null }, { "id": "11919", "postDate": "06/26/2012 18:57:14", "content": "I was one of the individuals who downloaded the code. As others have noted, the algorithm is described in the first post and it is a variant of the PageRank algorithm. So, if you didn't see the code, look up PageRank on WikiPedia and then look at Den Souleo's\r\n description above and the subsequent clarifications.
\r\nDid I benefit from Den Souleo's post?
\r\nYes. PageRank/EdgeRank was on the list of algorithms I planned on on investigating. Den's post moved its implementation up to the top of the list.
\r\nDid I use Den Souleo's code?
\r\nNot directly. I read his code to clarify his algorithm description. I also implemented a version of his algorithm in Java.
\r\nWould I have implemented the algorithm exactly the same way without seeing his post?
\r\nNo. But again, the algorithm is not particularly original. I would have made different initial implementation decisions. Also, I view his description as a good start but hardly an optimized implementation.
\r\nWould I have initially implemented Den's algorithm the same way without seeing his code?
\r\nMaybe. However, his original description did not match his code. Someone who now reads the thread can reconstruct the algorithm based on subsequent clarifications.
\r\n\r\n
I recognize that the Kaggle team is trying to strike a difficult balance here but I would suggest a slightly different tack. There are plenty of software similarity tools out there (MOSS is especially good). Let everyone know (or not) that you plan on\r\n running competitors' code thru such tools. Update the terms of the competition, if need be, to indicate that unattributed third-party code is grounds for dismissal.
\r\nAs the prize for this competition is a job interview, a winner should be prepared to discuss all aspects of his/her model implementation. I would hope that Facebook is most interested in one's thought process, how one worked through implementation details\r\n and iterated through potential model tweaks/blends. It is good form to document/footnote your algorithm selection. All this forms your "story". If your story is, "uhhh, I cut/paste some code from the competition forum" then you won't get very far in the interview\r\n (assuming you magically won the competition with someone else's code).
", "rawMarkdown": "", "votes": null }, { "id": "11921", "postDate": "06/26/2012 19:10:53", "content": "I agree with Dave. Of course you could just google it. If you know the competition name, and the exact score it would achieve you can reach the code.\r\nAnyway, put back the code. It's more fair. It won't make to top 10! Just enforce this "no complete code sharing" from now on!
", "rawMarkdown": "", "votes": null }, { "id": "11922", "postDate": "06/26/2012 19:11:20", "content": "The files have been republished. Many of you made the strong point that some have already seen the files so the most sensible solution is to allow everyone access to the same information. As Dave Klein points out, if the prize is a job interview, it won't\r\n be much help if you just c/p someone else's work.
\r\n\r\n
", "rawMarkdown": "", "votes": null }, { "id": "11923", "postDate": "06/26/2012 19:14:00", "content": "
[quote=apollobp;11916]
\r\nBy the way, this is a quote from your own rules:
\r\n"privately sharing code or data is not permitted outside of teams (sharing data or code is permissible if made available to all players, such as on the forums)"
\r\n[/quote]
\r\n\r\n
Touche. Usually, since this is a competition, there is an incentive not to share your entire solution.
", "rawMarkdown": "", "votes": null }, { "id": "11924", "postDate": "06/26/2012 19:19:23", "content": "[quote=Glider;11923]
\r\nTouche. Usually, since this is a competition, there is an incentive not to share your entire solution.
\r\n[/quote]
\r\nActually, when I saw Den's posting, my initial thought was he may even work for Facebook/Kaggle and "seed" a better solution into the pool of candidates, maybe because they detected that the overall progress had stalled. At least for genetic algorithms that's\r\n a common technique to kickstart the search and get better end results :-)
", "rawMarkdown": "", "votes": null }, { "id": "11925", "postDate": "06/26/2012 19:20:44", "content": "How about using the submissions before the post as an additional adjustment criterion?
", "rawMarkdown": "", "votes": null }, { "id": "11926", "postDate": "06/26/2012 19:26:29", "content": "[quote=apollobp;11924]
\r\n[quote=Glider;11923]
\r\nTouche. Usually, since this is a competition, there is an incentive not to share your entire solution.
\r\n[/quote]
\r\nActually, when I saw Den's posting, my initial thought was he may even work for Facebook/Kaggle and "seed" a better solution into the pool of candidates, maybe because they detected that the overall progress had stalled. At least for genetic algorithms that's\r\n a common technique to kickstart the search and get better end results :-)
\r\n[/quote]
\r\nhmm, didn't know you thought we were that devious.
", "rawMarkdown": "", "votes": null }, { "id": "11927", "postDate": "06/26/2012 19:40:32", "content": "The overall progress somehow stalled because the dataset is simple. If we had more info about the nodes, like gender, sex, etc, it would be very different.
\r\nAnd about the seed history, i believe he is just someone who wanted to drop the competition and left his progress behind...\r\n
[quote=Leustagos;11927]
\r\nAnd about the seed history, i believe he is just someone who wanted to drop the competition and left his progress behind...
\r\n[/quote]
\r\nThe Kaggle Leaderboard Conspiracy (2012).
\r\nEDIT: Now the user Den Souleo does not exist. The plot just got thicker.
\r\nSource: https://www.kaggle.com/users/48103/den-souleo
Hi all,
\r\nI haven't posted on the forum to date because I considered doing so would violate the non-collaboration rule.
\r\nFWIW, there is more to the "Den" story than appears here. (It may have been easier to connect the dots from here, in a radically different time zone.)
\r\nI've advised Kaggle admin last night (Melbourne time) that I consider the competition compromised from the time of the "Den" posting. My final submission was the same file I posted Sunday, simply so I could attach my code and README archive (in the state\r\n it happened to be in).
\r\nMy laptop's two cores can now cool down. :)
\r\nKudos to Kaggle for surfing this wave with agility and maturity. You don't get to push the cutting edge without losing some blood. And before anyone here gets too heated about the pandemonium, remember that the people running and judging this comp are at\r\n least as intelligent as you are. They're no fools. Go with the flow.
\r\nCheers all,
\r\nJohn Costella
\r\nJune 27, 2012
John,
\r\nAs a competitor, I wouldn't "consider this competition compromised" at this point.\r\n
\r\nRight now, everyone has access to the same information. With everything considered, this is probably the best way forward AFTER the "Den" past was made and I would still consider this competition fair. From a certain point view, the current situation is\r\n not so different from the hypothetical scenario where the "Den solution" was posted as one of the benchmarks by Kaggle.
\r\nWhile the "Den" post does leave a bad taste in my (and probably many others') mouth, let's not go to the extreme and declare this entire competition "compromised".\r\n
\r\nAs you said, "remember that the people running and judging this comp are at least as intelligent as you are". So let them judge.
\r\nOwen
", "rawMarkdown": "", "votes": null }, { "id": "11938", "postDate": "06/27/2012 01:23:25", "content": "Ima noob, and I have joined Kaggle to try to gain some practical machine learning experience. I find the discussion threads to be quite informative and an integral part of the education process. I am quite perturbed by any attempt to censor the discussion\r\n by Kaggle (of course excluding trolling, spiteful, and bullying comments).
\r\nThis whole non-collaboration clause in this particular contest is silly in my opinion. It reminds me of high school. The real world is an open-book, open-ended, and collaborative exam. I mean the whole idea is for the data scientist to be part of a team,\r\n and not work in isolation.
\r\nThe key is when you use another person's work that you give proper attribution and obey the terms of use. Of course FB and Kaggle can set up this particular contest any which way they want. However, I believe that this particular contest is setting a bad\r\n precedent in terms of the open learning environment that Kaggle is trying to promote on this site.\r\n
", "rawMarkdown": "", "votes": null }, { "id": "11963", "postDate": "06/27/2012 21:10:04", "content": "Although I first agreed that the path taken (making Den's code available to all) is the best option available, I'm not sure about it anymore. It's first Kaggle competition, and I don't know if that's the norm - but it's not the first fishy thing that happened.\r\n It was preceeded by users having multiple accounts. It seems to me that ending the competition early might have been better.
\r\nNow, Den's code does give certain advantage to people like me who came up with a very different approach. I can merge Den's approach with mine to get a better solution (and indeed I went up 3 spots), while users who used a similar approach to Den's might\r\n not find this a simple task.
\r\nMoreover, Den's code is far from an initial benchmark - it's non trivial (I've tried igraph's implementation of the "standard" pagerank early on, and wasn't very valuable), and has many parameters you can play with (number of cycles/how many points each\r\n nodes keeps/using non uniform weights).
\r\nTLDR: meh, I feel worse today than a week ago.
", "rawMarkdown": "", "votes": null }, { "id": "11973", "postDate": "06/28/2012 00:07:22", "content": "Recruiting model is kind of new to kaggle. Recruiting competitions seem to be different from cash reward ones. Most people like money, but many may not be interested in an job offer or interview. One who doesn't want a new job more likely pose some inappropriate\r\n information.
\r\nMy suggestion is to set a cash reward to recruiting competitions as well. More people should be serious.\r\n
", "rawMarkdown": "", "votes": null }, { "id": "12001", "postDate": "06/28/2012 18:43:15", "content": "I think you can be serious without being antisocial. Linux is serious, for instance.
\r\nThis whole situation is very interesting to me as a seeming demonstration that while systems that rely on everyone being selfless don't seem to work, neither can you necessarily rely on everyone being selfish.
", "rawMarkdown": "", "votes": null }, { "id": "12003", "postDate": "06/28/2012 20:00:20", "content": "I am new here so probably my opinion doesn't matter. But the way I see it the problem is with the particular competition. It is a lot like high-school exams as somebody already mentioned in the thread. Also the organizing parties try to get too much control\r\n over something that is not possible to get control of.
\r\nIf on the other hand this was research-like; i.e given an interesting real-life problem, which if solved it will help society, the best solutions (as decided by fellows and peer-reviews) win XYZ, things would be very different.
\r\nAny contribution towards a better solution would be seen as positive no matter what.
\r\nIn this case the contribution leaves most of us with negatives feelings.
", "rawMarkdown": "", "votes": null }, { "id": "12004", "postDate": "06/28/2012 20:47:50", "content": "[quote]This whole non-collaboration clause in this particular contest is silly in my opinion.[/quote]
\r\nI believe the intent is to prevent people from teaming up behind the scenes - getting more submissions while coordinating efforts - then merging their teams. Other than that - and this contest (there was another if I recall as well) - generally collaboration\r\n seems to be encouraged.
", "rawMarkdown": "", "votes": null }, { "id": "12009", "postDate": "06/29/2012 01:18:22", "content": "In this contest it is not possible to merge teams. What happened in my opinion is that, to get around 2 submissions per day people created fake profiles, got discovered and banned.\r\n
\r\nAfter that they problably got very angry...
\r\n2 submissions per day are enough, but kaggle should let non used submissions acumulate and be used in the next days. Maybe not accumulating indefinitly, but to a number, lets say, 5x the daily amount. Some people can only work on weekends...
[quote=Leustagos;12009]
\r\n2 submissions per day are enough, but kaggle should let non used submissions acumulate and be used in the next days. Maybe not accumulating indefinitly, but to a number, lets say, 5x the daily amount. Some people can only work on weekends...
\r\n[/quote]
\r\nAmen to that! I found myself submitting shi**y solutions just so I won't waste my daily allowance.
", "rawMarkdown": "", "votes": null }, { "id": "12021", "postDate": "06/29/2012 15:36:41", "content": "I could not agree more. I think it would be a great change if submissions would store up over the week, especially in competitions where setting up meaningful cross-validation is a challenge.
", "rawMarkdown": "", "votes": null }, { "id": "12040", "postDate": "06/30/2012 12:02:19", "content": "Totally agree,:). The submissions should be accumulated by days.
", "rawMarkdown": "", "votes": null }, { "id": "12046", "postDate": "06/30/2012 17:10:48", "content": "Regarding collaboration, the rules state that "the foregoing shall not apply to any public communications, such as forum participation or blog posts."
\r\nFor any competition in general, unless Kaggle explicitly states otherwise, I assume collaboration in forums is always allowed.
", "rawMarkdown": "", "votes": null }, { "id": "12212", "postDate": "07/09/2012 02:05:51", "content": "I support Leustagos! Many cannot do kaggle everyday and a cache for unused budgets of submission should be considered. Suppose a students work everyday but can only work on this at weekends. Two choices for him, first is to only submit at weekends or submit\r\n some careless work during the weekday in order not to waste...
\r\nMaybe we have have a cash contest, given the leader bord score and predict whether the work he submit that day is carefully designed or just want to take the budget. Further we can predict he can take which day to work at Kaggle out of the whole week.
\r\nActually, If it is possible, can we have a voting for such policy publicly? Everyday counts. If we can have such policy, why not have it as soon as possible.
", "rawMarkdown": "", "votes": null }, { "id": "12225", "postDate": "07/09/2012 13:49:45", "content": "Sorry for the stupid question. I'm new here and I'm still trying to learn more about kaggle & data mining.\r\n
\r\nCan someone please enlighten me about the benefit of submitting bad entries in order not to waste the submission allowance?
", "rawMarkdown": "", "votes": null }, { "id": "12227", "postDate": "07/09/2012 14:49:55", "content": "@yuenking I guess you're asking that because of a prior comment I've made. There's no point of submitting bad solutions, but sometimes I've submitted\r\n a poorly thought of solution, that if I could "bank" submission, I would have avoided.
", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 11884, "author_name": "strongai", "author_url": "", "post_date": "06/26/2012 04:16:04", "content": "It sounds a lot like you just reinvented PageRank/EdgeRank.
", "votes": null, "replies": [] }, { "id": 11885, "author_name": "", "author_url": "", "post_date": "06/26/2012 04:34:57", "content": "Never heard of edgerank.. But yeah I guess it looks somewhat like pagerank. But needs quite a lot if tweaking. At the current stage a backward connection with a lot of "friends" will probable get better score than one with less friend but already connected\r\n to the "user's immediate network". Also discovering lost single forward connections may need to get a boost when the user has has high "follower" index or high peer index. Anyway. Lots of ways to tweak it and play around. By the way to compute predictions\r\n for the "test" takes about 15 mins on a 10 node cluster. So anybody trying it be patient..", "votes": null, "replies": [] }, { "id": 11886, "author_name": "quidity", "author_url": "", "post_date": "06/26/2012 05:02:20", "content": "Looking at the code, it seems that on the first step you allow backward-connected nodes to retain credit, but on deeper steps you do not (contrast scorenetwork with propagatescore).
", "votes": null, "replies": [] }, { "id": 11887, "author_name": "", "author_url": "", "post_date": "06/26/2012 05:25:06", "content": "Yes. That's on purpose. We are interested in making forward connections only with the direct backward connections. 2nd line backwards (or forward->backward) are too of long shot to get connected to them.. (And if the are of interest they shall get some\r\n credit from forward connections.I am much more likely to follow somebody the people that follow me or that I follow, follow. Than to follow somebody who follows the one who follows me..). But they play critical role for transferring the credit to their forwards.\r\n I think I understand your concern of under-crediting a lost "friend". It's something that needs tweaking. Any suggestions?", "votes": null, "replies": [] }, { "id": 11888, "author_name": "quidity", "author_url": "", "post_date": "06/26/2012 05:50:05", "content": "The approach makes sense, I was just checking if the description or the code was what you meant.
\r\nHere, at least, it's going to take 20 days to actually run that, so no 0.711 for me!
", "votes": null, "replies": [] }, { "id": 11891, "author_name": "seba45683", "author_url": "", "post_date": "06/26/2012 12:26:56", "content": "My first attemp is similar to this method, using a modified random walk with restart from each node that need to be predict following nodes.
\r\nThat is, assume that q is node to be predicted, we start random walk with restart with probability 0.15 to return to q, and 0.85 to fly to other nodes.
\r\nAny other nodes except q will receive a random walk score that indicates the probability of reaching this node from q in 20 iteration.
\r\nThis is similar to : instead of using 0.5 to keep credit, I use 0.15, and the repeated times is set to 20.
\r\nBut I only get map@10 around 0.004 and then I give up this method totally.
\r\nVery curious why you achive at 0.71, will try and see:)
", "votes": null, "replies": [] }, { "id": 11901, "author_name": "wlattner", "author_url": "", "post_date": "06/26/2012 17:00:01", "content": "If you are using python on a linux/mac machine with multiple cores and plenty of ram, this method is very easy to parallelize. See the python documentation for the multiprocessing module: http://docs.python.org/library/multiprocessing.html
\r\nThe attached code runs considerbly faster than 20 hours on 4 cores, also note additional functions added to utilities.py
", "votes": null, "replies": [] }, { "id": 11902, "author_name": "glider", "author_url": "", "post_date": "06/26/2012 17:18:48", "content": "We have removed the code posted to this thread by Den Souleo and others. We encourage contestants to use the competition forums to discuss and learn from each other, but posting full solution code has a direct impact on the integrity of the recruiting competition. \r\n If you receive an interview with Facebook, you will be asked to present code that only reflects your own work. Using code that was posted by another user will not help you, and you should be aware of this before selecting such work as your final submission.
", "votes": null, "replies": [] }, { "id": 11903, "author_name": "rohits", "author_url": "", "post_date": "06/26/2012 17:27:06", "content": "You removed the code after 16 hours? Is that fair towards us who did not have a look at the code?
", "votes": null, "replies": [] }, { "id": 11904, "author_name": "ajschumacher", "author_url": "", "post_date": "06/26/2012 17:30:22", "content": "With respect, I've had equivalent code up on the forum for nineteen days already. At the time I posted it, this code yielded results not dissimilar in ranking from the current offers in this thread. The result, I believe, was to help kaggle users learn and\r\n advance more quickly, in effect raising the bar. Already with a score of 0.711, this code will scarcely get you into the top ten, and certainly will not by the end of the competition. I don't believe posting code here violates any agreement made by kaggle\r\n users. It is already the case that using code posted by other users will not help you to win. On what grounds can this selective censoring be truly justified?
", "votes": null, "replies": [] }, { "id": 11905, "author_name": "rohits", "author_url": "", "post_date": "06/26/2012 17:37:37", "content": "", "votes": null, "replies": [] }, { "id": 11906, "author_name": "glider", "author_url": "", "post_date": "06/26/2012 17:37:41", "content": "[quote=SivaP;11903]
\r\nYou removed the code after 16 hours? Is that fair towards us who did not have a look at the code?
\r\n[/quote]
\r\nWe are discussing the situation with users who downloaded the code during the 16 hour time window
\r\n", "votes": null, "replies": [] }, { "id": 11908, "author_name": "glider", "author_url": "", "post_date": "06/26/2012 17:43:37", "content": "
[quote=Aaron Schumacher;11904]
\r\nWith respect, I've had equivalent code up on the forum for nineteen days already. At the time I posted it, this code yielded results not dissimilar in ranking from the current offers in this thread. The result, I believe, was to help kaggle users learn and\r\n advance more quickly, in effect raising the bar. Already with a score of 0.711, this code will scarcely get you into the top ten, and certainly will not by the end of the competition. I don't believe posting code here violates any agreement made by kaggle\r\n users. It is already the case that using code posted by other users will not help you to win. On what grounds can this selective censoring be truly justified?
\r\n[/quote]
\r\n\r\n
Hi Aaron. I agree with you that the purpose of the forums should be to help Kaggle users climb the [machine] learning curve, and I don't want to be in the position of censoring anybody. However, this is starting to get out of hand. It is one thing to\r\n post code snippets, but another to post full solutions. The purpose of a recruiting competition is to give participants a chance to demonstrate their skills to a potential employer. Posting full solution code undermines this for everybody, throwing the originality\r\n of everyone's work in doubt.
", "votes": null, "replies": [] }, { "id": 11910, "author_name": "leustagos", "author_url": "", "post_date": "06/26/2012 17:54:55", "content": "Anyway, the algorithm is written in the start of this post, and it is is based on pageRank/edgeRank. Anyone can implement it now. The users who downloaded the file will have to re-implement it by themselves anyway! And as aaron stated, 0.71130 won't be enough\r\n to make it to top 10...
", "votes": null, "replies": [] }, { "id": 11911, "author_name": "songgc", "author_url": "", "post_date": "06/26/2012 18:01:25", "content": "Den's case is much different from Aaron since Den has NO ranking on this competition after posting the code! The purpose of doing that is kind of suspicious. On the other hand, Aaron is a very helpful person as we can see. BTW, Aaron's result could be also\r\n getton after obtaining stats of inbound and outbound links based on Bayes' theorem.
", "votes": null, "replies": [] }, { "id": 11912, "author_name": "glider", "author_url": "", "post_date": "06/26/2012 18:01:53", "content": "Yes, we could have deleted the entire post, but you can't put an idea back in the bottle (especially when its based on such a well known algo). We're not saying you can't use an approach based on PageRank/EdgeRank, but that you shouldn't just be recycling\r\n someone else's code to try to slip through an interview screen.
", "votes": null, "replies": [] }, { "id": 11913, "author_name": "yarreg", "author_url": "", "post_date": "06/26/2012 18:10:25", "content": "There is an issue. People who've seen the code will rewrite it, combine it with their own approaches and get better score than equally skilled people who haven't seen the code. That's pretty unfair.
", "votes": null, "replies": [] }, { "id": 11915, "author_name": "nocencorship", "author_url": "", "post_date": "06/26/2012 18:36:39", "content": "a) Just google for the code
\r\nb) The code will *not* put you into final top 10
", "votes": null, "replies": [] }, { "id": 11916, "author_name": "apollobp", "author_url": "", "post_date": "06/26/2012 18:44:02", "content": "I just received this email:
\r\n"You are receiving this email because you downloaded the files that were posted to competition thread " 0.711 is the new 0" before they were removed by the competition admins. There will be no backlash for having\r\n downloaded the files, but we respectfully ask that you inform us of any submissions you have already made based on this code so that we can remove them from the leaderboard. If you have any questions about why the files were removed, please feel free to contact\r\n us to discuss the situation."
\r\nSo how does this work? I never ran that actual code, but I did look at it and integrated some of the ideas into my solution. Sorry, but I can't "forget" what I saw. Does that mean I can't submit any solutions based on this idea anymore? This\r\n makes no sense.
\r\nI think Kaggle put itself in a difficult situation. How can you possibly know that everybody is honest and tells you exactly which submissions they made based on that code?
\r\nMy suggestion would be to put the code back on the forum for everyone to see, and clarify the rules as of what can be posted on the forums from now on. I'm sure the guys at Facebook can figure out in an interview, if someone has no idea about\r\n the code they submitted.
\r\nBy the way, this is a quote from your own rules:
\r\n"privately sharing code or data is not permitted outside of teams (sharing data or code is permissible if made available to all players, such as on the forums)"
", "votes": null, "replies": [] }, { "id": 11917, "author_name": "miguel2", "author_url": "", "post_date": "06/26/2012 18:46:01", "content": "I did receive the same email and replied basically the same thing.
\r\nMost probably, the code will be up again soon, it is the most sensible solution.
", "votes": null, "replies": [] }, { "id": 11919, "author_name": "drk123", "author_url": "", "post_date": "06/26/2012 18:57:14", "content": "I was one of the individuals who downloaded the code. As others have noted, the algorithm is described in the first post and it is a variant of the PageRank algorithm. So, if you didn't see the code, look up PageRank on WikiPedia and then look at Den Souleo's\r\n description above and the subsequent clarifications.
\r\nDid I benefit from Den Souleo's post?
\r\nYes. PageRank/EdgeRank was on the list of algorithms I planned on on investigating. Den's post moved its implementation up to the top of the list.
\r\nDid I use Den Souleo's code?
\r\nNot directly. I read his code to clarify his algorithm description. I also implemented a version of his algorithm in Java.
\r\nWould I have implemented the algorithm exactly the same way without seeing his post?
\r\nNo. But again, the algorithm is not particularly original. I would have made different initial implementation decisions. Also, I view his description as a good start but hardly an optimized implementation.
\r\nWould I have initially implemented Den's algorithm the same way without seeing his code?
\r\nMaybe. However, his original description did not match his code. Someone who now reads the thread can reconstruct the algorithm based on subsequent clarifications.
\r\n\r\n
I recognize that the Kaggle team is trying to strike a difficult balance here but I would suggest a slightly different tack. There are plenty of software similarity tools out there (MOSS is especially good). Let everyone know (or not) that you plan on\r\n running competitors' code thru such tools. Update the terms of the competition, if need be, to indicate that unattributed third-party code is grounds for dismissal.
\r\nAs the prize for this competition is a job interview, a winner should be prepared to discuss all aspects of his/her model implementation. I would hope that Facebook is most interested in one's thought process, how one worked through implementation details\r\n and iterated through potential model tweaks/blends. It is good form to document/footnote your algorithm selection. All this forms your "story". If your story is, "uhhh, I cut/paste some code from the competition forum" then you won't get very far in the interview\r\n (assuming you magically won the competition with someone else's code).
", "votes": null, "replies": [] }, { "id": 11921, "author_name": "leustagos", "author_url": "", "post_date": "06/26/2012 19:10:53", "content": "I agree with Dave. Of course you could just google it. If you know the competition name, and the exact score it would achieve you can reach the code.\r\nAnyway, put back the code. It's more fair. It won't make to top 10! Just enforce this "no complete code sharing" from now on!
", "votes": null, "replies": [] }, { "id": 11922, "author_name": "glider", "author_url": "", "post_date": "06/26/2012 19:11:20", "content": "The files have been republished. Many of you made the strong point that some have already seen the files so the most sensible solution is to allow everyone access to the same information. As Dave Klein points out, if the prize is a job interview, it won't\r\n be much help if you just c/p someone else's work.
\r\n\r\n
", "votes": null, "replies": [] }, { "id": 11923, "author_name": "glider", "author_url": "", "post_date": "06/26/2012 19:14:00", "content": "
[quote=apollobp;11916]
\r\nBy the way, this is a quote from your own rules:
\r\n"privately sharing code or data is not permitted outside of teams (sharing data or code is permissible if made available to all players, such as on the forums)"
\r\n[/quote]
\r\n\r\n
Touche. Usually, since this is a competition, there is an incentive not to share your entire solution.
", "votes": null, "replies": [] }, { "id": 11924, "author_name": "apollobp", "author_url": "", "post_date": "06/26/2012 19:19:23", "content": "[quote=Glider;11923]
\r\nTouche. Usually, since this is a competition, there is an incentive not to share your entire solution.
\r\n[/quote]
\r\nActually, when I saw Den's posting, my initial thought was he may even work for Facebook/Kaggle and "seed" a better solution into the pool of candidates, maybe because they detected that the overall progress had stalled. At least for genetic algorithms that's\r\n a common technique to kickstart the search and get better end results :-)
", "votes": null, "replies": [] }, { "id": 11925, "author_name": "songgc", "author_url": "", "post_date": "06/26/2012 19:20:44", "content": "How about using the submissions before the post as an additional adjustment criterion?
", "votes": null, "replies": [] }, { "id": 11926, "author_name": "glider", "author_url": "", "post_date": "06/26/2012 19:26:29", "content": "[quote=apollobp;11924]
\r\n[quote=Glider;11923]
\r\nTouche. Usually, since this is a competition, there is an incentive not to share your entire solution.
\r\n[/quote]
\r\nActually, when I saw Den's posting, my initial thought was he may even work for Facebook/Kaggle and "seed" a better solution into the pool of candidates, maybe because they detected that the overall progress had stalled. At least for genetic algorithms that's\r\n a common technique to kickstart the search and get better end results :-)
\r\n[/quote]
\r\nhmm, didn't know you thought we were that devious.
", "votes": null, "replies": [] }, { "id": 11927, "author_name": "leustagos", "author_url": "", "post_date": "06/26/2012 19:40:32", "content": "The overall progress somehow stalled because the dataset is simple. If we had more info about the nodes, like gender, sex, etc, it would be very different.
\r\nAnd about the seed history, i believe he is just someone who wanted to drop the competition and left his progress behind...\r\n
[quote=Leustagos;11927]
\r\nAnd about the seed history, i believe he is just someone who wanted to drop the competition and left his progress behind...
\r\n[/quote]
\r\nThe Kaggle Leaderboard Conspiracy (2012).
\r\nEDIT: Now the user Den Souleo does not exist. The plot just got thicker.
\r\nSource: https://www.kaggle.com/users/48103/den-souleo
Hi all,
\r\nI haven't posted on the forum to date because I considered doing so would violate the non-collaboration rule.
\r\nFWIW, there is more to the "Den" story than appears here. (It may have been easier to connect the dots from here, in a radically different time zone.)
\r\nI've advised Kaggle admin last night (Melbourne time) that I consider the competition compromised from the time of the "Den" posting. My final submission was the same file I posted Sunday, simply so I could attach my code and README archive (in the state\r\n it happened to be in).
\r\nMy laptop's two cores can now cool down. :)
\r\nKudos to Kaggle for surfing this wave with agility and maturity. You don't get to push the cutting edge without losing some blood. And before anyone here gets too heated about the pandemonium, remember that the people running and judging this comp are at\r\n least as intelligent as you are. They're no fools. Go with the flow.
\r\nCheers all,
\r\nJohn Costella
\r\nJune 27, 2012
John,
\r\nAs a competitor, I wouldn't "consider this competition compromised" at this point.\r\n
\r\nRight now, everyone has access to the same information. With everything considered, this is probably the best way forward AFTER the "Den" past was made and I would still consider this competition fair. From a certain point view, the current situation is\r\n not so different from the hypothetical scenario where the "Den solution" was posted as one of the benchmarks by Kaggle.
\r\nWhile the "Den" post does leave a bad taste in my (and probably many others') mouth, let's not go to the extreme and declare this entire competition "compromised".\r\n
\r\nAs you said, "remember that the people running and judging this comp are at least as intelligent as you are". So let them judge.
\r\nOwen
", "votes": null, "replies": [] }, { "id": 11938, "author_name": "uncadrew", "author_url": "", "post_date": "06/27/2012 01:23:25", "content": "Ima noob, and I have joined Kaggle to try to gain some practical machine learning experience. I find the discussion threads to be quite informative and an integral part of the education process. I am quite perturbed by any attempt to censor the discussion\r\n by Kaggle (of course excluding trolling, spiteful, and bullying comments).
\r\nThis whole non-collaboration clause in this particular contest is silly in my opinion. It reminds me of high school. The real world is an open-book, open-ended, and collaborative exam. I mean the whole idea is for the data scientist to be part of a team,\r\n and not work in isolation.
\r\nThe key is when you use another person's work that you give proper attribution and obey the terms of use. Of course FB and Kaggle can set up this particular contest any which way they want. However, I believe that this particular contest is setting a bad\r\n precedent in terms of the open learning environment that Kaggle is trying to promote on this site.\r\n
", "votes": null, "replies": [] }, { "id": 11963, "author_name": "roulinir", "author_url": "", "post_date": "06/27/2012 21:10:04", "content": "Although I first agreed that the path taken (making Den's code available to all) is the best option available, I'm not sure about it anymore. It's first Kaggle competition, and I don't know if that's the norm - but it's not the first fishy thing that happened.\r\n It was preceeded by users having multiple accounts. It seems to me that ending the competition early might have been better.
\r\nNow, Den's code does give certain advantage to people like me who came up with a very different approach. I can merge Den's approach with mine to get a better solution (and indeed I went up 3 spots), while users who used a similar approach to Den's might\r\n not find this a simple task.
\r\nMoreover, Den's code is far from an initial benchmark - it's non trivial (I've tried igraph's implementation of the "standard" pagerank early on, and wasn't very valuable), and has many parameters you can play with (number of cycles/how many points each\r\n nodes keeps/using non uniform weights).
\r\nTLDR: meh, I feel worse today than a week ago.
", "votes": null, "replies": [] }, { "id": 11973, "author_name": "songgc", "author_url": "", "post_date": "06/28/2012 00:07:22", "content": "Recruiting model is kind of new to kaggle. Recruiting competitions seem to be different from cash reward ones. Most people like money, but many may not be interested in an job offer or interview. One who doesn't want a new job more likely pose some inappropriate\r\n information.
\r\nMy suggestion is to set a cash reward to recruiting competitions as well. More people should be serious.\r\n
", "votes": null, "replies": [] }, { "id": 12001, "author_name": "ajschumacher", "author_url": "", "post_date": "06/28/2012 18:43:15", "content": "I think you can be serious without being antisocial. Linux is serious, for instance.
\r\nThis whole situation is very interesting to me as a seeming demonstration that while systems that rely on everyone being selfless don't seem to work, neither can you necessarily rely on everyone being selfish.
", "votes": null, "replies": [] }, { "id": 12003, "author_name": "nikoskavadias", "author_url": "", "post_date": "06/28/2012 20:00:20", "content": "I am new here so probably my opinion doesn't matter. But the way I see it the problem is with the particular competition. It is a lot like high-school exams as somebody already mentioned in the thread. Also the organizing parties try to get too much control\r\n over something that is not possible to get control of.
\r\nIf on the other hand this was research-like; i.e given an interesting real-life problem, which if solved it will help society, the best solutions (as decided by fellows and peer-reviews) win XYZ, things would be very different.
\r\nAny contribution towards a better solution would be seen as positive no matter what.
\r\nIn this case the contribution leaves most of us with negatives feelings.
", "votes": null, "replies": [] }, { "id": 12004, "author_name": "chrisraimondi", "author_url": "", "post_date": "06/28/2012 20:47:50", "content": "[quote]This whole non-collaboration clause in this particular contest is silly in my opinion.[/quote]
\r\nI believe the intent is to prevent people from teaming up behind the scenes - getting more submissions while coordinating efforts - then merging their teams. Other than that - and this contest (there was another if I recall as well) - generally collaboration\r\n seems to be encouraged.
", "votes": null, "replies": [] }, { "id": 12009, "author_name": "leustagos", "author_url": "", "post_date": "06/29/2012 01:18:22", "content": "In this contest it is not possible to merge teams. What happened in my opinion is that, to get around 2 submissions per day people created fake profiles, got discovered and banned.\r\n
\r\nAfter that they problably got very angry...
\r\n2 submissions per day are enough, but kaggle should let non used submissions acumulate and be used in the next days. Maybe not accumulating indefinitly, but to a number, lets say, 5x the daily amount. Some people can only work on weekends...
[quote=Leustagos;12009]
\r\n2 submissions per day are enough, but kaggle should let non used submissions acumulate and be used in the next days. Maybe not accumulating indefinitly, but to a number, lets say, 5x the daily amount. Some people can only work on weekends...
\r\n[/quote]
\r\nAmen to that! I found myself submitting shi**y solutions just so I won't waste my daily allowance.
", "votes": null, "replies": [] }, { "id": 12021, "author_name": "shumarci", "author_url": "", "post_date": "06/29/2012 15:36:41", "content": "I could not agree more. I think it would be a great change if submissions would store up over the week, especially in competitions where setting up meaningful cross-validation is a challenge.
", "votes": null, "replies": [] }, { "id": 12040, "author_name": "nhanitvn", "author_url": "", "post_date": "06/30/2012 12:02:19", "content": "Totally agree,:). The submissions should be accumulated by days.
", "votes": null, "replies": [] }, { "id": 12046, "author_name": "solorzano", "author_url": "", "post_date": "06/30/2012 17:10:48", "content": "Regarding collaboration, the rules state that "the foregoing shall not apply to any public communications, such as forum participation or blog posts."
\r\nFor any competition in general, unless Kaggle explicitly states otherwise, I assume collaboration in forums is always allowed.
", "votes": null, "replies": [] }, { "id": 12212, "author_name": "weilinear", "author_url": "", "post_date": "07/09/2012 02:05:51", "content": "I support Leustagos! Many cannot do kaggle everyday and a cache for unused budgets of submission should be considered. Suppose a students work everyday but can only work on this at weekends. Two choices for him, first is to only submit at weekends or submit\r\n some careless work during the weekday in order not to waste...
\r\nMaybe we have have a cash contest, given the leader bord score and predict whether the work he submit that day is carefully designed or just want to take the budget. Further we can predict he can take which day to work at Kaggle out of the whole week.
\r\nActually, If it is possible, can we have a voting for such policy publicly? Everyday counts. If we can have such policy, why not have it as soon as possible.
", "votes": null, "replies": [] }, { "id": 12225, "author_name": "yuenking", "author_url": "", "post_date": "07/09/2012 13:49:45", "content": "Sorry for the stupid question. I'm new here and I'm still trying to learn more about kaggle & data mining.\r\n
\r\nCan someone please enlighten me about the benefit of submitting bad entries in order not to waste the submission allowance?
", "votes": null, "replies": [] }, { "id": 12227, "author_name": "roulinir", "author_url": "", "post_date": "07/09/2012 14:49:55", "content": "@yuenking I guess you're asking that because of a prior comment I've made. There's no point of submitting bad solutions, but sometimes I've submitted\r\n a poorly thought of solution, that if I could "bank" submission, I would have avoided.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "11883": "", "11884": "", "11885": "", "11886": "", "11887": "", "11888": "", "11891": "", "11901": "", "11902": "", "11903": "", "11904": "", "11905": "", "11906": "", "11908": "", "11910": "", "11911": "", "11912": "", "11913": "", "11915": "", "11916": "", "11917": "", "11919": "", "11921": "", "11922": "", "11923": "", "11924": "", "11925": "", "11926": "", "11927": "", "11934": "", "11936": "", "11937": "", "11938": "", "11963": "", "11973": "", "12001": "", "12003": "", "12004": "", "12009": "", "12011": "", "12021": "", "12040": "", "12046": "", "12212": "", "12225": "", "12227": "" }, "source": "meta" }