I was far from the best, but in case you are interested in what I did to get an 8th place finish attached is my approach.
\r\nInteresting fact - Runtime was about 5 hrs on an Amazon EC2 \r\nm2.4xlarge machine (8 cpu, 64GB). Total cost for a submission: $0.70 on a spot instance. Of course, then there were all the experiments $$
\r\n\r\n
I would love to hear from others!
", "messages": [ { "id": "12290", "postDate": "07/11/2012 00:45:49", "content": "I was far from the best, but in case you are interested in what I did to get an 8th place finish attached is my approach.
\r\nInteresting fact - Runtime was about 5 hrs on an Amazon EC2 \r\nm2.4xlarge machine (8 cpu, 64GB). Total cost for a submission: $0.70 on a spot instance. Of course, then there were all the experiments $$
\r\n\r\n
I would love to hear from others!
", "rawMarkdown": "", "votes": null }, { "id": "12294", "postDate": "07/11/2012 02:15:52", "content": "Belated determination that only one edge of a matched set of output input edges was deleted from the graph since no test set nodes had 0 inputs and 0 outputs in training data. The training set was created by deleting edges except for the above exclusions.\r\n The public test sett was created by selecting edge/node pair until the number of nodes was 1/4 the number of deleted edges, so public test nodes were included in the private test set?
\r\nFor a given test node, the only possible predicted edges were outputs to nodes that were inputs to the test node? Guess not, while about 9 percent of training nodes had no inputs, about 2 percent of public test nodes had no inputs?
I believe the private test set included nodes from the public test set because non commutative edges were over 1 million in the public test set out of only about 2.9 million total in the training set. If the private test set was disjoint with identical\r\n probability distribution then the combined test sets would have over 4 million non commutative edges.
", "rawMarkdown": "", "votes": null }, { "id": "12297", "postDate": "07/11/2012 02:49:26", "content": "Hi, Brady
\r\nThanks.
", "rawMarkdown": "", "votes": null }, { "id": "12301", "postDate": "07/11/2012 04:22:44", "content": "Cannot agree more. It is extremely important to establish the true training, cross-validation set.
\r\nIt is a little surprised that the advantage of supervised learning methods over unsupervised ones for this problem seems slight. A sub-graph-based pagerank-like one only needs 2 hours for total computation while the performance loss would be around 2%.
\r\nI don't know if anybody tried MLR (http://en.wikipedia.org/wiki/Learning_to_rank)
", "rawMarkdown": "", "votes": null }, { "id": "12302", "postDate": "07/11/2012 07:05:08", "content": "I'm a little surprised about the mentioned running times. Those mentioned 5-hour and 2-hour running times include prediction time only, or also the time required to generate crossvalidation sets, train supervised algorithms, etc?
\r\nIn contrast to this, my solution took only 10 min to compute on a 2.5 GHz, 4 GB RAM MacBook Pro, without resorting to any GPU tricks nor similar enhancements
\r\nIt is just a weighted sum of random walk probabilities on the graph. The algorithm requires only 4 parameters, which I selected manually by checking on a small validation set. A more careful parameter selection would have yielded better results, but unfortunately\r\n I wasted too much time playing around with more complex solutions and supervised learning, so I did not have the time to set up a proper parameter search. I might try to do this later, just to check the maximum achievable score using this idea.
\r\nI'd like to hear from the admins whether it is ok to disclose the full solution here. I would like to post mine and read about others. This was both funny and educating!
", "rawMarkdown": "", "votes": null }, { "id": "12304", "postDate": "07/11/2012 11:42:08", "content": "Brady: I initially tried a supervised learning approach similar to yours, but didn't have much success. I suspect your secret sauce lies in the definition of your features :) I would be curious to see the actual code implementation, if you wouldn't mind\r\n posting it.
\r\nMiguel: I was curious about using random walks, but didn't have much time to explore. I would like to like see some more details on your "4 parameters", and how you did your implementation. Could you recommend some good references?
\r\nI spent a lot of time trying "Netflix style approaches", via randomized SVD/PCA, and also calculated localized weighted Page-ranks using breadth-first neighborhoods, but my implementation was always unsupervised, and never beat Dan Souleo's MAP@10 score\r\n :(
\r\nFor what it's worth, I found this competition really educational and fun! In particular, I learned how to properly use the python multiprocessing library (the secret is to properly set the chunk-size!), and I learned a lot about SVDs.
\r\nThank you Kaggle and Facebook for setting this up!
\r\n-V-
", "rawMarkdown": "", "votes": null }, { "id": "12307", "postDate": "07/11/2012 13:08:21", "content": "I used an approach that was similar to Brady's. I also tried random walk, nearest neighbor, and EdgeRank based approaches but found that they didn't perform as well for me.
\r\nI also tried an MLR type approach by using a random forest to compare pairs of data points. It didn't work as well and I'm thinking it's because there are very few correct links compared to the ten guesses you're allowed per node. Most of the guesses are\r\n going to be wrong, so their ranking order doesn't really affect the score that much.
\r\nOne thing I did differently from Brady and Ben Hammer is I tried to use simple features for my random forest. I found that using features that are good predictors on their own actually reduces accuracy. I ended up using an MTRY of 7 and I think the reason\r\n Brady ended up with an MTRY of 2 was due to using strong predictors as features.
\r\nHere is a list of features I found useful:
\r\nExistence of a reverse link between nodes. (1=yes/0=no)\nCount of forward-forward links between nodes.\nCount of forward-reverse links between nodes.\nCount of forward-bidirectional links between nodes.\nCount of reverse-forward links between nodes.\nCount of reverse-reverse links between nodes.\nCount of reverse-bidirectional links between nodes.\nCount of bidirectional-forward links between nodes.\nCount of bidirectional-reverse links between nodes.\nCount of bidirectional-bidirectional links between nodes.\nCount of common neighbors.\nNumber of links ending at the node to be predicted / Number of links starting at the node to be predicted.\nNumber of links starting at the node to be ranked.\nNumber of links ending at the node to be ranked.\nNumber of links ending at the node to be ranked / Number of links starting at the node to be ranked.\nCount of common neighbors / Count of all neighbors.\nCount of paths with exactly three links between nodes / Count of paths with exactly three links from node to be predicted to any node.\nCount of forward-forward-forward links between nodes / Count of all length three paths between nodes.\nCount of forward-forward-reverse links between nodes / Count of all length three paths between nodes.\nCount of forward-reverse-forward links between nodes / Count of all length three paths between nodes.\nCount of forward-reverse-reverse links between nodes / Count of all length three paths between nodes.\nCount of reverse-forward-forward links between nodes / Count of all length three paths between nodes.\nCount of reverse-forward-reverse links between nodes / Count of all length three paths between nodes.\nCount of reverse-forward-forward links between nodes / Count of all length three paths between nodes.\nCount of reverse-forward-reverse links between nodes / Count of all length three paths between nodes.\nCount of reverse-reverse-forward links between nodes / Count of all length three paths between nodes.\nCount of reverse-reverse-reverse links between nodes / Count of all length three paths between nodes.\nAverage length of all unique paths from a node to its immediate successors.\nAverage length of all unique paths from a node to its immediate predecessors.\n[quote=Miguel;12302]
\r\nThose mentioned 5-hour and 2-hour running times include prediction time only, or also the time required to generate crossvalidation sets, train supervised algorithms, etc?
\r\n[/quote]
\r\nMy 5-hours was for everything, including generating the validation set (a few minutes), extracting features for both validation and test sets (~2 hours), training 1600 decision trees (~2 hours) and making predictions (~1 hour).
\r\nMiguel, your technique sounds simple and elegant. I do hope you share the details. Was your 10 minutes for everything, including a parameter search? \r\n
\r\nNot that it matters in a contest like this, but one advantage to using a supervised learning approach (instead of relying on exclusively on an edge rank / page rank solution with fixed parameters) is that it could be applied to a different graph with very\r\n few, in any, modifications.
", "rawMarkdown": "", "votes": null }, { "id": "12316", "postDate": "07/11/2012 16:02:32", "content": "[quote=Glen;12307]
\r\nOne thing I did differently from Brady and Ben Hammer is I tried to use simple features for my random forest. I found that using features that are good predictors on their own actually reduces accuracy. I ended up using an MTRY of 7 and I think the reason\r\n Brady ended up with an MTRY of 2 was due to using strong predictors as features.
\r\n[/quote]
\r\nGlen, thanks for sharing. Don't take this the worng way, but I like your features :) One of the issues I kept contemplating was how to properly handle a node that belongs to multiple networks. I can understand how most of my feature would have best predictive\r\n capability for a node belonging to a single network, which is clearly a bad assumption and one I would have liked to avoid. I wonder if your weak predictors prehaps did a better job on highly connected nodes which are more likely to belong to multiple networks. \r\n I think I can justify how using absolute counts of connectedness, such as what you did, would work better in such a case than something that was normalized to the local network, such as what I did.
\r\nWas your focus on weak predictors based on a particular concept, or more from imperical observation about their performance?
", "rawMarkdown": "", "votes": null }, { "id": "12317", "postDate": "07/11/2012 16:19:25", "content": "Brandy, how did you add fake missing edges to your local training set? In the document, you mention only the way real missing edges were sampled. I think the way we choose fake missing edges is important also because there are features that discriminate\r\n these two types of missing edges.
", "rawMarkdown": "", "votes": null }, { "id": "12319", "postDate": "07/11/2012 16:55:10", "content": "[quote=Brady Benware;12316]
\r\nI think I can justify how using absolute counts of connectedness, such as what you did, would work better in such a case than something that was normalized to the local network, such as what I did.
\r\nWas your focus on weak predictors based on a particular concept, or more from imperical observation about their performance?
\r\n[/quote]
\r\nActually I made mistake in my post. The counts of forward-forward, forward-reverse, etc. connections were normalized. Although I found that normalization only produced slightly better scores.
\r\nThe focus on weak predictors came from trying out strong predictors as features and having them hurt performance and giving strange parameter values for things like MTRY. The whole idea of random forests is to ensemble a bunch of weak predictors that have\r\n a high degree of variance in their predictions.
", "rawMarkdown": "", "votes": null }, { "id": "12321", "postDate": "07/11/2012 19:55:45", "content": "Glen/Brady: Did either of you try any reduction techniques, to filter out either the weaker nodes or the "Bieber" nodes?\r\n
\r\nAlso when training your classifier, how did you optimize for the sparseness of the adjacency matrix?\r\n
\r\nOne problem I had when training my classifier was that all the 0's (from the sparsity) were throwing off my predictions. How do you handle that problem?
", "rawMarkdown": "", "votes": null }, { "id": "12323", "postDate": "07/12/2012 00:29:01", "content": "I think it would be a great idea if all of us with GitHub accounts throw up our code to discuss this. Make it a standard-ish name like "kaggle-facebook-lepieuvre" for example. I'll get mine up soon, I'm on a roadtrip.
\r\nI only finished 52nd but I'm happy with that considering I did the whole thing in Perl and on my laptop. I need to polish my Python skills for the next big competition.
", "rawMarkdown": "", "votes": null }, { "id": "12334", "postDate": "07/12/2012 11:05:23", "content": "[quote=vgoklani;12321]
\r\nGlen/Brady: Did either of you try any reduction techniques, to filter out either the weaker nodes or the "Bieber" nodes?
\r\nAlso when training your classifier, how did you optimize for the sparseness of the adjacency matrix?
\r\nOne problem I had when training my classifier was that all the 0's (from the sparsity) were throwing off my predictions. How do you handle that problem?
\r\n[/quote]
\r\nI did not try filtering out any nodes.
\r\nMy training data consisted of all nodes one or two hops away from the node being predicted. Only around 5% of these nodes are actually connected and I found sampling the data to have 10% connections produced better results.
", "rawMarkdown": "", "votes": null }, { "id": "12336", "postDate": "07/12/2012 12:45:17", "content": "So what do we learn from here:
\r\n1) Russian schooling still rules, I am proud of my motherland.
\r\n2) There is no killing algorithm.
\r\nI am curious what the winners did, but
\r\neven obvious solution like RootedPageRank (augmented with special stochatic matrix that
\r\nallows to porpagte both direction taking into accout fiendliness of the nodes) which I tried, performed slightly worse than
\r\nthe naive 7130-aproach.
\r\n3) Personally, I doubt that in real production environment I'd make (or my boss allowed me) an effort to imporve from 0.71 to 0.72, would you?
\r\nI did not have time (I spent in total only 15 hours or so), to adjust all the parameters and prepare good train and validation data
\r\nbut it was kind of curios to see that no matter what parameters I chose or taking different schemes to deal with dangling links
\r\nreally did not matter in practice.
", "rawMarkdown": "", "votes": null }, { "id": "12338", "postDate": "07/12/2012 15:46:52", "content": "Hey Axel,
\r\nPersonally, I've become obsessed with preparing "good" train and validation data. I didn't even get to make a submission!\r\n
\r\nBut if you'd like to demonstrate more of that russian schooling, talk to me about preparing balanced data to train models on (maybe not a bad idea since the algorithm used didnt seem to matter much)
Here is a short overview of my approach. It doesn't differs very much from other supervise learning approaches described above but may be somebody will find something useful. I'll be glad to answer any questions. Main points:
\r\n", "rawMarkdown": "", "votes": null }, { "id": "12341", "postDate": "07/12/2012 16:09:10", "content": "[quote=Axel;12336]
\r\n3) Personally, I doubt that in real production environment I'd make (or my boss allowed me) an effort to imporve from 0.71 to 0.72, would you?
\r\n[/quote]
\r\nAs long as 1% improvement leads to e.g. 0.1% rise in income of multibillion dollar corporation I suppose it's really worth it to make such an effort. :) But may be this is not the case.
", "rawMarkdown": "", "votes": null }, { "id": "12343", "postDate": "07/12/2012 17:58:04", "content": "vgoklani: A survey on random walks that I found very interesting is "Link Prediction in Complex Networks: A Survey"
\r\n\r\n
Brady: The 10 minutes time was only for prediction. Validation time can vary a lot depending on the size of your validation set and how finely you sweep the parameters. In my case, this was done interactively just before the competition ended. I'd say it\r\n took about 50 min, including time for running all validations and time for me to change the parameters for the next run. I had no time to do a proper sweep, this was about one hour before the competition ended.
\r\n\r\n
Also, I don't agree with this:
\r\n\r\n
"one advantage to using a supervised learning approach (instead of relying on exclusively on an edge rank / page rank solution with fixed parameters) is that it could be applied to a different graph with very few, in any, modifications."
\r\n\r\n
From my perspective, I don't see any difference. If you were given a different graph, you'd retrain your supervised learning algorithm and hope it'd perform well. I'd do the same, repeating the crossvalidation procedure and selecting the weights that are\r\n most appropriate for the new graph. As a matter of fact, PageRank has been used successfully on many different graphs. I might be missing something here, so please correct me if I'm wrong.
\r\n\r\n
My approach seems to be different from others in the sense that it does not involve any feature extraction or supervised learning, it is a monolithic random walk approach. Here's a full description:
\r\n\r\n
\r\n
This is a 4-parameter algorithm. The ranking is based on a weighted sum of random walks on the graph, with lengths ranging from 1 to 3. Walks are performed in an undirected way, but weighting takes directivity into account.
\r\n\r\n
First, the graph is augmented with a universal sink. Walks reaching the sink will stay there forever. This can be modeled by using an infinite number of self-loops on the sink node. The sink is connected to every node in the graph by the average number of\r\n connections per node (which in this graph is ~10).
\r\n\r\n
Given a test node i, let's denote with p( j | i, L ) the probability of reaching j in L steps by traversing the graph from i in an undirected way. This probability can be expressed as the sum p( j | i, L ) = p( j, f | i, L ) + p( j, b | i, L ), where\r\n f and b mean that the last traversed link was a forward link or a backward link.
\r\n\r\n
Then, the score for each candidate node j is computed as follows:
\r\n\r\n
score(i, j) = sum_L=1^3 wf(L) p( j, f | i, L ) + wb(L) p( j, b | i, L )
\r\n\r\n
For each test node i, all candidates j are sorted according to this score. Parameter wf(1) does not have any relevance (since it only affects already-known friends) and the remaining parameters can be scaled without affecting the result, so it is possible\r\n to set wf(1) = wb(1) = 1, and only 4 parameters wf(2), wb(2), wf(3), wb(3) need to be found.
\r\n\r\n
\r\n
Parameters were selected by computing the MAP@10 on a separate validation set, which was generated by randomly removing outgoing edges from nodes\r\nnot in the test set.
\r\n\r\n
\r\n
Since there is no harm in adding up to 10 predictions per test node, empty recommendations are filled in sequentially by a) traversing the graph up to the 4th level and b) using the 10 more popular nodes (by number of incoming connections)
\r\n\r\n
These refinements do not have much impact on the final score.
\r\n\r\n
Feel free to ask any questions if some parts aren't clear enough!
", "rawMarkdown": "", "votes": null }, { "id": "12352", "postDate": "07/12/2012 21:04:46", "content": "Thanks, Miguel.
\r\nWhat was the purpose or thought behind the universal sink? Why did this improve your algorithm?
", "rawMarkdown": "", "votes": null }, { "id": "12360", "postDate": "07/13/2012 13:19:14", "content": "If you work for statistical arbitrage hedge fund, yes, if you predict links in facebook - really, who cares?
\r\nBut the conclusion remains, with the huge amount of effort, one can increase only by 0.02 from the naive,
\r\n100 lines in python approach. That's frustraing to me, there must be something wrong with data. :)
", "rawMarkdown": "", "votes": null }, { "id": "12361", "postDate": "07/13/2012 18:32:04", "content": "I used localized page-ranks and SVDs in my unsupervised model, but it occurred me that perhaps I was solving the wrong problem: we are supposed to find and rank the top ten missing links (connections), NOT recommend new connections (or rather the best new\r\n connections). These are fundamentally different sets, and perhaps this is why the supervised models work so much better. Any thoughts, or am I just full of crap? :)
", "rawMarkdown": "", "votes": null }, { "id": "12363", "postDate": "07/13/2012 19:06:38", "content": "[quote=Akulov Yaroslav;12339]
\r\n3. Generating of training sample. For supervised learning we want to generate a list of "good" and "bad" user pairs (X, Y). For that purprose 4% random graph edges were deleted. For each user whose friend relation was deleted candidate list was generated.\r\n Real edges were marked as 1, other pairs were marked as 0. After that list of 0 user pairs was randomly cut to get nearly equal number of 0 and 1 pairs (~400k of each type).
\r\n\r\n[quote=Miguel;12343]
\r\nFrom my perspective, I don't see any difference. If you were given a different graph, you'd retrain your supervised learning algorithm and hope it'd perform well. I'd do the same, repeating the crossvalidation procedure and selecting the weights that are\r\n most appropriate for the new graph. As a matter of fact, PageRank has been used successfully on many different graphs. I might be missing something here, so please correct me if I'm wrong.
\r\n[/quote]
\r\nHi Miguel, you are absolutely correct. Now that I see your approach, it is clear that you created a training set in order to optimize your parameters. It appears that you did your parameter search in a manual way, but there is no reason you could not automate\r\n this such that it could be run on a different graph. However, this parameter search is an important aspect of a generic approach and should be included in any assessment of the runtime. I highly doubt this is a convex probability space so you might have\r\n a tough time using a gradient based search approach. Optimizing 4 parameters with a grid search could take a while.
\r\nBy the way, my points here are certainly not to criticize in any way, but rather to create a discussion about the various approaches such that I might learn how different approaches may be similar or dissimilar. From a competition standpoint, my view is\r\n that anything that generated a higher score was better. So without any reservation I can easily say that your approach was better than mine and I congratulate you. And, I thank you for engaging in the discussion!
", "rawMarkdown": "", "votes": null }, { "id": "12365", "postDate": "07/13/2012 19:40:24", "content": "[quote=Akulov Yaroslav;12339]
\r\n4. Learning. On a resulting list of pairs three ML methods were tried.
\r\n\r\n", "rawMarkdown": "", "votes": null }, { "id": "12366", "postDate": "07/13/2012 20:17:08", "content": "[quote=vgoklani;12361]
\r\nI used localized page-ranks and SVDs in my unsupervised model, but it occurred me that perhaps I was solving the wrong problem: we are supposed to find and rank the top ten missing links (connections), NOT recommend new connections (or rather the best new\r\n connections). These are fundamentally different sets, and perhaps this is why the supervised models work so much better. Any thoughts, or am I just full of crap? :)
\r\n[/quote]
\r\nI think you make a very good observation here. I approached this with a mindset of "links have been deleted", rather than "links that might be followed". I did this because of the way the problem was setup. Outside of this competition I would have changed\r\n my mindset, but I haven't thought enough about how that might change my approach, if at all.
\r\nAlso, a comment about supervised / unsupervised... This is really not the right disctinction to draw. I see three basic approaches have been presented so far:
\r\n#1 is not supervised learning, but it is also not unsupervised learning, there is just no learning. Perhaps there was learning along the way, but the final algorithm has no learning built in and it's transferability to another problem might be limited.
\r\n#2 is not supervised learning (strictly speaking) because the parameters are being learned but not the model/function. This is more of an optimization problem than a learning problem. This may seem like a subtle distinction but does have some pretty important\r\n ramifications.
\r\n#3 is supervised learning because it attempts to learn the model/function.
\r\nUnsupervised learning would be used to answer a different question all together. To quote wikipedia: "In machine learning, unsupervised learning refers to the problem of trying to find hidden structure in unlabeled data". "Supervised learning is the machine\r\n learning task of inferring a function from supervised (labeled) training data.".
\r\nIf taken to the limit, supervised learning should be able to outperform an optimization approach if presented with multiple different graphs (assuming the model could be different from graph to graph). However, with the right model an optimization approach\r\n could outperform a supervised learning approach on a single graph because it could be less distracted by nuissance features.
\r\n\r\n
\r\n
\r\n
", "rawMarkdown": "", "votes": null }, { "id": "12367", "postDate": "07/13/2012 20:21:46", "content": "
Owen: As far as I know, it is not available as a package or software. Could not find any papers either.
\r\nYou could find some details in \r\nthis slides. There are some other slides available in Russian: Deck 1,\r\n\r\nDeck 2.
", "rawMarkdown": "", "votes": null }, { "id": "12368", "postDate": "07/13/2012 20:46:46", "content": "[quote=Brady Benware;12364]
\r\nHowever, this parameter search is an important aspect of a generic approach and should be included in any assessment of the runtime. I highly doubt this is a convex probability space so you might have a tough time using a gradient based search approach. Optimizing\r\n 4 parameters with a grid search could take a while.
\r\n[/quote]
\r\nI completely agree. Search space is indeed nonconvex, I found at least two local minima. Also, I don't think you can use a gradient-based search, since you can't compute the gradient analytically (and the cost function is not smooth, so strictly speaking,\r\n it doesn't exist). I'd go about this using increasingly finer grids, since it seems that minima are reasonably "far" appart.
\r\nOne interesting thing I noticed was that if I used the mentioned probabilities as features, sampled positive and negative instances from the graph (after removing some edges), and then trained a linear machine (naive Bayes, decision tree, single layer perceptron..),\r\n the weights that came out weren't too bad, but weren't as good as those that came out from the empirical minimization of the MAP@10. So it might be the case that using your features could result in a higher score if you minimized the MAP@10 directly.
\r\nI'm curious to know: You were very careful setting up your training data. Did you notice that being less careful (say, randomly selecting edges) made you method slightly worse or dramatically worse?
", "rawMarkdown": "", "votes": null }, { "id": "12369", "postDate": "07/13/2012 20:54:01", "content": "[quote=Brady Benware;12366]
\r\n#2 is not supervised learning (strictly speaking) because the parameters are being learned but not the model/function. This is more of an optimization problem than a learning problem. This may seem like a subtle distinction but does have some pretty important\r\n ramifications.
\r\n[/quote]
\r\nIt is just a matter of notation, but I can't help it, I have to ask :)
\r\nWouldn't you say that a neural network, or an SVM, are supervised learning? They are just a fixed model in which many parameters (the weights) are selected so as to minimize a cost functional. The reason I would not call my approach supervised learning is\r\n the lack of explicit labels, not the fact that several weights are learned in a fixed model...
", "rawMarkdown": "", "votes": null }, { "id": "12370", "postDate": "07/13/2012 21:32:03", "content": "[quote=Miguel;12368]
\r\nSo it might be the case that using your features could result in a higher score if you minimized the MAP@10 directly.
\r\n[/quote]
\r\nI had the same thought, but a) couldn't figure out how to do this b) figured that as long as you can do a good job separating positive and negative class then it really shouldn't help. If you take it to the limit of perfect classification then it wouldn't\r\n help.
\r\n[quote=Miguel;12368]
\r\nI'm curious to know: You were very careful setting up your training data. Did you notice that being less careful (say, randomly selecting edges) made you method slightly worse or dramatically worse?
\r\n[/quote]
\r\nI didn't do a careful analysis of this, but roughly speaking it made about a 0.002 difference between being close and being perfect. Not sure what the effect would have been if I was way off. The reason I ended up being really careful was that I noticed\r\n that highly connected nodes had lower MAP@10 scores in general. So I thought highly connected nodes may have a different model than lowly connected nodes. I tried to introduce the connectivity of node A into my features, but it\r\n always had a negative effect. I figured that if I could not introduce the connection count then I better be working from the same distribution.
", "rawMarkdown": "", "votes": null }, { "id": "12372", "postDate": "07/14/2012 00:18:11", "content": "[quote=Miguel;12369]
\r\nIt is just a matter of notation, but I can't help it, I have to ask :)
\r\nWouldn't you say that a neural network, or an SVM, are supervised learning? They are just a fixed model in which many parameters (the weights) are selected so as to minimize a cost functional. The reason I would not call my approach supervised learning is\r\n the lack of explicit labels, not the fact that several weights are learned in a fixed model...
\r\n[/quote]
\r\nThis is a really interesting question as it raises questions about the difference between supervised learning and optimization. I would say that SVM has a limited number of parameters (not the high dimensional weights) that get optimized outside of SVM\r\n itself. SVM will build a model based on a fixed set of parameters - the kernel function, the kernel function parameters and the soft error margin. A fixed set of values for these parameters will create a model that is optimal (under the constraints) for\r\n the observed data, but not necessarily the unseen data. One can then form an optimization problem around SVM to find the best values to feed SVM (typically a grid search because it is also not convex). So in my mind, you skipped the model building step (by\r\n specifying a good one directly) and went straight to the optimization part. There is absolutely nothing wrong with this, but does require some domain expertise to do.
\r\nEven with my approach I had a parameter optimization phase. For random forests the most important parameter is typically Mtry. I manually performed argmax(Mtry)MAP@10 . And this optimization problem also is not necessarily\r\n convex so the search can be painful (this case happened to be convex but it is not guaranteed). In this sense, my approach is no more general than yours since I did not automate this optimization phase. I don't have any experience with GBM (although it looks\r\n like I need to learn that one for sure), but I would guess it has a parameter that needs to be optimized too.
\r\nThis is something that has always bothered me about machine learning because it's a two step process. In step 1 you optimize the accuracy of reproducing the observed data based on a complexity control parameter and in step 2 you optimize the complexity\r\n control for accuracy of predicting unseen data. It's always seemed to me that there should be a way to specify a unified optimization problem.
", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 12294, "author_name": "solutionmatrices", "author_url": "", "post_date": "07/11/2012 02:15:52", "content": "Belated determination that only one edge of a matched set of output input edges was deleted from the graph since no test set nodes had 0 inputs and 0 outputs in training data. The training set was created by deleting edges except for the above exclusions.\r\n The public test sett was created by selecting edge/node pair until the number of nodes was 1/4 the number of deleted edges, so public test nodes were included in the private test set?
\r\nFor a given test node, the only possible predicted edges were outputs to nodes that were inputs to the test node? Guess not, while about 9 percent of training nodes had no inputs, about 2 percent of public test nodes had no inputs?
I believe the private test set included nodes from the public test set because non commutative edges were over 1 million in the public test set out of only about 2.9 million total in the training set. If the private test set was disjoint with identical\r\n probability distribution then the combined test sets would have over 4 million non commutative edges.
", "votes": null, "replies": [] }, { "id": 12297, "author_name": "", "author_url": "", "post_date": "07/11/2012 02:49:26", "content": "Hi, Brady
\r\nThanks.
", "votes": null, "replies": [] }, { "id": 12301, "author_name": "songgc", "author_url": "", "post_date": "07/11/2012 04:22:44", "content": "Cannot agree more. It is extremely important to establish the true training, cross-validation set.
\r\nIt is a little surprised that the advantage of supervised learning methods over unsupervised ones for this problem seems slight. A sub-graph-based pagerank-like one only needs 2 hours for total computation while the performance loss would be around 2%.
\r\nI don't know if anybody tried MLR (http://en.wikipedia.org/wiki/Learning_to_rank)
", "votes": null, "replies": [] }, { "id": 12302, "author_name": "miguel2", "author_url": "", "post_date": "07/11/2012 07:05:08", "content": "I'm a little surprised about the mentioned running times. Those mentioned 5-hour and 2-hour running times include prediction time only, or also the time required to generate crossvalidation sets, train supervised algorithms, etc?
\r\nIn contrast to this, my solution took only 10 min to compute on a 2.5 GHz, 4 GB RAM MacBook Pro, without resorting to any GPU tricks nor similar enhancements
\r\nIt is just a weighted sum of random walk probabilities on the graph. The algorithm requires only 4 parameters, which I selected manually by checking on a small validation set. A more careful parameter selection would have yielded better results, but unfortunately\r\n I wasted too much time playing around with more complex solutions and supervised learning, so I did not have the time to set up a proper parameter search. I might try to do this later, just to check the maximum achievable score using this idea.
\r\nI'd like to hear from the admins whether it is ok to disclose the full solution here. I would like to post mine and read about others. This was both funny and educating!
", "votes": null, "replies": [] }, { "id": 12304, "author_name": "vgoklani", "author_url": "", "post_date": "07/11/2012 11:42:08", "content": "Brady: I initially tried a supervised learning approach similar to yours, but didn't have much success. I suspect your secret sauce lies in the definition of your features :) I would be curious to see the actual code implementation, if you wouldn't mind\r\n posting it.
\r\nMiguel: I was curious about using random walks, but didn't have much time to explore. I would like to like see some more details on your "4 parameters", and how you did your implementation. Could you recommend some good references?
\r\nI spent a lot of time trying "Netflix style approaches", via randomized SVD/PCA, and also calculated localized weighted Page-ranks using breadth-first neighborhoods, but my implementation was always unsupervised, and never beat Dan Souleo's MAP@10 score\r\n :(
\r\nFor what it's worth, I found this competition really educational and fun! In particular, I learned how to properly use the python multiprocessing library (the secret is to properly set the chunk-size!), and I learned a lot about SVDs.
\r\nThank you Kaggle and Facebook for setting this up!
\r\n-V-
", "votes": null, "replies": [] }, { "id": 12307, "author_name": "gkoundry", "author_url": "", "post_date": "07/11/2012 13:08:21", "content": "I used an approach that was similar to Brady's. I also tried random walk, nearest neighbor, and EdgeRank based approaches but found that they didn't perform as well for me.
\r\nI also tried an MLR type approach by using a random forest to compare pairs of data points. It didn't work as well and I'm thinking it's because there are very few correct links compared to the ten guesses you're allowed per node. Most of the guesses are\r\n going to be wrong, so their ranking order doesn't really affect the score that much.
\r\nOne thing I did differently from Brady and Ben Hammer is I tried to use simple features for my random forest. I found that using features that are good predictors on their own actually reduces accuracy. I ended up using an MTRY of 7 and I think the reason\r\n Brady ended up with an MTRY of 2 was due to using strong predictors as features.
\r\nHere is a list of features I found useful:
\r\nExistence of a reverse link between nodes. (1=yes/0=no)\nCount of forward-forward links between nodes.\nCount of forward-reverse links between nodes.\nCount of forward-bidirectional links between nodes.\nCount of reverse-forward links between nodes.\nCount of reverse-reverse links between nodes.\nCount of reverse-bidirectional links between nodes.\nCount of bidirectional-forward links between nodes.\nCount of bidirectional-reverse links between nodes.\nCount of bidirectional-bidirectional links between nodes.\nCount of common neighbors.\nNumber of links ending at the node to be predicted / Number of links starting at the node to be predicted.\nNumber of links starting at the node to be ranked.\nNumber of links ending at the node to be ranked.\nNumber of links ending at the node to be ranked / Number of links starting at the node to be ranked.\nCount of common neighbors / Count of all neighbors.\nCount of paths with exactly three links between nodes / Count of paths with exactly three links from node to be predicted to any node.\nCount of forward-forward-forward links between nodes / Count of all length three paths between nodes.\nCount of forward-forward-reverse links between nodes / Count of all length three paths between nodes.\nCount of forward-reverse-forward links between nodes / Count of all length three paths between nodes.\nCount of forward-reverse-reverse links between nodes / Count of all length three paths between nodes.\nCount of reverse-forward-forward links between nodes / Count of all length three paths between nodes.\nCount of reverse-forward-reverse links between nodes / Count of all length three paths between nodes.\nCount of reverse-forward-forward links between nodes / Count of all length three paths between nodes.\nCount of reverse-forward-reverse links between nodes / Count of all length three paths between nodes.\nCount of reverse-reverse-forward links between nodes / Count of all length three paths between nodes.\nCount of reverse-reverse-reverse links between nodes / Count of all length three paths between nodes.\nAverage length of all unique paths from a node to its immediate successors.\nAverage length of all unique paths from a node to its immediate predecessors.\n[quote=Miguel;12302]
\r\nThose mentioned 5-hour and 2-hour running times include prediction time only, or also the time required to generate crossvalidation sets, train supervised algorithms, etc?
\r\n[/quote]
\r\nMy 5-hours was for everything, including generating the validation set (a few minutes), extracting features for both validation and test sets (~2 hours), training 1600 decision trees (~2 hours) and making predictions (~1 hour).
\r\nMiguel, your technique sounds simple and elegant. I do hope you share the details. Was your 10 minutes for everything, including a parameter search? \r\n
\r\nNot that it matters in a contest like this, but one advantage to using a supervised learning approach (instead of relying on exclusively on an edge rank / page rank solution with fixed parameters) is that it could be applied to a different graph with very\r\n few, in any, modifications.
", "votes": null, "replies": [] }, { "id": 12316, "author_name": "bradybenware", "author_url": "", "post_date": "07/11/2012 16:02:32", "content": "[quote=Glen;12307]
\r\nOne thing I did differently from Brady and Ben Hammer is I tried to use simple features for my random forest. I found that using features that are good predictors on their own actually reduces accuracy. I ended up using an MTRY of 7 and I think the reason\r\n Brady ended up with an MTRY of 2 was due to using strong predictors as features.
\r\n[/quote]
\r\nGlen, thanks for sharing. Don't take this the worng way, but I like your features :) One of the issues I kept contemplating was how to properly handle a node that belongs to multiple networks. I can understand how most of my feature would have best predictive\r\n capability for a node belonging to a single network, which is clearly a bad assumption and one I would have liked to avoid. I wonder if your weak predictors prehaps did a better job on highly connected nodes which are more likely to belong to multiple networks. \r\n I think I can justify how using absolute counts of connectedness, such as what you did, would work better in such a case than something that was normalized to the local network, such as what I did.
\r\nWas your focus on weak predictors based on a particular concept, or more from imperical observation about their performance?
", "votes": null, "replies": [] }, { "id": 12317, "author_name": "nhanitvn", "author_url": "", "post_date": "07/11/2012 16:19:25", "content": "Brandy, how did you add fake missing edges to your local training set? In the document, you mention only the way real missing edges were sampled. I think the way we choose fake missing edges is important also because there are features that discriminate\r\n these two types of missing edges.
", "votes": null, "replies": [] }, { "id": 12319, "author_name": "gkoundry", "author_url": "", "post_date": "07/11/2012 16:55:10", "content": "[quote=Brady Benware;12316]
\r\nI think I can justify how using absolute counts of connectedness, such as what you did, would work better in such a case than something that was normalized to the local network, such as what I did.
\r\nWas your focus on weak predictors based on a particular concept, or more from imperical observation about their performance?
\r\n[/quote]
\r\nActually I made mistake in my post. The counts of forward-forward, forward-reverse, etc. connections were normalized. Although I found that normalization only produced slightly better scores.
\r\nThe focus on weak predictors came from trying out strong predictors as features and having them hurt performance and giving strange parameter values for things like MTRY. The whole idea of random forests is to ensemble a bunch of weak predictors that have\r\n a high degree of variance in their predictions.
", "votes": null, "replies": [] }, { "id": 12321, "author_name": "vgoklani", "author_url": "", "post_date": "07/11/2012 19:55:45", "content": "Glen/Brady: Did either of you try any reduction techniques, to filter out either the weaker nodes or the "Bieber" nodes?\r\n
\r\nAlso when training your classifier, how did you optimize for the sparseness of the adjacency matrix?\r\n
\r\nOne problem I had when training my classifier was that all the 0's (from the sparsity) were throwing off my predictions. How do you handle that problem?
", "votes": null, "replies": [] }, { "id": 12323, "author_name": "lepieuvre", "author_url": "", "post_date": "07/12/2012 00:29:01", "content": "I think it would be a great idea if all of us with GitHub accounts throw up our code to discuss this. Make it a standard-ish name like "kaggle-facebook-lepieuvre" for example. I'll get mine up soon, I'm on a roadtrip.
\r\nI only finished 52nd but I'm happy with that considering I did the whole thing in Perl and on my laptop. I need to polish my Python skills for the next big competition.
", "votes": null, "replies": [] }, { "id": 12334, "author_name": "gkoundry", "author_url": "", "post_date": "07/12/2012 11:05:23", "content": "[quote=vgoklani;12321]
\r\nGlen/Brady: Did either of you try any reduction techniques, to filter out either the weaker nodes or the "Bieber" nodes?
\r\nAlso when training your classifier, how did you optimize for the sparseness of the adjacency matrix?
\r\nOne problem I had when training my classifier was that all the 0's (from the sparsity) were throwing off my predictions. How do you handle that problem?
\r\n[/quote]
\r\nI did not try filtering out any nodes.
\r\nMy training data consisted of all nodes one or two hops away from the node being predicted. Only around 5% of these nodes are actually connected and I found sampling the data to have 10% connections produced better results.
", "votes": null, "replies": [] }, { "id": 12336, "author_name": "alexoz", "author_url": "", "post_date": "07/12/2012 12:45:17", "content": "So what do we learn from here:
\r\n1) Russian schooling still rules, I am proud of my motherland.
\r\n2) There is no killing algorithm.
\r\nI am curious what the winners did, but
\r\neven obvious solution like RootedPageRank (augmented with special stochatic matrix that
\r\nallows to porpagte both direction taking into accout fiendliness of the nodes) which I tried, performed slightly worse than
\r\nthe naive 7130-aproach.
\r\n3) Personally, I doubt that in real production environment I'd make (or my boss allowed me) an effort to imporve from 0.71 to 0.72, would you?
\r\nI did not have time (I spent in total only 15 hours or so), to adjust all the parameters and prepare good train and validation data
\r\nbut it was kind of curios to see that no matter what parameters I chose or taking different schemes to deal with dangling links
\r\nreally did not matter in practice.
", "votes": null, "replies": [] }, { "id": 12338, "author_name": "omarwugui", "author_url": "", "post_date": "07/12/2012 15:46:52", "content": "Hey Axel,
\r\nPersonally, I've become obsessed with preparing "good" train and validation data. I didn't even get to make a submission!\r\n
\r\nBut if you'd like to demonstrate more of that russian schooling, talk to me about preparing balanced data to train models on (maybe not a bad idea since the algorithm used didnt seem to matter much)
Here is a short overview of my approach. It doesn't differs very much from other supervise learning approaches described above but may be somebody will find something useful. I'll be glad to answer any questions. Main points:
\r\n", "votes": null, "replies": [] }, { "id": 12341, "author_name": "yarreg", "author_url": "", "post_date": "07/12/2012 16:09:10", "content": "[quote=Axel;12336]
\r\n3) Personally, I doubt that in real production environment I'd make (or my boss allowed me) an effort to imporve from 0.71 to 0.72, would you?
\r\n[/quote]
\r\nAs long as 1% improvement leads to e.g. 0.1% rise in income of multibillion dollar corporation I suppose it's really worth it to make such an effort. :) But may be this is not the case.
", "votes": null, "replies": [] }, { "id": 12343, "author_name": "miguel2", "author_url": "", "post_date": "07/12/2012 17:58:04", "content": "vgoklani: A survey on random walks that I found very interesting is "Link Prediction in Complex Networks: A Survey"
\r\n\r\n
Brady: The 10 minutes time was only for prediction. Validation time can vary a lot depending on the size of your validation set and how finely you sweep the parameters. In my case, this was done interactively just before the competition ended. I'd say it\r\n took about 50 min, including time for running all validations and time for me to change the parameters for the next run. I had no time to do a proper sweep, this was about one hour before the competition ended.
\r\n\r\n
Also, I don't agree with this:
\r\n\r\n
"one advantage to using a supervised learning approach (instead of relying on exclusively on an edge rank / page rank solution with fixed parameters) is that it could be applied to a different graph with very few, in any, modifications."
\r\n\r\n
From my perspective, I don't see any difference. If you were given a different graph, you'd retrain your supervised learning algorithm and hope it'd perform well. I'd do the same, repeating the crossvalidation procedure and selecting the weights that are\r\n most appropriate for the new graph. As a matter of fact, PageRank has been used successfully on many different graphs. I might be missing something here, so please correct me if I'm wrong.
\r\n\r\n
My approach seems to be different from others in the sense that it does not involve any feature extraction or supervised learning, it is a monolithic random walk approach. Here's a full description:
\r\n\r\n
\r\n
This is a 4-parameter algorithm. The ranking is based on a weighted sum of random walks on the graph, with lengths ranging from 1 to 3. Walks are performed in an undirected way, but weighting takes directivity into account.
\r\n\r\n
First, the graph is augmented with a universal sink. Walks reaching the sink will stay there forever. This can be modeled by using an infinite number of self-loops on the sink node. The sink is connected to every node in the graph by the average number of\r\n connections per node (which in this graph is ~10).
\r\n\r\n
Given a test node i, let's denote with p( j | i, L ) the probability of reaching j in L steps by traversing the graph from i in an undirected way. This probability can be expressed as the sum p( j | i, L ) = p( j, f | i, L ) + p( j, b | i, L ), where\r\n f and b mean that the last traversed link was a forward link or a backward link.
\r\n\r\n
Then, the score for each candidate node j is computed as follows:
\r\n\r\n
score(i, j) = sum_L=1^3 wf(L) p( j, f | i, L ) + wb(L) p( j, b | i, L )
\r\n\r\n
For each test node i, all candidates j are sorted according to this score. Parameter wf(1) does not have any relevance (since it only affects already-known friends) and the remaining parameters can be scaled without affecting the result, so it is possible\r\n to set wf(1) = wb(1) = 1, and only 4 parameters wf(2), wb(2), wf(3), wb(3) need to be found.
\r\n\r\n
\r\n
Parameters were selected by computing the MAP@10 on a separate validation set, which was generated by randomly removing outgoing edges from nodes\r\nnot in the test set.
\r\n\r\n
\r\n
Since there is no harm in adding up to 10 predictions per test node, empty recommendations are filled in sequentially by a) traversing the graph up to the 4th level and b) using the 10 more popular nodes (by number of incoming connections)
\r\n\r\n
These refinements do not have much impact on the final score.
\r\n\r\n
Feel free to ask any questions if some parts aren't clear enough!
", "votes": null, "replies": [] }, { "id": 12352, "author_name": "erdman", "author_url": "", "post_date": "07/12/2012 21:04:46", "content": "Thanks, Miguel.
\r\nWhat was the purpose or thought behind the universal sink? Why did this improve your algorithm?
", "votes": null, "replies": [] }, { "id": 12360, "author_name": "alexoz", "author_url": "", "post_date": "07/13/2012 13:19:14", "content": "If you work for statistical arbitrage hedge fund, yes, if you predict links in facebook - really, who cares?
\r\nBut the conclusion remains, with the huge amount of effort, one can increase only by 0.02 from the naive,
\r\n100 lines in python approach. That's frustraing to me, there must be something wrong with data. :)
", "votes": null, "replies": [] }, { "id": 12361, "author_name": "vgoklani", "author_url": "", "post_date": "07/13/2012 18:32:04", "content": "I used localized page-ranks and SVDs in my unsupervised model, but it occurred me that perhaps I was solving the wrong problem: we are supposed to find and rank the top ten missing links (connections), NOT recommend new connections (or rather the best new\r\n connections). These are fundamentally different sets, and perhaps this is why the supervised models work so much better. Any thoughts, or am I just full of crap? :)
", "votes": null, "replies": [] }, { "id": 12363, "author_name": "onthefly", "author_url": "", "post_date": "07/13/2012 19:06:38", "content": "[quote=Akulov Yaroslav;12339]
\r\n3. Generating of training sample. For supervised learning we want to generate a list of "good" and "bad" user pairs (X, Y). For that purprose 4% random graph edges were deleted. For each user whose friend relation was deleted candidate list was generated.\r\n Real edges were marked as 1, other pairs were marked as 0. After that list of 0 user pairs was randomly cut to get nearly equal number of 0 and 1 pairs (~400k of each type).
\r\n\r\n[quote=Miguel;12343]
\r\nFrom my perspective, I don't see any difference. If you were given a different graph, you'd retrain your supervised learning algorithm and hope it'd perform well. I'd do the same, repeating the crossvalidation procedure and selecting the weights that are\r\n most appropriate for the new graph. As a matter of fact, PageRank has been used successfully on many different graphs. I might be missing something here, so please correct me if I'm wrong.
\r\n[/quote]
\r\nHi Miguel, you are absolutely correct. Now that I see your approach, it is clear that you created a training set in order to optimize your parameters. It appears that you did your parameter search in a manual way, but there is no reason you could not automate\r\n this such that it could be run on a different graph. However, this parameter search is an important aspect of a generic approach and should be included in any assessment of the runtime. I highly doubt this is a convex probability space so you might have\r\n a tough time using a gradient based search approach. Optimizing 4 parameters with a grid search could take a while.
\r\nBy the way, my points here are certainly not to criticize in any way, but rather to create a discussion about the various approaches such that I might learn how different approaches may be similar or dissimilar. From a competition standpoint, my view is\r\n that anything that generated a higher score was better. So without any reservation I can easily say that your approach was better than mine and I congratulate you. And, I thank you for engaging in the discussion!
", "votes": null, "replies": [] }, { "id": 12365, "author_name": "owenzhang1", "author_url": "", "post_date": "07/13/2012 19:40:24", "content": "[quote=Akulov Yaroslav;12339]
\r\n4. Learning. On a resulting list of pairs three ML methods were tried.
\r\n\r\n", "votes": null, "replies": [] }, { "id": 12366, "author_name": "bradybenware", "author_url": "", "post_date": "07/13/2012 20:17:08", "content": "[quote=vgoklani;12361]
\r\nI used localized page-ranks and SVDs in my unsupervised model, but it occurred me that perhaps I was solving the wrong problem: we are supposed to find and rank the top ten missing links (connections), NOT recommend new connections (or rather the best new\r\n connections). These are fundamentally different sets, and perhaps this is why the supervised models work so much better. Any thoughts, or am I just full of crap? :)
\r\n[/quote]
\r\nI think you make a very good observation here. I approached this with a mindset of "links have been deleted", rather than "links that might be followed". I did this because of the way the problem was setup. Outside of this competition I would have changed\r\n my mindset, but I haven't thought enough about how that might change my approach, if at all.
\r\nAlso, a comment about supervised / unsupervised... This is really not the right disctinction to draw. I see three basic approaches have been presented so far:
\r\n#1 is not supervised learning, but it is also not unsupervised learning, there is just no learning. Perhaps there was learning along the way, but the final algorithm has no learning built in and it's transferability to another problem might be limited.
\r\n#2 is not supervised learning (strictly speaking) because the parameters are being learned but not the model/function. This is more of an optimization problem than a learning problem. This may seem like a subtle distinction but does have some pretty important\r\n ramifications.
\r\n#3 is supervised learning because it attempts to learn the model/function.
\r\nUnsupervised learning would be used to answer a different question all together. To quote wikipedia: "In machine learning, unsupervised learning refers to the problem of trying to find hidden structure in unlabeled data". "Supervised learning is the machine\r\n learning task of inferring a function from supervised (labeled) training data.".
\r\nIf taken to the limit, supervised learning should be able to outperform an optimization approach if presented with multiple different graphs (assuming the model could be different from graph to graph). However, with the right model an optimization approach\r\n could outperform a supervised learning approach on a single graph because it could be less distracted by nuissance features.
\r\n\r\n
\r\n
\r\n
", "votes": null, "replies": [] }, { "id": 12367, "author_name": "tuzzeg", "author_url": "", "post_date": "07/13/2012 20:21:46", "content": "
Owen: As far as I know, it is not available as a package or software. Could not find any papers either.
\r\nYou could find some details in \r\nthis slides. There are some other slides available in Russian: Deck 1,\r\n\r\nDeck 2.
", "votes": null, "replies": [] }, { "id": 12368, "author_name": "miguel2", "author_url": "", "post_date": "07/13/2012 20:46:46", "content": "[quote=Brady Benware;12364]
\r\nHowever, this parameter search is an important aspect of a generic approach and should be included in any assessment of the runtime. I highly doubt this is a convex probability space so you might have a tough time using a gradient based search approach. Optimizing\r\n 4 parameters with a grid search could take a while.
\r\n[/quote]
\r\nI completely agree. Search space is indeed nonconvex, I found at least two local minima. Also, I don't think you can use a gradient-based search, since you can't compute the gradient analytically (and the cost function is not smooth, so strictly speaking,\r\n it doesn't exist). I'd go about this using increasingly finer grids, since it seems that minima are reasonably "far" appart.
\r\nOne interesting thing I noticed was that if I used the mentioned probabilities as features, sampled positive and negative instances from the graph (after removing some edges), and then trained a linear machine (naive Bayes, decision tree, single layer perceptron..),\r\n the weights that came out weren't too bad, but weren't as good as those that came out from the empirical minimization of the MAP@10. So it might be the case that using your features could result in a higher score if you minimized the MAP@10 directly.
\r\nI'm curious to know: You were very careful setting up your training data. Did you notice that being less careful (say, randomly selecting edges) made you method slightly worse or dramatically worse?
", "votes": null, "replies": [] }, { "id": 12369, "author_name": "miguel2", "author_url": "", "post_date": "07/13/2012 20:54:01", "content": "[quote=Brady Benware;12366]
\r\n#2 is not supervised learning (strictly speaking) because the parameters are being learned but not the model/function. This is more of an optimization problem than a learning problem. This may seem like a subtle distinction but does have some pretty important\r\n ramifications.
\r\n[/quote]
\r\nIt is just a matter of notation, but I can't help it, I have to ask :)
\r\nWouldn't you say that a neural network, or an SVM, are supervised learning? They are just a fixed model in which many parameters (the weights) are selected so as to minimize a cost functional. The reason I would not call my approach supervised learning is\r\n the lack of explicit labels, not the fact that several weights are learned in a fixed model...
", "votes": null, "replies": [] }, { "id": 12370, "author_name": "bradybenware", "author_url": "", "post_date": "07/13/2012 21:32:03", "content": "[quote=Miguel;12368]
\r\nSo it might be the case that using your features could result in a higher score if you minimized the MAP@10 directly.
\r\n[/quote]
\r\nI had the same thought, but a) couldn't figure out how to do this b) figured that as long as you can do a good job separating positive and negative class then it really shouldn't help. If you take it to the limit of perfect classification then it wouldn't\r\n help.
\r\n[quote=Miguel;12368]
\r\nI'm curious to know: You were very careful setting up your training data. Did you notice that being less careful (say, randomly selecting edges) made you method slightly worse or dramatically worse?
\r\n[/quote]
\r\nI didn't do a careful analysis of this, but roughly speaking it made about a 0.002 difference between being close and being perfect. Not sure what the effect would have been if I was way off. The reason I ended up being really careful was that I noticed\r\n that highly connected nodes had lower MAP@10 scores in general. So I thought highly connected nodes may have a different model than lowly connected nodes. I tried to introduce the connectivity of node A into my features, but it\r\n always had a negative effect. I figured that if I could not introduce the connection count then I better be working from the same distribution.
", "votes": null, "replies": [] }, { "id": 12372, "author_name": "bradybenware", "author_url": "", "post_date": "07/14/2012 00:18:11", "content": "[quote=Miguel;12369]
\r\nIt is just a matter of notation, but I can't help it, I have to ask :)
\r\nWouldn't you say that a neural network, or an SVM, are supervised learning? They are just a fixed model in which many parameters (the weights) are selected so as to minimize a cost functional. The reason I would not call my approach supervised learning is\r\n the lack of explicit labels, not the fact that several weights are learned in a fixed model...
\r\n[/quote]
\r\nThis is a really interesting question as it raises questions about the difference between supervised learning and optimization. I would say that SVM has a limited number of parameters (not the high dimensional weights) that get optimized outside of SVM\r\n itself. SVM will build a model based on a fixed set of parameters - the kernel function, the kernel function parameters and the soft error margin. A fixed set of values for these parameters will create a model that is optimal (under the constraints) for\r\n the observed data, but not necessarily the unseen data. One can then form an optimization problem around SVM to find the best values to feed SVM (typically a grid search because it is also not convex). So in my mind, you skipped the model building step (by\r\n specifying a good one directly) and went straight to the optimization part. There is absolutely nothing wrong with this, but does require some domain expertise to do.
\r\nEven with my approach I had a parameter optimization phase. For random forests the most important parameter is typically Mtry. I manually performed argmax(Mtry)MAP@10 . And this optimization problem also is not necessarily\r\n convex so the search can be painful (this case happened to be convex but it is not guaranteed). In this sense, my approach is no more general than yours since I did not automate this optimization phase. I don't have any experience with GBM (although it looks\r\n like I need to learn that one for sure), but I would guess it has a parameter that needs to be optimized too.
\r\nThis is something that has always bothered me about machine learning because it's a two step process. In step 1 you optimize the accuracy of reproducing the observed data based on a complexity control parameter and in step 2 you optimize the complexity\r\n control for accuracy of predicting unseen data. It's always seemed to me that there should be a way to specify a unified optimization problem.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "12290": "", "12294": "", "12297": "", "12301": "", "12302": "", "12304": "", "12307": "", "12308": "", "12316": "", "12317": "", "12319": "", "12321": "", "12323": "", "12334": "", "12336": "", "12338": "", "12339": "", "12341": "", "12343": "", "12352": "", "12360": "", "12361": "", "12363": "", "12364": "", "12365": "", "12366": "", "12367": "", "12368": "", "12369": "", "12370": "", "12372": "" }, "source": "existing_discussion", "complete": true, "competition_id": 2947, "competition_slug": "FacebookRecruiting", "team_id": 19158, "solution_url": "/competitions/FacebookRecruiting/writeups/brady-benware-curious-minds-want-to-know" }