{ "id": 2000, "title": "Good first step: make graph more commutative", "url": "/competitions/FacebookRecruiting/discussion/2000", "author_name": "Aaron Schumacher", "post_date": "2012-06-06T18:56:59.963000", "votes": 22, "comment_count": 10, "views": 8129, "content": "

Hi! So, this graph is directed, meaning you can have A -> B without having B -> A. For kicks, I did a solution where my only predictions were those that made the graph more commutative. That is, if in the training set A -> B but not B -> A, and I had to\r\n predict for B, I predicted A. When there were more than ten such possible predictions, I just took the first that happened to be in the list. For some entries in the test set, I didn't make any prediction at all. I was wondering if I would get a score of zero,\r\n since this is such a simple technique and maybe the data was designed so that all good predictions didn't exist going either direction in the training set. But instead I got a score of 0.61523, which is currently good enough for third place. So: I suggest\r\n as a first step just introducing all the missing reciprocal edges that you can - then work on making a smart algorithm for your remaining guesses. No more low scores!

", "messages": [ { "id": 11426, "postDate": "2012-06-06T18:56:59.963Z", "content": "

Hi! So, this graph is directed, meaning you can have A -> B without having B -> A. For kicks, I did a solution where my only predictions were those that made the graph more commutative. That is, if in the training set A -> B but not B -> A, and I had to\r\n predict for B, I predicted A. When there were more than ten such possible predictions, I just took the first that happened to be in the list. For some entries in the test set, I didn't make any prediction at all. I was wondering if I would get a score of zero,\r\n since this is such a simple technique and maybe the data was designed so that all good predictions didn't exist going either direction in the training set. But instead I got a score of 0.61523, which is currently good enough for third place. So: I suggest\r\n as a first step just introducing all the missing reciprocal edges that you can - then work on making a smart algorithm for your remaining guesses. No more low scores!

", "votes": 22 }, { "id": 11434, "postDate": "2012-06-07T01:03:36.550Z", "content": "
# This code generates a file that gets a score of 0.61525
\r\n
# Also at https://gist.github.com/2885835
\r\n
 
\r\n
import csv

r = csv.reader(open('train.csv','r'))
r.next()

edges = set()
#commutative_graph = dict()
for edge in r:
    edges.add((edge[0], edge[1]))
#    commutative_graph.setdefault(edge[0], set()).add(edge[1])
#    commutative_graph.setdefault(edge[1], set()).add(edge[0])

missing_edges = set()
for edge in edges:
    if (edge[1], edge[0]) not in edges:
        missing_edges.add((edge[1], edge[0]))

missing_graph = dict()
for edge in missing_edges:
    missing_graph.setdefault(edge[0], set()).add(edge[1])

r = csv.reader(open('test.csv','r'))
r.next()

test_list = list()
for line in r:
    test_list.append(line[0])

test_lists = dict()
for node in test_list:
    test_lists[node] = list(missing_graph.get(node, set()))

file = open('output.csv','w')
w = csv.writer(file)
w.writerow(['source_node','destination_nodes'])
for node in test_list:
    w.writerow([node, " ".join(test_lists[node][0:9])])
file.close()
", "votes": 14 }, { "id": 11899, "postDate": "2012-06-26T16:51:13.733Z", "content": "

The technique you describe is at the heart of \r\nDon Souleo's link-following technique .

\r\n

On a slightly different note, I've found it useful to think in terms of how balanced the outbound links (from Node A --> Node B) are to the inbound links (A <-- B). 

\r\n

Let's say Node A links to 100 nodes and is linked to by 50 nodes.  Then its 'outbound ratio' is 100 / (100 + 50) or 0.667.  So, 'outbound ratio' lies in [0,1].  Looking at the conditional probability that an inbound link will be matched by an outbound link\r\n (that is, if B-->A, what is the probability that A-->B), and plotting according to a node's outbound ratio, I get the following chart.

\r\n

 

\r\n

\"Probability

\r\n

Intuitively, this makes sense (or perhaps is obvious ;-)

\r\n

The X-axis is the 'outbound ratio'.  Smaller numbers mean more inbound links than outbound.  Thus, we'd expect a low probability for an inbound link to be matched by an outbound link.  The Y-axis is the conditional probability of\r\n a node having an outbound link given an inbound link.

\r\n

As 'outbound ratio' increases, so does probability of that an inbound link will be matched to an outbound link.  After 50%, the probability levels off.  Again, this might be stating the obvious, but understanding the predictive power\r\n of an inbound link based on a node's outbound ratio can be useful when blending multiple strategies.  If a node has many inbound links and very few outbound (Justin Bieber on Twitter), then you're not going to get a lot from using inbound links.  

", "votes": 4 }, { "id": 11889, "postDate": "2012-06-26T06:00:40.193Z", "content": "

Wow, i was working on a truncated file! had half the inputs - gave me half the score. Your reference helped me found it, thanks!

", "votes": 1 }, { "id": 11862, "postDate": "2012-06-25T02:56:24.383Z", "content": "
w.writerow([node," ".join(test_lists[node][0:9])]) -> should be test_lists[node][0:10] for the right number of suggestions (10)
\r\n
 
", "votes": 1 }, { "id": 11861, "postDate": "2012-06-25T01:14:05.940Z", "content": "

Hi drago! That's interesting; I don't know what's happening on your system. Let me know if you figure it out. Good luck!

", "votes": 1 }, { "id": 11860, "postDate": "2012-06-25T00:22:18.197Z", "content": "

I ran your code - it scored only 0.34736.

" }, { "id": 11639, "postDate": "2012-06-16T02:09:43.867Z" }, { "id": 11503, "postDate": "2012-06-10T06:12:21.557Z", "content": "

You just beat me to it, that's exactly the first thing I wanted to submit, but decided to first load everything to a db :)

" }, { "id": 11498, "postDate": "2012-06-09T21:16:44.097Z", "content": "

wow, that's completely counterintuitive to what I've been doing i.e. filter out all non-reciprocating relationships and only work with reciprocating relationships (wouldn't want to be recommended as a friend to someone who's friend request I didn't accept\r\n - or to their friends).

" }, { "id": 11429, "postDate": "2012-06-06T19:09:39.910Z", "rawMarkdown": "", "votes": 1, "isDeleted": true } ], "comments": [ { "id": 11434, "author_name": "Aaron Schumacher", "author_url": "", "post_date": "2012-06-07T01:03:36.550000", "content": "
# This code generates a file that gets a score of 0.61525
\r\n
# Also at https://gist.github.com/2885835
\r\n
 
\r\n
import csv

r = csv.reader(open('train.csv','r'))
r.next()

edges = set()
#commutative_graph = dict()
for edge in r:
    edges.add((edge[0], edge[1]))
#    commutative_graph.setdefault(edge[0], set()).add(edge[1])
#    commutative_graph.setdefault(edge[1], set()).add(edge[0])

missing_edges = set()
for edge in edges:
    if (edge[1], edge[0]) not in edges:
        missing_edges.add((edge[1], edge[0]))

missing_graph = dict()
for edge in missing_edges:
    missing_graph.setdefault(edge[0], set()).add(edge[1])

r = csv.reader(open('test.csv','r'))
r.next()

test_list = list()
for line in r:
    test_list.append(line[0])

test_lists = dict()
for node in test_list:
    test_lists[node] = list(missing_graph.get(node, set()))

file = open('output.csv','w')
w = csv.writer(file)
w.writerow(['source_node','destination_nodes'])
for node in test_list:
    w.writerow([node, " ".join(test_lists[node][0:9])])
file.close()
", "votes": 14, "replies": [] }, { "id": 11899, "author_name": "Dave Klein", "author_url": "", "post_date": "2012-06-26T16:51:13.733000", "content": "

The technique you describe is at the heart of \r\nDon Souleo's link-following technique .

\r\n

On a slightly different note, I've found it useful to think in terms of how balanced the outbound links (from Node A --> Node B) are to the inbound links (A <-- B). 

\r\n

Let's say Node A links to 100 nodes and is linked to by 50 nodes.  Then its 'outbound ratio' is 100 / (100 + 50) or 0.667.  So, 'outbound ratio' lies in [0,1].  Looking at the conditional probability that an inbound link will be matched by an outbound link\r\n (that is, if B-->A, what is the probability that A-->B), and plotting according to a node's outbound ratio, I get the following chart.

\r\n

 

\r\n

\"Probability

\r\n

Intuitively, this makes sense (or perhaps is obvious ;-)

\r\n

The X-axis is the 'outbound ratio'.  Smaller numbers mean more inbound links than outbound.  Thus, we'd expect a low probability for an inbound link to be matched by an outbound link.  The Y-axis is the conditional probability of\r\n a node having an outbound link given an inbound link.

\r\n

As 'outbound ratio' increases, so does probability of that an inbound link will be matched to an outbound link.  After 50%, the probability levels off.  Again, this might be stating the obvious, but understanding the predictive power\r\n of an inbound link based on a node's outbound ratio can be useful when blending multiple strategies.  If a node has many inbound links and very few outbound (Justin Bieber on Twitter), then you're not going to get a lot from using inbound links.  

", "votes": 4, "replies": [] }, { "id": 11889, "author_name": "Doron", "author_url": "", "post_date": "2012-06-26T06:00:40.193000", "content": "

Wow, i was working on a truncated file! had half the inputs - gave me half the score. Your reference helped me found it, thanks!

", "votes": 1, "replies": [] }, { "id": 11862, "author_name": "nhan vu", "author_url": "", "post_date": "2012-06-25T02:56:24.383000", "content": "
w.writerow([node," ".join(test_lists[node][0:9])]) -> should be test_lists[node][0:10] for the right number of suggestions (10)
\r\n
 
", "votes": 1, "replies": [] }, { "id": 11861, "author_name": "Aaron Schumacher", "author_url": "", "post_date": "2012-06-25T01:14:05.940000", "content": "

Hi drago! That's interesting; I don't know what's happening on your system. Let me know if you figure it out. Good luck!

", "votes": 1, "replies": [] }, { "id": 11860, "author_name": "Doron", "author_url": "", "post_date": "2012-06-25T00:22:18.197000", "content": "

I ran your code - it scored only 0.34736.

", "votes": 0, "replies": [] }, { "id": 11639, "author_name": "vgoklani", "author_url": "", "post_date": "2012-06-16T02:09:43.867000", "content": "", "votes": 0, "replies": [] }, { "id": 11503, "author_name": "r0u1i", "author_url": "", "post_date": "2012-06-10T06:12:21.557000", "content": "

You just beat me to it, that's exactly the first thing I wanted to submit, but decided to first load everything to a db :)

", "votes": 0, "replies": [] }, { "id": 11498, "author_name": "djmachet", "author_url": "", "post_date": "2012-06-09T21:16:44.097000", "content": "

wow, that's completely counterintuitive to what I've been doing i.e. filter out all non-reciprocating relationships and only work with reciprocating relationships (wouldn't want to be recommended as a friend to someone who's friend request I didn't accept\r\n - or to their friends).

", "votes": 0, "replies": [] }, { "id": 11429, "author_name": "", "author_url": "", "post_date": "2012-06-06T19:09:39.910000", "content": "", "votes": 1, "replies": [] } ], "raw_markdown_by_id": { "11426": "", "11434": "", "11899": "", "11889": "", "11862": "", "11861": "", "11860": "", "11639": "", "11503": "", "11498": "", "11429": "" } }