{ "id": 2047, "title": "Reducing run time for graph creation?", "url": "/competitions/FacebookRecruiting/discussion/2047", "author_name": "", "post_date": "2012-06-18T04:13:50.373Z", "votes": null, "comment_count": 7, "views": 4201, "content": "

Any tips on reducing the run time for graph creation? It takes nearly 40 seconds on my i7 with 4 GB of RAM. If I add one more graph, it shoots up to 80 seconds. If I use pickle to save the data structure, I save 30 seconds for the 1st graph (with a 68 MB\r\n file). Not able to successfully save the second graph with pickle yet. Eventually, for submitting, I guess we have to let the graph computation happen in run-time.

", "messages": [ { "id": "11698", "postDate": "06/18/2012 04:13:50", "content": "

Any tips on reducing the run time for graph creation? It takes nearly 40 seconds on my i7 with 4 GB of RAM. If I add one more graph, it shoots up to 80 seconds. If I use pickle to save the data structure, I save 30 seconds for the 1st graph (with a 68 MB\r\n file). Not able to successfully save the second graph with pickle yet. Eventually, for submitting, I guess we have to let the graph computation happen in run-time.

", "rawMarkdown": "", "votes": null }, { "id": "11701", "postDate": "06/18/2012 08:37:09", "content": "

My tip for speeding up a python program would be to rewrite the performance critical part (or the whole program) in C / C++. Obviously this isn't without downsides...

", "rawMarkdown": "", "votes": null }, { "id": "11706", "postDate": "06/18/2012 11:44:54", "content": "

I didn't see any performance requirement in the evaluation criteria. As you are only submitting predictions, you can take as long as you want (within reason) to generate the predictions. The goal here is ACCURACY.

\r\n

Depending on how this result is going to be used (if it is going to be used) in production, the need to have "real-time" predictions may or may not come up.

\r\n

Beyond writing C/C++ code for the critical part of your code, you can always consider parallel algorithms. But as roobs said, that "isn't withou downsides.." either.

", "rawMarkdown": "", "votes": null }, { "id": "11711", "postDate": "06/18/2012 17:21:25", "content": "

I also use python and does not have such an issue. good side with python is you can use the interpreter. so that previous data is on the memory and you can play with them the way you like.\r\n
\r\nI use the python interpreter on linux can can generate the full graph with in 15-20 seconds on my i5 4gb machine\r\n

", "rawMarkdown": "", "votes": null }, { "id": "11745", "postDate": "06/19/2012 18:43:54", "content": "

For Python, I'm using the igraph library, it's pretty fast since it's using native code libigraph in the background.

", "rawMarkdown": "", "votes": null }, { "id": "11779", "postDate": "06/21/2012 02:32:51", "content": "

I don't see too much downside, except it takes a bit longer to write the code. Well, you do lose the ability -- if you want that -- to easily explore the graph interactively.  Anyway, it took me only a few minutes to code in c++, and I can read the graph\r\n in just under 3 seconds.  This is on a fairly new linux box with a few G of RAM.

", "rawMarkdown": "", "votes": null }, { "id": "11781", "postDate": "06/21/2012 03:33:09", "content": "

Thanks for all the replies.

\r\n

Hi David,

\r\n

The more complex my algorithm is getting, the more I feel that I need to move away from Python. It's becoming extremely slow handling such a big graph. What datastructure are you using? Are you using Boost library?

", "rawMarkdown": "", "votes": null }, { "id": "11815", "postDate": "06/22/2012 03:08:31", "content": "

I use STL in c++. My representation of the graph: map<int, set<int> > which maps

\r\n

{person -> set of people the person is following}

\r\n

I've used boost before, and know it is well thought of, but personally find their documentation --esp. for the graph library -- abysmal.  Some day I'll have enough free time to master it (perhaps).

\r\n

I *really* like python for quick prototyping, but when performance becomes an issue I graduate the code to C++. My experience with pickle is: for anything very large it's *very* slow.

\r\n

 

", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 11701, "author_name": "roobs5218", "author_url": "", "post_date": "06/18/2012 08:37:09", "content": "

My tip for speeding up a python program would be to rewrite the performance critical part (or the whole program) in C / C++. Obviously this isn't without downsides...

", "votes": null, "replies": [] }, { "id": 11706, "author_name": "owenzhang1", "author_url": "", "post_date": "06/18/2012 11:44:54", "content": "

I didn't see any performance requirement in the evaluation criteria. As you are only submitting predictions, you can take as long as you want (within reason) to generate the predictions. The goal here is ACCURACY.

\r\n

Depending on how this result is going to be used (if it is going to be used) in production, the need to have "real-time" predictions may or may not come up.

\r\n

Beyond writing C/C++ code for the critical part of your code, you can always consider parallel algorithms. But as roobs said, that "isn't withou downsides.." either.

", "votes": null, "replies": [] }, { "id": 11711, "author_name": "udnisap", "author_url": "", "post_date": "06/18/2012 17:21:25", "content": "

I also use python and does not have such an issue. good side with python is you can use the interpreter. so that previous data is on the memory and you can play with them the way you like.\r\n
\r\nI use the python interpreter on linux can can generate the full graph with in 15-20 seconds on my i5 4gb machine\r\n

", "votes": null, "replies": [] }, { "id": 11745, "author_name": "amastilovic", "author_url": "", "post_date": "06/19/2012 18:43:54", "content": "

For Python, I'm using the igraph library, it's pretty fast since it's using native code libigraph in the background.

", "votes": null, "replies": [] }, { "id": 11779, "author_name": "davidhysom", "author_url": "", "post_date": "06/21/2012 02:32:51", "content": "

I don't see too much downside, except it takes a bit longer to write the code. Well, you do lose the ability -- if you want that -- to easily explore the graph interactively.  Anyway, it took me only a few minutes to code in c++, and I can read the graph\r\n in just under 3 seconds.  This is on a fairly new linux box with a few G of RAM.

", "votes": null, "replies": [] }, { "id": 11781, "author_name": "macsdev", "author_url": "", "post_date": "06/21/2012 03:33:09", "content": "

Thanks for all the replies.

\r\n

Hi David,

\r\n

The more complex my algorithm is getting, the more I feel that I need to move away from Python. It's becoming extremely slow handling such a big graph. What datastructure are you using? Are you using Boost library?

", "votes": null, "replies": [] }, { "id": 11815, "author_name": "davidhysom", "author_url": "", "post_date": "06/22/2012 03:08:31", "content": "

I use STL in c++. My representation of the graph: map<int, set<int> > which maps

\r\n

{person -> set of people the person is following}

\r\n

I've used boost before, and know it is well thought of, but personally find their documentation --esp. for the graph library -- abysmal.  Some day I'll have enough free time to master it (perhaps).

\r\n

I *really* like python for quick prototyping, but when performance becomes an issue I graduate the code to C++. My experience with pickle is: for anything very large it's *very* slow.

\r\n

 

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "11698": "", "11701": "", "11706": "", "11711": "", "11745": "", "11779": "", "11781": "", "11815": "" }, "source": "meta" }