Ok, so I loaded the data into a mysql database, ran a simple select statement i.e. select * where source_node = 24;
\r\nit takes roughly 1 min to run!!!! If I have to do this for +- 200000 records it'll take 300 days.
\r\nAre there ways to speed up the db querying (first thing in my mind was to put the database on a db hosting site but I'm not sure that'll make it any quicker)
", "messages": [ { "id": "11511", "postDate": "06/10/2012 19:23:13", "content": "Ok, so I loaded the data into a mysql database, ran a simple select statement i.e. select * where source_node = 24;
\r\nit takes roughly 1 min to run!!!! If I have to do this for +- 200000 records it'll take 300 days.
\r\nAre there ways to speed up the db querying (first thing in my mind was to put the database on a db hosting site but I'm not sure that'll make it any quicker)
", "rawMarkdown": "", "votes": null }, { "id": "11512", "postDate": "06/10/2012 19:40:38", "content": "add index on both columns?
", "rawMarkdown": "", "votes": null }, { "id": "11513", "postDate": "06/10/2012 20:08:03", "content": "I truly believe putting it into a DB will kill performance... you can optimize it all you want but I dont think a DB is the solution
\r\nYou should really try to write a program where you can manually manage memory as it will reduce the overhead that a DB has
", "rawMarkdown": "", "votes": null }, { "id": "11514", "postDate": "06/10/2012 21:39:45", "content": "[quote=osphere;11513]
\r\nI truly believe putting it into a DB will kill performance... you can optimize it all you want but I dont think a DB is the solution
\r\nYou should really try to write a program where you can manually manage memory as it will reduce the overhead that a DB has
\r\n[/quote]
\r\nDo you think that Facebook does not use databases to "reduce the overhead" as you put it?
", "rawMarkdown": "", "votes": null }, { "id": "11515", "postDate": "06/10/2012 22:46:51", "content": "There are noSQL databases to deal with huge information. And about that time for executing a queue.. maybe you need to put an index on the node columns.\r\n
\r\nBut i dont think that we need an db for a dataset this size...
[quote=Momchil Georgiev;11514]
\r\n[quote=osphere;11513]
\r\nI truly believe putting it into a DB will kill performance... you can optimize it all you want but I dont think a DB is the solution
\r\nYou should really try to write a program where you can manually manage memory as it will reduce the overhead that a DB has
\r\n[/quote]
\r\nDo you think that Facebook does not use databases to "reduce the overhead" as you put it?
\r\n[/quote]
\r\n\r\n
They do use databases in facebook, but remember: you gotta use the right tool for the job
\r\nMySQL or any other relational database has a lot of overhead to maintain ACID, which may be right for some jobs, but not this one.
\r\nnoSQL databases as Leustagos said is like.. having all your dataset in RAM which is similar to writing your own program where you can manage memory management and access. Which may be helpful for this job, but again: having your data inside a database has\r\n a some overhead (maybe the overhead can be neglected on a noSQLdatabase, I have no idea, however it exists, thats for sure)
\r\nI suggest to djmachet to try using a different solution than MySQL, thats all
", "rawMarkdown": "", "votes": null }, { "id": "11523", "postDate": "06/11/2012 15:59:41", "content": "Create indexes on both columns.
\r\nI put it on a mysql table and the same query takes 118ms on my laptop (2.5GHz intel mac)
", "rawMarkdown": "", "votes": null }, { "id": "11528", "postDate": "06/11/2012 18:03:06", "content": "I generate an adjacency matrix eta with
\r\neta(i,j)=1 if there is an edge from i to j, 0 otherwise
\r\nI know that this method is the worst in terms of memory consumption, but it is the fastest to look up whether there is an edge or not.
You could generate an adjacency list using hashmaps and hashsets... space would be a little worst than pure list of adjacencies but lookup will be O(1).\r\n
\r\nI'm using it.
I created myself some little helpers. I have two hash maps (dictionaries in python). The first has a node as a key and list of vertices to which it points as its values. The second has a node as a key and a list of vertices which point to it. The latter\r\n is useful if interested in following the edges backwards quickly. Looking up a node is pretty much instantly on my laptop.
", "rawMarkdown": "", "votes": null }, { "id": "11536", "postDate": "06/11/2012 22:44:08", "content": "Don't use a db for this. A python dictionary is more than adequate. However, if you do start to create attributes for a node, then maybe a db would be handy, but use sqlite with loads of indexes.
\r\nStill, Christian has the right idea. Spend a bit of time creating dictionaries (of A to B and B to A) upfront and then pickle them to the hard drive. That way you can get access to them quickly.\r\n
[quote=Leustagos;11533]
\r\nYou could generate an adjacency list using hashmaps and hashsets... space would be a little worst than pure list of adjacencies but lookup will be O(1).\r\n
\r\nI'm using it.
[/quote]
\r\nAs a small variation to this approach, you can get O(1) lookup with a smaller constant than a hash table by replacing the use of the hash table with an array. This lets you directly index into the array using the node index, at the cost of potentially wasting\r\n some memory if many of the indices smaller than max_node_index correspond to nodes that don't exist or have no associated values.
", "rawMarkdown": "", "votes": null }, { "id": "11685", "postDate": "06/17/2012 14:04:47", "content": "djmachet, no need for a DB due to its overhead. A 'awk' command is acceptable enough for your query 'select * where source_node = 24;' :D
\r\n$ time awk -F, '$1==24{print}' train.csv
\r\n24,322832
\r\n24,423929
\r\n24,1435
\r\n24,1310872
real 0m16.000s
\r\nuser 0m15.718s
\r\nsys 0m0.202s
add index on both columns?
", "votes": null, "replies": [] }, { "id": 11513, "author_name": "osphere", "author_url": "", "post_date": "06/10/2012 20:08:03", "content": "I truly believe putting it into a DB will kill performance... you can optimize it all you want but I dont think a DB is the solution
\r\nYou should really try to write a program where you can manually manage memory as it will reduce the overhead that a DB has
", "votes": null, "replies": [] }, { "id": 11514, "author_name": "sirguessalot", "author_url": "", "post_date": "06/10/2012 21:39:45", "content": "[quote=osphere;11513]
\r\nI truly believe putting it into a DB will kill performance... you can optimize it all you want but I dont think a DB is the solution
\r\nYou should really try to write a program where you can manually manage memory as it will reduce the overhead that a DB has
\r\n[/quote]
\r\nDo you think that Facebook does not use databases to "reduce the overhead" as you put it?
", "votes": null, "replies": [] }, { "id": 11515, "author_name": "leustagos", "author_url": "", "post_date": "06/10/2012 22:46:51", "content": "There are noSQL databases to deal with huge information. And about that time for executing a queue.. maybe you need to put an index on the node columns.\r\n
\r\nBut i dont think that we need an db for a dataset this size...
[quote=Momchil Georgiev;11514]
\r\n[quote=osphere;11513]
\r\nI truly believe putting it into a DB will kill performance... you can optimize it all you want but I dont think a DB is the solution
\r\nYou should really try to write a program where you can manually manage memory as it will reduce the overhead that a DB has
\r\n[/quote]
\r\nDo you think that Facebook does not use databases to "reduce the overhead" as you put it?
\r\n[/quote]
\r\n\r\n
They do use databases in facebook, but remember: you gotta use the right tool for the job
\r\nMySQL or any other relational database has a lot of overhead to maintain ACID, which may be right for some jobs, but not this one.
\r\nnoSQL databases as Leustagos said is like.. having all your dataset in RAM which is similar to writing your own program where you can manage memory management and access. Which may be helpful for this job, but again: having your data inside a database has\r\n a some overhead (maybe the overhead can be neglected on a noSQLdatabase, I have no idea, however it exists, thats for sure)
\r\nI suggest to djmachet to try using a different solution than MySQL, thats all
", "votes": null, "replies": [] }, { "id": 11523, "author_name": "esteban1", "author_url": "", "post_date": "06/11/2012 15:59:41", "content": "Create indexes on both columns.
\r\nI put it on a mysql table and the same query takes 118ms on my laptop (2.5GHz intel mac)
", "votes": null, "replies": [] }, { "id": 11528, "author_name": "commandercain", "author_url": "", "post_date": "06/11/2012 18:03:06", "content": "I generate an adjacency matrix eta with
\r\neta(i,j)=1 if there is an edge from i to j, 0 otherwise
\r\nI know that this method is the worst in terms of memory consumption, but it is the fastest to look up whether there is an edge or not.
You could generate an adjacency list using hashmaps and hashsets... space would be a little worst than pure list of adjacencies but lookup will be O(1).\r\n
\r\nI'm using it.
I created myself some little helpers. I have two hash maps (dictionaries in python). The first has a node as a key and list of vertices to which it points as its values. The second has a node as a key and a list of vertices which point to it. The latter\r\n is useful if interested in following the edges backwards quickly. Looking up a node is pretty much instantly on my laptop.
", "votes": null, "replies": [] }, { "id": 11536, "author_name": "diarmuidwrenne", "author_url": "", "post_date": "06/11/2012 22:44:08", "content": "Don't use a db for this. A python dictionary is more than adequate. However, if you do start to create attributes for a node, then maybe a db would be handy, but use sqlite with loads of indexes.
\r\nStill, Christian has the right idea. Spend a bit of time creating dictionaries (of A to B and B to A) upfront and then pickle them to the hard drive. That way you can get access to them quickly.\r\n
[quote=Leustagos;11533]
\r\nYou could generate an adjacency list using hashmaps and hashsets... space would be a little worst than pure list of adjacencies but lookup will be O(1).\r\n
\r\nI'm using it.
[/quote]
\r\nAs a small variation to this approach, you can get O(1) lookup with a smaller constant than a hash table by replacing the use of the hash table with an array. This lets you directly index into the array using the node index, at the cost of potentially wasting\r\n some memory if many of the indices smaller than max_node_index correspond to nodes that don't exist or have no associated values.
", "votes": null, "replies": [] }, { "id": 11685, "author_name": "nhanitvn", "author_url": "", "post_date": "06/17/2012 14:04:47", "content": "djmachet, no need for a DB due to its overhead. A 'awk' command is acceptable enough for your query 'select * where source_node = 24;' :D
\r\n$ time awk -F, '$1==24{print}' train.csv
\r\n24,322832
\r\n24,423929
\r\n24,1435
\r\n24,1310872
real 0m16.000s
\r\nuser 0m15.718s
\r\nsys 0m0.202s