{ "id": 5206, "title": "PostgreSQL script to load data into a database", "url": "/competitions/accelerometer-biometric-competition/discussion/5206", "author_name": "", "post_date": "2013-07-26T16:56:36.893Z", "votes": 5, "comment_count": 8, "views": 3935, "content": "

With a few minor changes you can adapt this code for any other SQL db such as MS SQL Server, MySQL etc...

\n

/*load training data*/
CREATE TABLE rawtrain
(
T float
, X float
, Y float
, Z float
, Device int
);

\n

copy rawtrain from 'D:/Accel/train.csv' CSV HEADER;
CREATE INDEX Device_idx ON RAWTRAIN USING BTREE(Device ASC);

\n

/*load testing data*/
CREATE TABLE rawtest
(
T float
, X float
, Y float
, Z float
, SequenceId int
);

\n

copy rawtest from 'D:/Accel/test.csv' CSV HEADER;
CREATE INDEX SequenceId_idx ON RAWtest USING BTREE(SequenceId ASC);

\n

/*load questions file*/
CREATE TABLE questions
(
QuestionId int
, SequenceId int
, QuizDevice int
);
copy questions from 'D:/Accel/questions.csv' CSV HEADER;
CREATE INDEX SequenceId_idx2 ON questions USING BTREE(SequenceId ASC);
CREATE INDEX QuizDevice_idx ON questions USING BTREE(QuizDevice ASC);

", "messages": [ { "id": "27694", "postDate": "07/26/2013 16:56:36", "content": "

With a few minor changes you can adapt this code for any other SQL db such as MS SQL Server, MySQL etc...

\n

/*load training data*/
CREATE TABLE rawtrain
(
T float
, X float
, Y float
, Z float
, Device int
);

\n

copy rawtrain from 'D:/Accel/train.csv' CSV HEADER;
CREATE INDEX Device_idx ON RAWTRAIN USING BTREE(Device ASC);

\n

/*load testing data*/
CREATE TABLE rawtest
(
T float
, X float
, Y float
, Z float
, SequenceId int
);

\n

copy rawtest from 'D:/Accel/test.csv' CSV HEADER;
CREATE INDEX SequenceId_idx ON RAWtest USING BTREE(SequenceId ASC);

\n

/*load questions file*/
CREATE TABLE questions
(
QuestionId int
, SequenceId int
, QuizDevice int
);
copy questions from 'D:/Accel/questions.csv' CSV HEADER;
CREATE INDEX SequenceId_idx2 ON questions USING BTREE(SequenceId ASC);
CREATE INDEX QuizDevice_idx ON questions USING BTREE(QuizDevice ASC);

", "rawMarkdown": "", "votes": null }, { "id": "27740", "postDate": "07/29/2013 10:06:46", "content": "

Any suggestion how to do this in R? I am stuck in this. Data set are too big to read in R. 

", "rawMarkdown": "", "votes": null }, { "id": "27741", "postDate": "07/29/2013 12:25:47", "content": "

[quote=Bilal Nizami;27740]

\n

Any suggestion how to do this in R? I am stuck in this. Data set are too big to read in R. 

\n

[/quote]

\n

If read.csv function did not work for you because of memory issues you may want to use big memory packages. Search for Big Memory & ff packages.

\n

 

\n

Ps: I'm using R too - but I use a SQL db as a backend to store the competition data and only query the data that I need to work on from the DB in order to keep the memory footprint low.

", "rawMarkdown": "", "votes": null }, { "id": "28563", "postDate": "08/11/2013 21:43:49", "content": "

So when you say that you use an SQL db as a backend are you connecting the db with R or do you use the db to get the relevant data and then import that in R. Essentially - is there an R library that let's us connect with an SQL db and write "sql type" code to extract and load a subset of data directly in R?

\n

 

\n

Thanks

", "rawMarkdown": "", "votes": null }, { "id": "28564", "postDate": "08/11/2013 22:34:56", "content": "

Yes, packages such as RODBC (a generic package to connect to most SQL dbs via ODBC drivers) and db specific packages such as RMySQL, RPostgreSQL, teradataR do exist that allow you to connect to a SQL db and write sql queries against. Most return the resultset as a data frame to R.

", "rawMarkdown": "", "votes": null }, { "id": "28584", "postDate": "08/12/2013 14:40:58", "content": "

Here's an SQLite version (meant to be run in the sqlite3 command line utility):

\n

 

\n

/*tables match existing csv structure*/
CREATE TABLE questions (QuestionId INTEGER, SequenceId INTEGER, QuizDevice INTEGER);
CREATE TABLE test (T INTEGER, X REAL, Y REAL, Z REAL, SequenceId INTEGER);
CREATE TABLE train (T INTEGER, X REAL, Y REAL, Z REAL, Device INTEGER);

\n

/*import data*/
.mode csv
.import questions.csv questions
.import test.csv test
.import train.csv train

\n

/*remove redundant header leftover from csv*/
DELETE FROM questions WHERE QuestionId = 'QuestionId';
DELETE FROM test WHERE T = 'T';
DELETE FROM train WHERE T = 'T';

\n

/*index for speed*/
CREATE INDEX idx_questions_SequenceId ON questions(SequenceId);
CREATE INDEX idx_questions_QuizDevice ON questions(QuizDevice);
CREATE INDEX idx_test_SequenceId ON test(SequenceId);
CREATE INDEX idx_train_Device ON train(Device);

", "rawMarkdown": "", "votes": null }, { "id": "29238", "postDate": "08/22/2013 15:34:35", "content": "

I'm trying to use RMySQL to read data from the Test and Training sets that I've uploaded on an sql server. With the following query I'm trying to see if every block of 300 consecutive records in Test actually has the same sequence ID. (I know it says it does but I'm still checking):

\n

sequences = 0
for (i in 0:90023)
{
pos = i*300+1
check_query = paste('select * from TestData limit 300 offset ',pos, sep='')
check_test = dbGetQuery(con, check_query)    ##con is the SQL connection
sequences = c(sequences,unique(check_test$SeqID))
print(i) ## prints the block being printed
}

\n

What I notice is that it starts off rather fast but then the reading of the test data really slows down as 'i' starts increasing. It becomes less then 1 block/second as 'i' approaches 19,000.

\n

Now, if my code has run for the first 19400 blocks and I rerun it beginning from 19000 then it runs really fast for the first 400 blocks and then slows down again. This seems like R stores the results of dbGetQuery and is, hence, able to retrieve them really fast. Does anybody know if I can flush this memory since I just need the variable sequences? Can that help in speeding the code?

\n

Please let me know if this question doesn't make sense.

\n

 

\n

Thanks

\n

 

\n

 

", "rawMarkdown": "", "votes": null }, { "id": "29397", "postDate": "08/24/2013 20:32:50", "content": "

Street Hawk, you can look into rm() and gc(). These functions delete variables and force garbage collection to occur.

\n

Best,

\n

jon

", "rawMarkdown": "", "votes": null }, { "id": "30213", "postDate": "09/05/2013 08:14:21", "content": "

It is said that offset + limit is inefficient for retrieving data trunk by trunk. A better way to retrieve data by parts is to use cursor + fetch (http://www.postgresql.org/docs/9.2/static/plpgsql-cursors.html).

\n

 

\n

 

", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 27740, "author_name": "bilalnizami", "author_url": "", "post_date": "07/29/2013 10:06:46", "content": "

Any suggestion how to do this in R? I am stuck in this. Data set are too big to read in R. 

", "votes": null, "replies": [] }, { "id": 27741, "author_name": "sashikanthdareddy", "author_url": "", "post_date": "07/29/2013 12:25:47", "content": "

[quote=Bilal Nizami;27740]

\n

Any suggestion how to do this in R? I am stuck in this. Data set are too big to read in R. 

\n

[/quote]

\n

If read.csv function did not work for you because of memory issues you may want to use big memory packages. Search for Big Memory & ff packages.

\n

 

\n

Ps: I'm using R too - but I use a SQL db as a backend to store the competition data and only query the data that I need to work on from the DB in order to keep the memory footprint low.

", "votes": null, "replies": [] }, { "id": 28563, "author_name": "kunalrajani", "author_url": "", "post_date": "08/11/2013 21:43:49", "content": "

So when you say that you use an SQL db as a backend are you connecting the db with R or do you use the db to get the relevant data and then import that in R. Essentially - is there an R library that let's us connect with an SQL db and write "sql type" code to extract and load a subset of data directly in R?

\n

 

\n

Thanks

", "votes": null, "replies": [] }, { "id": 28564, "author_name": "sashikanthdareddy", "author_url": "", "post_date": "08/11/2013 22:34:56", "content": "

Yes, packages such as RODBC (a generic package to connect to most SQL dbs via ODBC drivers) and db specific packages such as RMySQL, RPostgreSQL, teradataR do exist that allow you to connect to a SQL db and write sql queries against. Most return the resultset as a data frame to R.

", "votes": null, "replies": [] }, { "id": 28584, "author_name": "megashark", "author_url": "", "post_date": "08/12/2013 14:40:58", "content": "

Here's an SQLite version (meant to be run in the sqlite3 command line utility):

\n

 

\n

/*tables match existing csv structure*/
CREATE TABLE questions (QuestionId INTEGER, SequenceId INTEGER, QuizDevice INTEGER);
CREATE TABLE test (T INTEGER, X REAL, Y REAL, Z REAL, SequenceId INTEGER);
CREATE TABLE train (T INTEGER, X REAL, Y REAL, Z REAL, Device INTEGER);

\n

/*import data*/
.mode csv
.import questions.csv questions
.import test.csv test
.import train.csv train

\n

/*remove redundant header leftover from csv*/
DELETE FROM questions WHERE QuestionId = 'QuestionId';
DELETE FROM test WHERE T = 'T';
DELETE FROM train WHERE T = 'T';

\n

/*index for speed*/
CREATE INDEX idx_questions_SequenceId ON questions(SequenceId);
CREATE INDEX idx_questions_QuizDevice ON questions(QuizDevice);
CREATE INDEX idx_test_SequenceId ON test(SequenceId);
CREATE INDEX idx_train_Device ON train(Device);

", "votes": null, "replies": [] }, { "id": 29238, "author_name": "kunalrajani", "author_url": "", "post_date": "08/22/2013 15:34:35", "content": "

I'm trying to use RMySQL to read data from the Test and Training sets that I've uploaded on an sql server. With the following query I'm trying to see if every block of 300 consecutive records in Test actually has the same sequence ID. (I know it says it does but I'm still checking):

\n

sequences = 0
for (i in 0:90023)
{
pos = i*300+1
check_query = paste('select * from TestData limit 300 offset ',pos, sep='')
check_test = dbGetQuery(con, check_query)    ##con is the SQL connection
sequences = c(sequences,unique(check_test$SeqID))
print(i) ## prints the block being printed
}

\n

What I notice is that it starts off rather fast but then the reading of the test data really slows down as 'i' starts increasing. It becomes less then 1 block/second as 'i' approaches 19,000.

\n

Now, if my code has run for the first 19400 blocks and I rerun it beginning from 19000 then it runs really fast for the first 400 blocks and then slows down again. This seems like R stores the results of dbGetQuery and is, hence, able to retrieve them really fast. Does anybody know if I can flush this memory since I just need the variable sequences? Can that help in speeding the code?

\n

Please let me know if this question doesn't make sense.

\n

 

\n

Thanks

\n

 

\n

 

", "votes": null, "replies": [] }, { "id": 29397, "author_name": "jondavis", "author_url": "", "post_date": "08/24/2013 20:32:50", "content": "

Street Hawk, you can look into rm() and gc(). These functions delete variables and force garbage collection to occur.

\n

Best,

\n

jon

", "votes": null, "replies": [] }, { "id": 30213, "author_name": "bwpanda", "author_url": "", "post_date": "09/05/2013 08:14:21", "content": "

It is said that offset + limit is inefficient for retrieving data trunk by trunk. A better way to retrieve data by parts is to use cursor + fetch (http://www.postgresql.org/docs/9.2/static/plpgsql-cursors.html).

\n

 

\n

 

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "27694": "", "27740": "", "27741": "", "28563": "", "28564": "", "28584": "", "29238": "", "29397": "", "30213": "" }, "source": "meta" }