{ "id": 5606, "title": "SQL Script to load data into PostgreSQL DB", "url": "/competitions/facebook-recruiting-iii-keyword-extraction/discussion/5606", "author_name": "Sashikanth Dareddy", "post_date": "2013-08-31T21:30:56.190000", "votes": 17, "comment_count": 14, "views": 7559, "content": "

I find PostgreSQL to be extremely versatile to manipulate data ( albeit at a slower speed) so, I've loaded this data using the following script without any issues. I've also posted summary stats here: http://www.kaggle.com/c/facebook-recruiting-iii-keyword-extraction/forums/t/5605/extended-summary-stats-for-the-data-provided

\n

CREATE TABLE RAW_TRAIN(
ID BIGINT PRIMARY KEY
,TITLE TEXT
,BODY TEXT
,TAGS TEXT
);
copy RAW_TRAIN from 'D:\\\\FacebookIII\\\\Data\\\\Train.csv' csv header;
/*Query returned successfully: 6034195 rows affected, 1072534 ms execution time.*/

\n

 

\n

CREATE TABLE RAW_TEST(
ID BIGINT PRIMARY KEY
,TITLE TEXT
,BODY TEXT
);
copy RAW_TEST from 'D:\\\\FacebookIII\\\\Data\\\\Test.csv' csv header;
/*Query returned successfully: 2013337 rows affected, 116390 ms execution time.*/

", "messages": [ { "id": 29868, "postDate": "2013-08-31T21:30:56.190Z", "content": "

I find PostgreSQL to be extremely versatile to manipulate data ( albeit at a slower speed) so, I've loaded this data using the following script without any issues. I've also posted summary stats here: http://www.kaggle.com/c/facebook-recruiting-iii-keyword-extraction/forums/t/5605/extended-summary-stats-for-the-data-provided

\n

CREATE TABLE RAW_TRAIN(
ID BIGINT PRIMARY KEY
,TITLE TEXT
,BODY TEXT
,TAGS TEXT
);
copy RAW_TRAIN from 'D:\\\\FacebookIII\\\\Data\\\\Train.csv' csv header;
/*Query returned successfully: 6034195 rows affected, 1072534 ms execution time.*/

\n

 

\n

CREATE TABLE RAW_TEST(
ID BIGINT PRIMARY KEY
,TITLE TEXT
,BODY TEXT
);
copy RAW_TEST from 'D:\\\\FacebookIII\\\\Data\\\\Test.csv' csv header;
/*Query returned successfully: 2013337 rows affected, 116390 ms execution time.*/

", "votes": 17 }, { "id": 35266, "postDate": "2013-11-26T12:58:05.867Z", "content": "

select sum(array_length(regexp_split_to_array(TAGS, '\\s'),1)) from TRAINDATA;

", "votes": 3 }, { "id": 36244, "postDate": "2013-12-17T09:36:47.803Z", "content": "

I am late to the party. Hope the dataset is alive even after the contest ends. Following is my java code for parsing if somebody is trying in java (now!!).

\n

-------------------------------------------------

\n

public class IssueInstance {

\n

public int id;
public String heading="";
public String body="";
public String tags="";
}

\n

---------------------------------------------------

\n

public static String separateInstances() {

\n

LinkedList<IssueInstance> instances = new LinkedList<IssueInstance>();

\n

try {
// Open the file
FileInputStream fstream = new FileInputStream(pathName);
BufferedReader br = new BufferedReader(new InputStreamReader(
fstream));
int count = 0;

String strLine;
String strLineprev = "";

\n

String Body = "";

\n

strLine = br.readLine();
System.out.println(strLine);// "Id","Title","Body","Tags"

\n

while ((strLine = br.readLine()) != null) {
 
instances.add(new IssueInstance());
if (strLine.startsWith("\\"") == true
&& strLine.startsWith("\\",") == false
&& strLine.startsWith("\\"\\"") == false) {

\n

// System.out.println(strLineprev.split("\\"")[strLineprev.split("\\"").length-1]);

\n

if (count > 0) {
instances.get(count - 1).tags = strLineprev.split("\\"")[strLineprev
.split("\\"").length - 1];
instances.get(count - 1).body = Body
.replace("\\"" + instances.get(count - 1).id, "")
.replace(instances.get(count - 1).heading, "")
.replace(instances.get(count - 1).tags, "")
.replaceAll("\\".\\"", "");
Body = "";
}
// System.out.println(strLineprev.split("\\"")[strLineprev.split("\\"").length-1]);
// System.out.println(count);

\n

instances.get(count).id = count + 1;

\n

// System.out.println(strLine.split("\\",\\"")[1]);
instances.get(count).heading = strLine.split("\\",\\"")[1];
count++;

\n

} else
strLineprev = strLine;

\n


{
Body = Body + " " + strLine;
}

}

\n

 
// Close the input stream
br.close();
 

\n

} catch (Exception e) {
System.out.println(e.getMessage());
}

\n

return "";
}

", "votes": 1 }, { "id": 35265, "postDate": "2013-11-26T12:48:34.170Z", "content": "

Just wondering, how did you calculate the number of unique tags from the RAW_TRAIN table? Each row for TAGS column is a space separated string. Did you create another table to keep each tag separately? 

", "votes": 1 }, { "id": 35082, "postDate": "2013-11-23T10:52:44.520Z", "content": "

[quote=venkat ram reddy kunta;35071]

\n

How to skip the header of the csv file ?

\n

because when we are copying data from csv file to table which has integer values throw error. right?

\n

 

\n

[/quote]

\n

Notice the "Header" option in the "Copy from" statement. This is what skips the header.

\n

copy RAW_TRAIN from 'D:\\\\FacebookIII\\\\Data\\\\Train.csv' csv header;

", "votes": 1 }, { "id": 35335, "postDate": "2013-11-27T04:24:57.653Z", "content": "

You can update your path is Environment Variables if you think the path is the issue.

" }, { "id": 35324, "postDate": "2013-11-27T00:19:44.973Z", "content": "

I have installed PostgreSQL and PSYCOPG, for connecting PostgreSQL to Python and retrieve data from it.  And I am able to retrieve data successfully.

\n

Then I was using NLTK to perform some operations on the data.

\n

I got the following error: 

\n

"ImportError: No module named nltk" .

\n

So I tried to install NLTK again using the URL: http://nltk.org/install.html. 

\n

Output message:

\n

"Requirement already up-to-date: pyyaml in /Library/Python/2.7/site-packages
Requirement already up-to-date: nltk in /Library/Python/2.7/site-packages
Cleaning up... "

\n

this means that NLTK is already installed and up to date.

\n

but i cannot figure out the problem. May be during the installation of PSYCOPG ; PYTHON path has changed? 

\n

Please help me out !

" }, { "id": 35268, "postDate": "2013-11-26T13:14:30.853Z", "content": "

Thanks! That's very cool. I am becoming a fan of PostgreSQL :D

\n

 

\n

This gives the number of occurrences of tags. But still pretty cool to get it in one query.

" }, { "id": 35071, "postDate": "2013-11-23T00:20:37.007Z", "content": "

How to skip the header of the csv file ?

\n

because when we are copying data from csv file to table which has integer values throw error. right?

\n

 

" }, { "id": 32009, "postDate": "2013-10-04T02:15:20.327Z", "content": "

Thanks Thats nice .I have just installed postgre sql and started working looks nice that it can handle large data sources

" }, { "id": 31994, "postDate": "2013-10-03T17:43:17.067Z", "content": "

Hi, you may have overlooked but the execution time (in milliseconds) is right beneath the COPY statement in my original post.

" }, { "id": 31993, "postDate": "2013-10-03T17:35:50.060Z", "content": "

Hi sashikanth 

\n

 

\n

how log did it take to load the data in postgresql db ?

" }, { "id": 30090, "postDate": "2013-09-03T20:29:06.543Z", "content": "

If somebody was interested, after reading the raw data with the script above, the folder with the table takes 5,85GB.

\n

Just for comparison

\n

 

\n

Query returned successfully: 6034195 rows affected, 300542 ms execution time.

\n

but that is due to the SSD drive I have. It could be faster if I had better faster SATA in my laptop.

" }, { "id": 29901, "postDate": "2013-09-01T08:10:58.013Z", "content": "

Brilliant, thanks mate! Keeping it simple saved some time. 

" }, { "id": 30550, "postDate": "2013-09-09T17:42:00.440Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 35266, "author_name": "Abhishek Thakur", "author_url": "", "post_date": "2013-11-26T12:58:05.867000", "content": "

select sum(array_length(regexp_split_to_array(TAGS, '\\s'),1)) from TRAINDATA;

", "votes": 3, "replies": [] }, { "id": 36244, "author_name": "Mensch", "author_url": "", "post_date": "2013-12-17T09:36:47.803000", "content": "

I am late to the party. Hope the dataset is alive even after the contest ends. Following is my java code for parsing if somebody is trying in java (now!!).

\n

-------------------------------------------------

\n

public class IssueInstance {

\n

public int id;
public String heading="";
public String body="";
public String tags="";
}

\n

---------------------------------------------------

\n

public static String separateInstances() {

\n

LinkedList<IssueInstance> instances = new LinkedList<IssueInstance>();

\n

try {
// Open the file
FileInputStream fstream = new FileInputStream(pathName);
BufferedReader br = new BufferedReader(new InputStreamReader(
fstream));
int count = 0;

String strLine;
String strLineprev = "";

\n

String Body = "";

\n

strLine = br.readLine();
System.out.println(strLine);// "Id","Title","Body","Tags"

\n

while ((strLine = br.readLine()) != null) {
 
instances.add(new IssueInstance());
if (strLine.startsWith("\\"") == true
&& strLine.startsWith("\\",") == false
&& strLine.startsWith("\\"\\"") == false) {

\n

// System.out.println(strLineprev.split("\\"")[strLineprev.split("\\"").length-1]);

\n

if (count > 0) {
instances.get(count - 1).tags = strLineprev.split("\\"")[strLineprev
.split("\\"").length - 1];
instances.get(count - 1).body = Body
.replace("\\"" + instances.get(count - 1).id, "")
.replace(instances.get(count - 1).heading, "")
.replace(instances.get(count - 1).tags, "")
.replaceAll("\\".\\"", "");
Body = "";
}
// System.out.println(strLineprev.split("\\"")[strLineprev.split("\\"").length-1]);
// System.out.println(count);

\n

instances.get(count).id = count + 1;

\n

// System.out.println(strLine.split("\\",\\"")[1]);
instances.get(count).heading = strLine.split("\\",\\"")[1];
count++;

\n

} else
strLineprev = strLine;

\n


{
Body = Body + " " + strLine;
}

}

\n

 
// Close the input stream
br.close();
 

\n

} catch (Exception e) {
System.out.println(e.getMessage());
}

\n

return "";
}

", "votes": 1, "replies": [] }, { "id": 35265, "author_name": "Rakebul Hasan", "author_url": "", "post_date": "2013-11-26T12:48:34.170000", "content": "

Just wondering, how did you calculate the number of unique tags from the RAW_TRAIN table? Each row for TAGS column is a space separated string. Did you create another table to keep each tag separately? 

", "votes": 1, "replies": [] }, { "id": 35082, "author_name": "Sashikanth Dareddy", "author_url": "", "post_date": "2013-11-23T10:52:44.520000", "content": "

[quote=venkat ram reddy kunta;35071]

\n

How to skip the header of the csv file ?

\n

because when we are copying data from csv file to table which has integer values throw error. right?

\n

 

\n

[/quote]

\n

Notice the "Header" option in the "Copy from" statement. This is what skips the header.

\n

copy RAW_TRAIN from 'D:\\\\FacebookIII\\\\Data\\\\Train.csv' csv header;

", "votes": 1, "replies": [] }, { "id": 35335, "author_name": "Nikhil Agarwal", "author_url": "", "post_date": "2013-11-27T04:24:57.653000", "content": "

You can update your path is Environment Variables if you think the path is the issue.

", "votes": 0, "replies": [] }, { "id": 35324, "author_name": "Raghu Nandan Immaneni", "author_url": "", "post_date": "2013-11-27T00:19:44.973000", "content": "

I have installed PostgreSQL and PSYCOPG, for connecting PostgreSQL to Python and retrieve data from it.  And I am able to retrieve data successfully.

\n

Then I was using NLTK to perform some operations on the data.

\n

I got the following error: 

\n

"ImportError: No module named nltk" .

\n

So I tried to install NLTK again using the URL: http://nltk.org/install.html. 

\n

Output message:

\n

"Requirement already up-to-date: pyyaml in /Library/Python/2.7/site-packages
Requirement already up-to-date: nltk in /Library/Python/2.7/site-packages
Cleaning up... "

\n

this means that NLTK is already installed and up to date.

\n

but i cannot figure out the problem. May be during the installation of PSYCOPG ; PYTHON path has changed? 

\n

Please help me out !

", "votes": 0, "replies": [] }, { "id": 35268, "author_name": "Rakebul Hasan", "author_url": "", "post_date": "2013-11-26T13:14:30.853000", "content": "

Thanks! That's very cool. I am becoming a fan of PostgreSQL :D

\n

 

\n

This gives the number of occurrences of tags. But still pretty cool to get it in one query.

", "votes": 0, "replies": [] }, { "id": 35071, "author_name": "Raghu Nandan Immaneni", "author_url": "", "post_date": "2013-11-23T00:20:37.007000", "content": "

How to skip the header of the csv file ?

\n

because when we are copying data from csv file to table which has integer values throw error. right?

\n

 

", "votes": 0, "replies": [] }, { "id": 32009, "author_name": "", "author_url": "", "post_date": "2013-10-04T02:15:20.327000", "content": "

Thanks Thats nice .I have just installed postgre sql and started working looks nice that it can handle large data sources

", "votes": 0, "replies": [] }, { "id": 31994, "author_name": "Sashikanth Dareddy", "author_url": "", "post_date": "2013-10-03T17:43:17.067000", "content": "

Hi, you may have overlooked but the execution time (in milliseconds) is right beneath the COPY statement in my original post.

", "votes": 0, "replies": [] }, { "id": 31993, "author_name": "", "author_url": "", "post_date": "2013-10-03T17:35:50.060000", "content": "

Hi sashikanth 

\n

 

\n

how log did it take to load the data in postgresql db ?

", "votes": 0, "replies": [] }, { "id": 30090, "author_name": "Marcin Pękalski", "author_url": "", "post_date": "2013-09-03T20:29:06.543000", "content": "

If somebody was interested, after reading the raw data with the script above, the folder with the table takes 5,85GB.

\n

Just for comparison

\n

 

\n

Query returned successfully: 6034195 rows affected, 300542 ms execution time.

\n

but that is due to the SSD drive I have. It could be faster if I had better faster SATA in my laptop.

", "votes": 0, "replies": [] }, { "id": 29901, "author_name": "", "author_url": "", "post_date": "2013-09-01T08:10:58.013000", "content": "

Brilliant, thanks mate! Keeping it simple saved some time. 

", "votes": 0, "replies": [] }, { "id": 30550, "author_name": "", "author_url": "", "post_date": "2013-09-09T17:42:00.440000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "29868": "", "35266": "", "36244": "", "35265": "", "35082": "", "35335": "", "35324": "", "35268": "", "35071": "", "32009": "", "31994": "", "31993": "", "30090": "", "29901": "", "30550": "" } }