I know all the cool kids are going to the Hadoop cluster, but this data is THAT big =)
\n\nAnyway it is big enough that I cannot load into memory--my first thought was putting it in a personal database--has anyone else thought along that line? I got my start in traditional SQL dbs so it seemed natural to just load the data up and even manipulate it using that method to take advantage of hard drive space, rather than memory.
\n\nAnyone else approaching it that way? I went postgresql and got the numeric data loaded pretty quick (a few minutes).
", "messages": [ { "id": "134063", "postDate": "09/04/2016 22:21:23", "content": "I know all the cool kids are going to the Hadoop cluster, but this data is THAT big =)
\n\nAnyway it is big enough that I cannot load into memory--my first thought was putting it in a personal database--has anyone else thought along that line? I got my start in traditional SQL dbs so it seemed natural to just load the data up and even manipulate it using that method to take advantage of hard drive space, rather than memory.
\n\nAnyone else approaching it that way? I went postgresql and got the numeric data loaded pretty quick (a few minutes).
", "rawMarkdown": "I know all the cool kids are going to the Hadoop cluster, but this data is THAT big =)\r\n\r\nAnyway it is big enough that I cannot load into memory--my first thought was putting it in a personal database--has anyone else thought along that line? I got my start in traditional SQL dbs so it seemed natural to just load the data up and even manipulate it using that method to take advantage of hard drive space, rather than memory.\r\n\r\n\r\nAnyone else approaching it that way? I went postgresql and got the numeric data loaded pretty quick (a few minutes).", "votes": null }, { "id": "135805", "postDate": "09/17/2016 05:05:00", "content": "I'm experimenting with that route as well.
", "rawMarkdown": "I'm experimenting with that route as well.", "votes": null }, { "id": "135844", "postDate": "09/17/2016 16:16:38", "content": "Since you can't use SQL to build models and make any kind of predictive analysis anyway, why not using data.table in R? It is capable of handling way bigger datasets. And in some sense it behaves similar to SQL (you can also make joins, for example).\nIt took me 1 minute on a laptop to read "test_numeric.csv".
\n\nrequire(data.table)\nrequire(bit64)\n\ntest = fread("test_numeric.csv")\n> test = fread("test_numeric.csv")\nRead 1183748 rows and 969 (of 969) columns from 1.991 GB file in 00:01:08\n(of 969) columns from 1.991 GB file in 00:01:08\n\n> dim(test)\n[1] 1183748 969\nMemory limits. I only have 8GB of memory.
", "rawMarkdown": "Memory limits. I only have 8GB of memory.", "votes": null }, { "id": "135944", "postDate": "09/18/2016 17:22:10", "content": "Hi,
\n\nI have used the same.I am getting the below error.
\n\ntest = fread("test_numeric.csv")
\n\nRead 0.0% of 1183748 rows
\n\nError: cannot allocate vector of size 9.0 Mb
\n\nIn addition: There were 20 warnings (use warnings() to see them)
\n\nCan someone help me to resolve this?
\n\nThank you.
", "rawMarkdown": "Hi, \r\n\r\nI have used the same.I am getting the below error. \r\n\r\ntest = fread(\"test_numeric.csv\")\r\n\r\nRead 0.0% of 1183748 rows\r\n\r\nError: cannot allocate vector of size 9.0 Mb\r\n\r\nIn addition: There were 20 warnings (use warnings() to see them)\r\n\r\n\r\n\r\nCan someone help me to resolve this?\r\n\r\n\r\nThank you.", "votes": null }, { "id": "135947", "postDate": "09/18/2016 17:45:41", "content": "Now,I got it.I changes to :
\n\nmemory.limit(size=10000)
\n\nRead 1183748 rows and 969 (of 969) columns from 1.991 GB file in 00:05:10
", "rawMarkdown": "Now,I got it.I changes to :\r\n\r\nmemory.limit(size=10000)\r\n\r\nRead 1183748 rows and 969 (of 969) columns from 1.991 GB file in 00:05:10", "votes": null }, { "id": "136175", "postDate": "09/20/2016 12:19:03", "content": "Here is the script I used to load the data into SQLight. In case you don't have enough RAM to execute it, you could play with nrows and skip in fread() to process data in chunks. For this to work, you might need to write an additional INSERT-statement and execute it with dbSendQuery. Better/faster approaches are welcome :)
\n\n# install/load packages\nif (!require("pacman")) install.packages("pacman")\npacman::p_load(data.table, RSQLite)\n\n# Set path and database name\npath <- "./data"\nsetwd(path)\nsqlite <- dbDriver("SQLite")\ncon <- dbConnect(sqlite, "20160920_kgl_Bosch.sqlite3")\n\n# Loop through all csv files in path and write them to SQLight\nfor (file in list.files(pattern = "*.csv")){\n print(Sys.time())\n print(file)\n data <- fread(file)\n tbl_name <- gsub(".csv","", file)\n\n if (dbExistsTable(con, tbl_name) == FALSE){\n dbWriteTable(con, tbl_name, data)\n }\n\n # Create an index on the Id-column to improve performance on later queries\n dbSendQuery(con, paste0("CREATE UNIQUE INDEX idx_", tbl_name, " ON ", tbl_name, "(Id)"))\n rm(data)\n gc()\n}\nWord of advise - don't use MYSQL. It has a hard column limit of ~4k columns (1000 columns with inno db)
", "rawMarkdown": "Word of advise - don't use MYSQL. It has a hard column limit of ~4k columns (1000 columns with inno db)", "votes": null }, { "id": "136560", "postDate": "09/22/2016 21:47:03", "content": "@Daniel_Kuehn,
\n\nCan you pls advise how did you change max.number of columns in sqlite? Did you recompile your sqlite?
\n\nThank you
", "rawMarkdown": "Daniel_Kuehn,\r\n\r\nCan you pls advise how did you change max.number of columns in sqlite? Did you recompile your sqlite?\r\n\r\nThank you", "votes": null }, { "id": "136613", "postDate": "09/23/2016 11:23:34", "content": "[quote=Chandrak1907;136560]
\n\n@Daniel_Kuehn,
\n\nCan you pls advise how did you change max.number of columns in sqlite? Did you recompile your sqlite?
\n\nThank you
\n\n[/quote]
\n\n Actually I didn't need to change max.number in this case, since I wrote each .csv file to a new table. SQLITE_MAX_COLUMN should be 2000 per table and non of the .csv files has more than 2k columns.\nI guess dbWriteTable handles cases with more than 2000 columns per table, since it works for me without any problem.
You can use joins on the Id column to retrieve all columns when you need them.\nE. g. try something like this to load the first 10k rows to R with all columns:
\n\ndbSendQuery(con, "SELECT * \nFROM train_categorical As c \n INNER JOIN train_date As d ON c.Id = d.Id\n INNER JOIN train_numeric AS n ON c.Id = n.Id\nLIMIT 10000")\nWhat I did was to normalize the database based on feature name to make the data-set "longer" By parsing the name you can get a Station Value, Line Value, and Feature Value --then each measurement is characterized by those 3 numbers (and even add in the date)
\n\nId Value L-Value S-Value F Value Date\n 4 0.1 1 1 5 187.2\n 7 T1 2 3 89 192.6\nOf course, my background is in Databases so this was a natural way to proceed...
\n\nThen to get things like all the unique values for L2_S3_F89:
\n\nSELECT COUNT(1), VALUE, S-VALUE, F-VALUE FROM MY_TABLE\nWHERE L-VALUE=2 AND S-VALUE=3 AND F-VALUE=89\nGROUP BY L-VALUE,S-VALUE, F-VALUE\n@Daniel_Kuehn,
\n\nThank you for replying.\nAre you sure that train_categorical.csv has less than 2000 columns. When I read in first 100 rows , I see 2141 columns.
", "rawMarkdown": "Daniel_Kuehn,\r\n\r\nThank you for replying.\r\nAre you sure that train_categorical.csv has less than 2000 columns. When I read in first 100 rows , I see 2141 columns.", "votes": null }, { "id": "136668", "postDate": "09/23/2016 20:12:04", "content": "@Chandrak1907
\n\nI see the same number and have seen that number tossed around as well.
\n\nThere was talk of some of the features not being in the test set (all NAN for that particular feature) but I do not know if that is enough to reduce below 2000.
", "rawMarkdown": "Chandrak1907\r\n\r\nI see the same number and have seen that number tossed around as well.\r\n\r\nThere was talk of some of the features not being in the test set (all NAN for that particular feature) but I do not know if that is enough to reduce below 2000.", "votes": null } ], "comments": [ { "id": 135805, "author_name": "datakungfu", "author_url": "", "post_date": "09/17/2016 05:05:00", "content": "I'm experimenting with that route as well.
", "votes": null, "replies": [] }, { "id": 135844, "author_name": "ievgenii1101", "author_url": "", "post_date": "09/17/2016 16:16:38", "content": "Since you can't use SQL to build models and make any kind of predictive analysis anyway, why not using data.table in R? It is capable of handling way bigger datasets. And in some sense it behaves similar to SQL (you can also make joins, for example).\nIt took me 1 minute on a laptop to read "test_numeric.csv".
\n\nrequire(data.table)\nrequire(bit64)\n\ntest = fread("test_numeric.csv")\n> test = fread("test_numeric.csv")\nRead 1183748 rows and 969 (of 969) columns from 1.991 GB file in 00:01:08\n(of 969) columns from 1.991 GB file in 00:01:08\n\n> dim(test)\n[1] 1183748 969\nMemory limits. I only have 8GB of memory.
", "votes": null, "replies": [] }, { "id": 135944, "author_name": "sandeepkumar", "author_url": "", "post_date": "09/18/2016 17:22:10", "content": "Hi,
\n\nI have used the same.I am getting the below error.
\n\ntest = fread("test_numeric.csv")
\n\nRead 0.0% of 1183748 rows
\n\nError: cannot allocate vector of size 9.0 Mb
\n\nIn addition: There were 20 warnings (use warnings() to see them)
\n\nCan someone help me to resolve this?
\n\nThank you.
", "votes": null, "replies": [] }, { "id": 135947, "author_name": "sandeepkumar", "author_url": "", "post_date": "09/18/2016 17:45:41", "content": "Now,I got it.I changes to :
\n\nmemory.limit(size=10000)
\n\nRead 1183748 rows and 969 (of 969) columns from 1.991 GB file in 00:05:10
", "votes": null, "replies": [] }, { "id": 136175, "author_name": "dkuehn", "author_url": "", "post_date": "09/20/2016 12:19:03", "content": "Here is the script I used to load the data into SQLight. In case you don't have enough RAM to execute it, you could play with nrows and skip in fread() to process data in chunks. For this to work, you might need to write an additional INSERT-statement and execute it with dbSendQuery. Better/faster approaches are welcome :)
\n\n# install/load packages\nif (!require("pacman")) install.packages("pacman")\npacman::p_load(data.table, RSQLite)\n\n# Set path and database name\npath <- "./data"\nsetwd(path)\nsqlite <- dbDriver("SQLite")\ncon <- dbConnect(sqlite, "20160920_kgl_Bosch.sqlite3")\n\n# Loop through all csv files in path and write them to SQLight\nfor (file in list.files(pattern = "*.csv")){\n print(Sys.time())\n print(file)\n data <- fread(file)\n tbl_name <- gsub(".csv","", file)\n\n if (dbExistsTable(con, tbl_name) == FALSE){\n dbWriteTable(con, tbl_name, data)\n }\n\n # Create an index on the Id-column to improve performance on later queries\n dbSendQuery(con, paste0("CREATE UNIQUE INDEX idx_", tbl_name, " ON ", tbl_name, "(Id)"))\n rm(data)\n gc()\n}\nWord of advise - don't use MYSQL. It has a hard column limit of ~4k columns (1000 columns with inno db)
", "votes": null, "replies": [] }, { "id": 136560, "author_name": "chandrak", "author_url": "", "post_date": "09/22/2016 21:47:03", "content": "@Daniel_Kuehn,
\n\nCan you pls advise how did you change max.number of columns in sqlite? Did you recompile your sqlite?
\n\nThank you
", "votes": null, "replies": [] }, { "id": 136613, "author_name": "dkuehn", "author_url": "", "post_date": "09/23/2016 11:23:34", "content": "[quote=Chandrak1907;136560]
\n\n@Daniel_Kuehn,
\n\nCan you pls advise how did you change max.number of columns in sqlite? Did you recompile your sqlite?
\n\nThank you
\n\n[/quote]
\n\n Actually I didn't need to change max.number in this case, since I wrote each .csv file to a new table. SQLITE_MAX_COLUMN should be 2000 per table and non of the .csv files has more than 2k columns.\nI guess dbWriteTable handles cases with more than 2000 columns per table, since it works for me without any problem.
You can use joins on the Id column to retrieve all columns when you need them.\nE. g. try something like this to load the first 10k rows to R with all columns:
\n\ndbSendQuery(con, "SELECT * \nFROM train_categorical As c \n INNER JOIN train_date As d ON c.Id = d.Id\n INNER JOIN train_numeric AS n ON c.Id = n.Id\nLIMIT 10000")\nWhat I did was to normalize the database based on feature name to make the data-set "longer" By parsing the name you can get a Station Value, Line Value, and Feature Value --then each measurement is characterized by those 3 numbers (and even add in the date)
\n\nId Value L-Value S-Value F Value Date\n 4 0.1 1 1 5 187.2\n 7 T1 2 3 89 192.6\nOf course, my background is in Databases so this was a natural way to proceed...
\n\nThen to get things like all the unique values for L2_S3_F89:
\n\nSELECT COUNT(1), VALUE, S-VALUE, F-VALUE FROM MY_TABLE\nWHERE L-VALUE=2 AND S-VALUE=3 AND F-VALUE=89\nGROUP BY L-VALUE,S-VALUE, F-VALUE\n@Daniel_Kuehn,
\n\nThank you for replying.\nAre you sure that train_categorical.csv has less than 2000 columns. When I read in first 100 rows , I see 2141 columns.
", "votes": null, "replies": [] }, { "id": 136668, "author_name": "rdslater", "author_url": "", "post_date": "09/23/2016 20:12:04", "content": "@Chandrak1907
\n\nI see the same number and have seen that number tossed around as well.
\n\nThere was talk of some of the features not being in the test set (all NAN for that particular feature) but I do not know if that is enough to reduce below 2000.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "134063": "I know all the cool kids are going to the Hadoop cluster, but this data is THAT big =)\r\n\r\nAnyway it is big enough that I cannot load into memory--my first thought was putting it in a personal database--has anyone else thought along that line? I got my start in traditional SQL dbs so it seemed natural to just load the data up and even manipulate it using that method to take advantage of hard drive space, rather than memory.\r\n\r\n\r\nAnyone else approaching it that way? I went postgresql and got the numeric data loaded pretty quick (a few minutes).", "135805": "I'm experimenting with that route as well.", "135844": "Since you can't use SQL to build models and make any kind of predictive analysis anyway, why not using *data.table* in R? It is capable of handling way bigger datasets. And in some sense it behaves similar to SQL (you can also make joins, for example).\r\nIt took me 1 minute on a laptop to read \"test_numeric.csv\". \r\n\r\n require(data.table)\r\n require(bit64)\r\n\r\n test = fread(\"test_numeric.csv\")\r\n > test = fread(\"test_numeric.csv\")\r\n Read 1183748 rows and 969 (of 969) columns from 1.991 GB file in 00:01:08\r\n (of 969) columns from 1.991 GB file in 00:01:08\r\n\r\n > dim(test)\r\n [1] 1183748 969", "135943": "Memory limits. I only have 8GB of memory.", "135944": "Hi, \r\n\r\nI have used the same.I am getting the below error. \r\n\r\ntest = fread(\"test_numeric.csv\")\r\n\r\nRead 0.0% of 1183748 rows\r\n\r\nError: cannot allocate vector of size 9.0 Mb\r\n\r\nIn addition: There were 20 warnings (use warnings() to see them)\r\n\r\n\r\n\r\nCan someone help me to resolve this?\r\n\r\n\r\nThank you.", "135947": "Now,I got it.I changes to :\r\n\r\nmemory.limit(size=10000)\r\n\r\nRead 1183748 rows and 969 (of 969) columns from 1.991 GB file in 00:05:10", "136175": "Here is the script I used to load the data into SQLight. In case you don't have enough RAM to execute it, you could play with nrows and skip in fread() to process data in chunks. For this to work, you might need to write an additional INSERT-statement and execute it with dbSendQuery. Better/faster approaches are welcome :)\r\n\r\n # install/load packages\r\n if (!require(\"pacman\")) install.packages(\"pacman\")\r\n pacman::p_load(data.table, RSQLite)\r\n \r\n # Set path and database name\r\n path <- \"./data\"\r\n setwd(path)\r\n sqlite <- dbDriver(\"SQLite\")\r\n con <- dbConnect(sqlite, \"20160920_kgl_Bosch.sqlite3\")\r\n \r\n # Loop through all csv files in path and write them to SQLight\r\n for (file in list.files(pattern = \"*.csv\")){\r\n print(Sys.time())\r\n print(file)\r\n data <- fread(file)\r\n tbl_name <- gsub(\".csv\",\"\", file)\r\n \r\n if (dbExistsTable(con, tbl_name) == FALSE){\r\n dbWriteTable(con, tbl_name, data)\r\n }\r\n \r\n # Create an index on the Id-column to improve performance on later queries\r\n dbSendQuery(con, paste0(\"CREATE UNIQUE INDEX idx_\", tbl_name, \" ON \", tbl_name, \"(Id)\"))\r\n rm(data)\r\n gc()\r\n }", "136402": "Word of advise - don't use MYSQL. It has a hard column limit of ~4k columns (1000 columns with inno db)", "136560": "Daniel_Kuehn,\r\n\r\nCan you pls advise how did you change max.number of columns in sqlite? Did you recompile your sqlite?\r\n\r\nThank you", "136613": "[quote=Chandrak1907;136560]\r\n\r\n@Daniel_Kuehn,\r\n\r\nCan you pls advise how did you change max.number of columns in sqlite? Did you recompile your sqlite?\r\n\r\nThank you\r\n\r\n[/quote]\r\n\r\n