I'm curious to know if anyone is using R and if it becomes non-responsive during any of your attempts to join (a/k/a \"merge\") your data frames. I had to read some of the tables for this competition in to my machine (Mac Power Book 2015, 16MB Ram, 250GB storage (113GB free)) in chunks. I successfully joined the RData objects corresponding to train.csv and historical_transactions.csv on card_id, however, when I try to join new_merchant_transactions to this new object R becomes non-responsive.
\n\nHas anyone else experienced this? Is there a workaround? Thanks.
", "messages": [ { "id": "438525", "postDate": "12/13/2018 20:53:01", "content": "I'm curious to know if anyone is using R and if it becomes non-responsive during any of your attempts to join (a/k/a \"merge\") your data frames. I had to read some of the tables for this competition in to my machine (Mac Power Book 2015, 16MB Ram, 250GB storage (113GB free)) in chunks. I successfully joined the RData objects corresponding to train.csv and historical_transactions.csv on card_id, however, when I try to join new_merchant_transactions to this new object R becomes non-responsive.
\n\nHas anyone else experienced this? Is there a workaround? Thanks.
", "rawMarkdown": "I'm curious to know if anyone is using R and if it becomes non-responsive during any of your attempts to join (a/k/a \"merge\") your data frames. I had to read some of the tables for this competition in to my machine (Mac Power Book 2015, 16MB Ram, 250GB storage (113GB free)) in chunks. I successfully joined the RData objects corresponding to train.csv and historical_transactions.csv on card_id, however, when I try to join new_merchant_transactions to this new object R becomes non-responsive. \n\nHas anyone else experienced this? Is there a workaround? Thanks.", "votes": null }, { "id": "438553", "postDate": "12/13/2018 21:56:05", "content": "You mean you have 16GB of RAM, correct?
\n\nHow are you processing your joins (what package, code, etc)? There are multiple keys and if R is hanging my guess would be it is having trouble processing joins when there are multiple keys that haven't been explicitly identified.
", "rawMarkdown": "You mean you have 16GB of RAM, correct?\n\nHow are you processing your joins (what package, code, etc)? There are multiple keys and if R is hanging my guess would be it is having trouble processing joins when there are multiple keys that haven't been explicitly identified.", "votes": null }, { "id": "438559", "postDate": "12/13/2018 22:04:00", "content": "Yes, 16MB of RAM. I'm not doing anything fancy. The command was train.hn = merge(train.hist, new.merch, by.x=\"card_id\", by.y=\"card_id\", all.x=TRUE). There may be a couple of typos there, but it was a left join on a data frame that contained tain.csv+historical.tansactions.csv and second data frame \n containing new.merchant.transactions.csv on the column \"card.id,\" which I had typecast to a character vector because there was a mismatch between two of the .csv files.
", "rawMarkdown": "Yes, 16MB of RAM. I'm not doing anything fancy. The command was train.hn = merge(train.hist, new.merch, by.x=\"card_id\", by.y=\"card_id\", all.x=TRUE). There may be a couple of typos there, but it was a left join on a data frame that contained tain.csv+historical.tansactions.csv and second data frame \n containing new.merchant.transactions.csv on the column \"card.id,\" which I had typecast to a character vector because there was a mismatch between two of the .csv files.", "votes": null }, { "id": "438575", "postDate": "12/13/2018 22:40:19", "content": "Using datatable instead of dataframes would be much faster and memory efficient.
", "rawMarkdown": "Using datatable instead of dataframes would be much faster and memory efficient.", "votes": null }, { "id": "438589", "postDate": "12/13/2018 23:13:54", "content": "Hello Ryan,
\n\nYou could try creating a MySQL database, loading the train.csv and historicaltransactions.csv files and joining the tables in MySQL, performing any aggregations and processing and then exporting the resulting, joined, clean data set to a .csv file which you would then load in R.
\n\nAlternatively, you could setup a connection between R and MySQL and execute queries from R. I'd guess MySQL would be better suited than R for these data joining and manipulation tasks.
", "rawMarkdown": "Hello Ryan,\n\nYou could try creating a MySQL database, loading the train.csv and historicaltransactions.csv files and joining the tables in MySQL, performing any aggregations and processing and then exporting the resulting, joined, clean data set to a .csv file which you would then load in R.\n\nAlternatively, you could setup a connection between R and MySQL and execute queries from R. I'd guess MySQL would be better suited than R for these data joining and manipulation tasks.", "votes": null }, { "id": "438693", "postDate": "12/14/2018 03:17:49", "content": "Many Thanks!
", "rawMarkdown": "Many Thanks!", "votes": null }, { "id": "438695", "postDate": "12/14/2018 03:20:04", "content": "Appreciate the reply! I'd like to avoid loading a new application and complicating things beyond the ML aspects of the project, but this is an important option to know about. Thanks for the heads up!
", "rawMarkdown": "Appreciate the reply! I'd like to avoid loading a new application and complicating things beyond the ML aspects of the project, but this is an important option to know about. Thanks for the heads up!", "votes": null }, { "id": "439186", "postDate": "12/14/2018 21:19:33", "content": "This worked like a dream. Thanks so much for the suggestion. The specs for the data.table package look terrific. Are there any other general purpose packaged that you believe are \"must haves,\" aside from ggplot?
", "rawMarkdown": "This worked like a dream. Thanks so much for the suggestion. The specs for the data.table package look terrific. Are there any other general purpose packaged that you believe are \"must haves,\" aside from ggplot?", "votes": null } ], "comments": [ { "id": 438553, "author_name": "pgreene", "author_url": "", "post_date": "12/13/2018 21:56:05", "content": "You mean you have 16GB of RAM, correct?
\n\nHow are you processing your joins (what package, code, etc)? There are multiple keys and if R is hanging my guess would be it is having trouble processing joins when there are multiple keys that haven't been explicitly identified.
", "votes": null, "replies": [ { "id": 438559, "author_name": "rcd1693", "author_url": "", "post_date": "12/13/2018 22:04:00", "content": "Yes, 16MB of RAM. I'm not doing anything fancy. The command was train.hn = merge(train.hist, new.merch, by.x=\"card_id\", by.y=\"card_id\", all.x=TRUE). There may be a couple of typos there, but it was a left join on a data frame that contained tain.csv+historical.tansactions.csv and second data frame \n containing new.merchant.transactions.csv on the column \"card.id,\" which I had typecast to a character vector because there was a mismatch between two of the .csv files.
", "votes": null, "replies": [] } ] }, { "id": 438575, "author_name": "darraghdog", "author_url": "", "post_date": "12/13/2018 22:40:19", "content": "Using datatable instead of dataframes would be much faster and memory efficient.
", "votes": null, "replies": [ { "id": 438693, "author_name": "rcd1693", "author_url": "", "post_date": "12/14/2018 03:17:49", "content": "Many Thanks!
", "votes": null, "replies": [] }, { "id": 439186, "author_name": "rcd1693", "author_url": "", "post_date": "12/14/2018 21:19:33", "content": "This worked like a dream. Thanks so much for the suggestion. The specs for the data.table package look terrific. Are there any other general purpose packaged that you believe are \"must haves,\" aside from ggplot?
", "votes": null, "replies": [] } ] }, { "id": 438589, "author_name": "jgutierrezh", "author_url": "", "post_date": "12/13/2018 23:13:54", "content": "Hello Ryan,
\n\nYou could try creating a MySQL database, loading the train.csv and historicaltransactions.csv files and joining the tables in MySQL, performing any aggregations and processing and then exporting the resulting, joined, clean data set to a .csv file which you would then load in R.
\n\nAlternatively, you could setup a connection between R and MySQL and execute queries from R. I'd guess MySQL would be better suited than R for these data joining and manipulation tasks.
", "votes": null, "replies": [ { "id": 438695, "author_name": "rcd1693", "author_url": "", "post_date": "12/14/2018 03:20:04", "content": "Appreciate the reply! I'd like to avoid loading a new application and complicating things beyond the ML aspects of the project, but this is an important option to know about. Thanks for the heads up!
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "438525": "I'm curious to know if anyone is using R and if it becomes non-responsive during any of your attempts to join (a/k/a \"merge\") your data frames. I had to read some of the tables for this competition in to my machine (Mac Power Book 2015, 16MB Ram, 250GB storage (113GB free)) in chunks. I successfully joined the RData objects corresponding to train.csv and historical_transactions.csv on card_id, however, when I try to join new_merchant_transactions to this new object R becomes non-responsive. \n\nHas anyone else experienced this? Is there a workaround? Thanks.", "438553": "You mean you have 16GB of RAM, correct?\n\nHow are you processing your joins (what package, code, etc)? There are multiple keys and if R is hanging my guess would be it is having trouble processing joins when there are multiple keys that haven't been explicitly identified.", "438559": "Yes, 16MB of RAM. I'm not doing anything fancy. The command was train.hn = merge(train.hist, new.merch, by.x=\"card_id\", by.y=\"card_id\", all.x=TRUE). There may be a couple of typos there, but it was a left join on a data frame that contained tain.csv+historical.tansactions.csv and second data frame \n containing new.merchant.transactions.csv on the column \"card.id,\" which I had typecast to a character vector because there was a mismatch between two of the .csv files.", "438575": "Using datatable instead of dataframes would be much faster and memory efficient.", "438589": "Hello Ryan,\n\nYou could try creating a MySQL database, loading the train.csv and historicaltransactions.csv files and joining the tables in MySQL, performing any aggregations and processing and then exporting the resulting, joined, clean data set to a .csv file which you would then load in R.\n\nAlternatively, you could setup a connection between R and MySQL and execute queries from R. I'd guess MySQL would be better suited than R for these data joining and manipulation tasks.", "438693": "Many Thanks!", "438695": "Appreciate the reply! I'd like to avoid loading a new application and complicating things beyond the ML aspects of the project, but this is an important option to know about. Thanks for the heads up!", "439186": "This worked like a dream. Thanks so much for the suggestion. The specs for the data.table package look terrific. Are there any other general purpose packaged that you believe are \"must haves,\" aside from ggplot?" }, "source": "meta" }