How to handle this data while merging the data frames. It falls in memory error
", "messages": [ { "id": "685351", "postDate": "12/01/2019 13:29:11", "content": "How to handle this data while merging the data frames. It falls in memory error
", "rawMarkdown": "How to handle this data while merging the data frames. It falls in memory error", "votes": null }, { "id": "685417", "postDate": "12/01/2019 16:14:11", "content": "In R ? Take a look at https://cran.r-project.org/web/packages/data.table/vignettes/datatable-intro.html\nI also spend my day (and yesterday) figure out how to make my kernel not crashing during normalization of data. (Spoiler alert : gc() after every steps, but one was difficult to spot).
", "rawMarkdown": "In R ? Take a look at https://cran.r-project.org/web/packages/data.table/vignettes/datatable-intro.html\nI also spend my day (and yesterday) figure out how to make my kernel not crashing during normalization of data. (Spoiler alert : gc() after every steps, but one was difficult to spot).", "votes": null }, { "id": "685434", "postDate": "12/01/2019 16:57:29", "content": "In python reduce memory of every data frame before merging, so reduce: building,weather and train/test.\nIn R data.table is quiet strong, so you can merge without any issues.
", "rawMarkdown": "In python reduce memory of every data frame before merging, so reduce: building,weather and train/test.\nIn R data.table is quiet strong, so you can merge without any issues.", "votes": null }, { "id": "685437", "postDate": "12/01/2019 17:11:28", "content": "", "rawMarkdown": "", "votes": null }, { "id": "685465", "postDate": "12/01/2019 18:33:52", "content": "In R, use left_join instead of merge and do it by combining variables, e.g. by=c(\"site_id\",\"timestamp\").
", "rawMarkdown": "In R, use left__join_ instead of merge and do it by combining variables, e.g. by=c(\"site_id\",\"timestamp\").", "votes": null }, { "id": "685631", "postDate": "12/02/2019 02:58:19", "content": "Can you help me in one thing. Do i need to track the training set for each building separately and there meters also?
", "rawMarkdown": "Can you help me in one thing. Do i need to track the training set for each building separately and there meters also?", "votes": null }, { "id": "685695", "postDate": "12/02/2019 05:59:21", "content": "@rackson3861\nIt depends from the model you are using. In some kernel as divide and conquer it has been created one model for each meter (or site_id).\nIn my experience one single model for all data fine tuned with right features is better because lgbm are quite strong.\nBut if you're running every time out of memory and you don't want to optimize to much your code you can analyse every different meter or siteid separately and merge final results, this will surely help to reduce memory issues.
", "rawMarkdown": "rackson3861\nIt depends from the model you are using. In some kernel as divide and conquer it has been created one model for each meter (or site_id).\nIn my experience one single model for all data fine tuned with right features is better because lgbm are quite strong.\nBut if you're running every time out of memory and you don't want to optimize to much your code you can analyse every different meter or siteid separately and merge final results, this will surely help to reduce memory issues.", "votes": null } ], "comments": [ { "id": 685417, "author_name": "cdk292", "author_url": "", "post_date": "12/01/2019 16:14:11", "content": "In R ? Take a look at https://cran.r-project.org/web/packages/data.table/vignettes/datatable-intro.html\nI also spend my day (and yesterday) figure out how to make my kernel not crashing during normalization of data. (Spoiler alert : gc() after every steps, but one was difficult to spot).
", "votes": null, "replies": [] }, { "id": 685434, "author_name": "stenford23", "author_url": "", "post_date": "12/01/2019 16:57:29", "content": "In python reduce memory of every data frame before merging, so reduce: building,weather and train/test.\nIn R data.table is quiet strong, so you can merge without any issues.
", "votes": null, "replies": [ { "id": 685631, "author_name": "sumitrkt", "author_url": "", "post_date": "12/02/2019 02:58:19", "content": "Can you help me in one thing. Do i need to track the training set for each building separately and there meters also?
", "votes": null, "replies": [] }, { "id": 685695, "author_name": "stenford23", "author_url": "", "post_date": "12/02/2019 05:59:21", "content": "@rackson3861\nIt depends from the model you are using. In some kernel as divide and conquer it has been created one model for each meter (or site_id).\nIn my experience one single model for all data fine tuned with right features is better because lgbm are quite strong.\nBut if you're running every time out of memory and you don't want to optimize to much your code you can analyse every different meter or siteid separately and merge final results, this will surely help to reduce memory issues.
", "votes": null, "replies": [] } ] }, { "id": 685437, "author_name": "sumitrkt", "author_url": "", "post_date": "12/01/2019 17:11:28", "content": "", "votes": null, "replies": [] }, { "id": 685465, "author_name": "panosc", "author_url": "", "post_date": "12/01/2019 18:33:52", "content": "In R, use left_join instead of merge and do it by combining variables, e.g. by=c(\"site_id\",\"timestamp\").
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "685351": "How to handle this data while merging the data frames. It falls in memory error", "685417": "In R ? Take a look at https://cran.r-project.org/web/packages/data.table/vignettes/datatable-intro.html\nI also spend my day (and yesterday) figure out how to make my kernel not crashing during normalization of data. (Spoiler alert : gc() after every steps, but one was difficult to spot).", "685434": "In python reduce memory of every data frame before merging, so reduce: building,weather and train/test.\nIn R data.table is quiet strong, so you can merge without any issues.", "685437": "", "685465": "In R, use left__join_ instead of merge and do it by combining variables, e.g. by=c(\"site_id\",\"timestamp\").", "685631": "Can you help me in one thing. Do i need to track the training set for each building separately and there meters also?", "685695": "rackson3861\nIt depends from the model you are using. In some kernel as divide and conquer it has been created one model for each meter (or site_id).\nIn my experience one single model for all data fine tuned with right features is better because lgbm are quite strong.\nBut if you're running every time out of memory and you don't want to optimize to much your code you can analyse every different meter or siteid separately and merge final results, this will surely help to reduce memory issues." }, "source": "meta" }