The data is pretty big as you know. I have tried GPU options and reduce the memory as suggested in a few kernels, but both of them don't seem help. Even if I just starting the work with EDA, the kernel is so slow it draws me crazy ... If someone have any suggestion I will be very appreciated.
", "messages": [ { "id": "667023", "postDate": "11/06/2019 18:19:14", "content": "The data is pretty big as you know. I have tried GPU options and reduce the memory as suggested in a few kernels, but both of them don't seem help. Even if I just starting the work with EDA, the kernel is so slow it draws me crazy ... If someone have any suggestion I will be very appreciated.
", "rawMarkdown": "The data is pretty big as you know. I have tried GPU options and reduce the memory as suggested in a few kernels, but both of them don't seem help. Even if I just starting the work with EDA, the kernel is so slow it draws me crazy ... If someone have any suggestion I will be very appreciated.", "votes": null }, { "id": "667218", "postDate": "11/07/2019 00:58:22", "content": "You can either reduce the csv file by reading some samples for experimentation purpose, or download the dataset to run it on your local PC/server if it has a better HW specs.
", "rawMarkdown": "You can either reduce the csv file by reading some samples for experimentation purpose, or download the dataset to run it on your local PC/server if it has a better HW specs.", "votes": null }, { "id": "668647", "postDate": "11/08/2019 16:59:27", "content": "You may also try google's colab - has higher RAM specs and can be integrated with Kaggle and G-drive easily.
", "rawMarkdown": "You may also try google's colab - has higher RAM specs and can be integrated with Kaggle and G-drive easily.", "votes": null }, { "id": "669268", "postDate": "11/09/2019 19:36:54", "content": "Thanks for the advice, How can integrated Kaggle Google colab ?
", "rawMarkdown": "Thanks for the advice, How can integrated Kaggle Google colab ?", "votes": null }, { "id": "669585", "postDate": "11/10/2019 07:51:21", "content": "@aaysbt please can you help me knowing what is the purpose of the competition\nI am new to this platform\nI even asked the same question in forum but no one replied.
", "rawMarkdown": "aaysbt please can you help me knowing what is the purpose of the competition\nI am new to this platform\nI even asked the same question in forum but no one replied.", "votes": null }, { "id": "670328", "postDate": "11/11/2019 10:34:52", "content": "For me learning, helps me how can improve my self and my algorithm and eventually, I can see these improvements in competition.
", "rawMarkdown": "For me learning, helps me how can improve my self and my algorithm and eventually, I can see these improvements in competition.", "votes": null }, { "id": "671362", "postDate": "11/12/2019 15:16:32", "content": "The size you limit your test on 100.000 records fe, makes the query much more faster
\n\nhoh, the biggest problem with all datasets is that you have to 'prepare' the data to a better level.\nfe: this datasets has alot of 'hiates', hence you have to find a trick to fill the empty lacking values (say you do a prediction of the empties... with the most easy ones firts, and the most relevant first, probably with coupling of external data...\nSecond problem is that energy consumption of industrial processes,water heating, versus heating/cooling, is not function of the weather... So the population would need 4 different regressions ifo application...
\n\nThird: the energy consumption of a building is function of the insulation ( kWh/m2/difference kelving) / energy efficiency techniques (heat pump, COP6 or heat pump COP 3, or CHC combined heat coupling, or absorption gas pump , or trigeneration systems = cold/heat/waste heat/electricity ) / All that information seems not to be available...
\n\nSo i find it a rather useless exercise...
\n\nUsually i am concentrating on creating functions, that 'automate' that kind of classification / labelencoding stuff... Or i wait until someone creates a kernel that shows a solution of those mentioned problems... Otherwise you gonna spend months of work doing all those exercises... While the 'AI' stuff is rather 30% of the solution of most problems..
", "rawMarkdown": "The size you limit your test on 100.000 records fe, makes the query much more faster\n\n\nhoh, the biggest problem with all datasets is that you have to 'prepare' the data to a better level.\nfe: this datasets has alot of 'hiates', hence you have to find a trick to fill the empty lacking values (say you do a prediction of the empties... with the most easy ones firts, and the most relevant first, probably with coupling of external data...\nSecond problem is that energy consumption of industrial processes,water heating, versus heating/cooling, is not function of the weather... So the population would need 4 different regressions ifo application...\n\nThird: the energy consumption of a building is function of the insulation ( kWh/m2/difference kelving) / energy efficiency techniques (heat pump, COP6 or heat pump COP 3, or CHC combined heat coupling, or absorption gas pump , or trigeneration systems = cold/heat/waste heat/electricity ) / All that information seems not to be available...\n\nSo i find it a rather useless exercise...\n\nUsually i am concentrating on creating functions, that 'automate' that kind of classification / labelencoding stuff... Or i wait until someone creates a kernel that shows a solution of those mentioned problems... Otherwise you gonna spend months of work doing all those exercises... While the 'AI' stuff is rather 30% of the solution of most problems..", "votes": null } ], "comments": [ { "id": 667218, "author_name": "klloke", "author_url": "", "post_date": "11/07/2019 00:58:22", "content": "You can either reduce the csv file by reading some samples for experimentation purpose, or download the dataset to run it on your local PC/server if it has a better HW specs.
", "votes": null, "replies": [] }, { "id": 668647, "author_name": "imeintanis", "author_url": "", "post_date": "11/08/2019 16:59:27", "content": "You may also try google's colab - has higher RAM specs and can be integrated with Kaggle and G-drive easily.
", "votes": null, "replies": [ { "id": 669268, "author_name": "aaysbt", "author_url": "", "post_date": "11/09/2019 19:36:54", "content": "Thanks for the advice, How can integrated Kaggle Google colab ?
", "votes": null, "replies": [] }, { "id": 669585, "author_name": "sahib12", "author_url": "", "post_date": "11/10/2019 07:51:21", "content": "@aaysbt please can you help me knowing what is the purpose of the competition\nI am new to this platform\nI even asked the same question in forum but no one replied.
", "votes": null, "replies": [] }, { "id": 670328, "author_name": "aaysbt", "author_url": "", "post_date": "11/11/2019 10:34:52", "content": "For me learning, helps me how can improve my self and my algorithm and eventually, I can see these improvements in competition.
", "votes": null, "replies": [] } ] }, { "id": 671362, "author_name": "plarmuseau", "author_url": "", "post_date": "11/12/2019 15:16:32", "content": "The size you limit your test on 100.000 records fe, makes the query much more faster
\n\nhoh, the biggest problem with all datasets is that you have to 'prepare' the data to a better level.\nfe: this datasets has alot of 'hiates', hence you have to find a trick to fill the empty lacking values (say you do a prediction of the empties... with the most easy ones firts, and the most relevant first, probably with coupling of external data...\nSecond problem is that energy consumption of industrial processes,water heating, versus heating/cooling, is not function of the weather... So the population would need 4 different regressions ifo application...
\n\nThird: the energy consumption of a building is function of the insulation ( kWh/m2/difference kelving) / energy efficiency techniques (heat pump, COP6 or heat pump COP 3, or CHC combined heat coupling, or absorption gas pump , or trigeneration systems = cold/heat/waste heat/electricity ) / All that information seems not to be available...
\n\nSo i find it a rather useless exercise...
\n\nUsually i am concentrating on creating functions, that 'automate' that kind of classification / labelencoding stuff... Or i wait until someone creates a kernel that shows a solution of those mentioned problems... Otherwise you gonna spend months of work doing all those exercises... While the 'AI' stuff is rather 30% of the solution of most problems..
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "667023": "The data is pretty big as you know. I have tried GPU options and reduce the memory as suggested in a few kernels, but both of them don't seem help. Even if I just starting the work with EDA, the kernel is so slow it draws me crazy ... If someone have any suggestion I will be very appreciated.", "667218": "You can either reduce the csv file by reading some samples for experimentation purpose, or download the dataset to run it on your local PC/server if it has a better HW specs.", "668647": "You may also try google's colab - has higher RAM specs and can be integrated with Kaggle and G-drive easily.", "669268": "Thanks for the advice, How can integrated Kaggle Google colab ?", "669585": "aaysbt please can you help me knowing what is the purpose of the competition\nI am new to this platform\nI even asked the same question in forum but no one replied.", "670328": "For me learning, helps me how can improve my self and my algorithm and eventually, I can see these improvements in competition.", "671362": "The size you limit your test on 100.000 records fe, makes the query much more faster\n\n\nhoh, the biggest problem with all datasets is that you have to 'prepare' the data to a better level.\nfe: this datasets has alot of 'hiates', hence you have to find a trick to fill the empty lacking values (say you do a prediction of the empties... with the most easy ones firts, and the most relevant first, probably with coupling of external data...\nSecond problem is that energy consumption of industrial processes,water heating, versus heating/cooling, is not function of the weather... So the population would need 4 different regressions ifo application...\n\nThird: the energy consumption of a building is function of the insulation ( kWh/m2/difference kelving) / energy efficiency techniques (heat pump, COP6 or heat pump COP 3, or CHC combined heat coupling, or absorption gas pump , or trigeneration systems = cold/heat/waste heat/electricity ) / All that information seems not to be available...\n\nSo i find it a rather useless exercise...\n\nUsually i am concentrating on creating functions, that 'automate' that kind of classification / labelencoding stuff... Or i wait until someone creates a kernel that shows a solution of those mentioned problems... Otherwise you gonna spend months of work doing all those exercises... While the 'AI' stuff is rather 30% of the solution of most problems.." }, "source": "meta" }