{
  "id": 94918,
  "title": "Faster Feature Generation",
  "url": "/competitions/champs-scalar-coupling/discussion/94918",
  "author_name": "",
  "post_date": "2019-06-08T05:22:03.539583300Z",
  "votes": 2,
  "comment_count": 3,
  "views": 0,
  "content": "<p>While creating features for the train and test set it came to my notice that the CPU utilization is always around 100% and that can be increased to 400% (if you use kaggle kernel). If you are creating features using information from one row of a dataframe at a time then it can be easily made parallel to do multiple rows at a time. I used python <a href=\"https://docs.python.org/3.4/library/multiprocessing.html\">multiprocessing library</a> and the time was reduced to nearly 1/4th. This post is just a heads-up/reminder for noobs like me that things can be made parallel. </p>\n\n<p>happy kaggling.</p>",
  "messages": [
    {
      "id": "547674",
      "postDate": "06/08/2019 05:22:03",
      "content": "<p>While creating features for the train and test set it came to my notice that the CPU utilization is always around 100% and that can be increased to 400% (if you use kaggle kernel). If you are creating features using information from one row of a dataframe at a time then it can be easily made parallel to do multiple rows at a time. I used python <a href=\"https://docs.python.org/3.4/library/multiprocessing.html\">multiprocessing library</a> and the time was reduced to nearly 1/4th. This post is just a heads-up/reminder for noobs like me that things can be made parallel. </p>\n\n<p>happy kaggling.</p>",
      "rawMarkdown": "While creating features for the train and test set it came to my notice that the CPU utilization is always around 100% and that can be increased to 400% (if you use kaggle kernel). If you are creating features using information from one row of a dataframe at a time then it can be easily made parallel to do multiple rows at a time. I used python [multiprocessing library](https://docs.python.org/3.4/library/multiprocessing.html) and the time was reduced to nearly 1/4th. This post is just a heads-up/reminder for noobs like me that things can be made parallel. \n\nhappy kaggling.",
      "votes": null
    },
    {
      "id": "547770",
      "postDate": "06/08/2019 08:41:00",
      "content": "<p>how many features do you create?</p>",
      "rawMarkdown": "how many features do you create?",
      "votes": null
    },
    {
      "id": "547831",
      "postDate": "06/08/2019 11:07:19",
      "content": "<p>you can try reduce mem usage function like in this  <a href=\"https://www.kaggle.com/kabure/lightgbm-full-pipeline-model\">kabure kernel</a> \nHope it helps</p>",
      "rawMarkdown": "you can try reduce mem usage function like in this  [kabure kernel](https://www.kaggle.com/kabure/lightgbm-full-pipeline-model) \nHope it helps",
      "votes": null
    },
    {
      "id": "547920",
      "postDate": "06/08/2019 14:11:33",
      "content": "<p>current count is 24 Most of them can be created from sructures.csv but I prefer the xyz files so I need that parallelism. </p>",
      "rawMarkdown": "current count is 24 Most of them can be created from sructures.csv but I prefer the xyz files so I need that parallelism.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 547770,
      "author_name": "hadxu123",
      "author_url": "",
      "post_date": "06/08/2019 08:41:00",
      "content": "<p>how many features do you create?</p>",
      "votes": null,
      "replies": [
        {
          "id": 547920,
          "author_name": "dhananjay3",
          "author_url": "",
          "post_date": "06/08/2019 14:11:33",
          "content": "<p>current count is 24 Most of them can be created from sructures.csv but I prefer the xyz files so I need that parallelism. </p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 547831,
      "author_name": "hervind",
      "author_url": "",
      "post_date": "06/08/2019 11:07:19",
      "content": "<p>you can try reduce mem usage function like in this  <a href=\"https://www.kaggle.com/kabure/lightgbm-full-pipeline-model\">kabure kernel</a> \nHope it helps</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "547674": "While creating features for the train and test set it came to my notice that the CPU utilization is always around 100% and that can be increased to 400% (if you use kaggle kernel). If you are creating features using information from one row of a dataframe at a time then it can be easily made parallel to do multiple rows at a time. I used python [multiprocessing library](https://docs.python.org/3.4/library/multiprocessing.html) and the time was reduced to nearly 1/4th. This post is just a heads-up/reminder for noobs like me that things can be made parallel. \n\nhappy kaggling.",
    "547770": "how many features do you create?",
    "547831": "you can try reduce mem usage function like in this  [kabure kernel](https://www.kaggle.com/kabure/lightgbm-full-pipeline-model) \nHope it helps",
    "547920": "current count is 24 Most of them can be created from sructures.csv but I prefer the xyz files so I need that parallelism."
  },
  "source": "meta"
}