{
  "id": 95392,
  "title": "Merging all the data",
  "url": "/competitions/champs-scalar-coupling/discussion/95392",
  "author_name": "",
  "post_date": "2019-06-12T00:01:07.811256800Z",
  "votes": null,
  "comment_count": 2,
  "views": 0,
  "content": "<p>Anyone have any suggestions on how to  parallel process all the merged data?</p>\n\n<p>Merging all data is causing memory issues. </p>",
  "messages": [
    {
      "id": "550699",
      "postDate": "06/12/2019 00:01:07",
      "content": "<p>Anyone have any suggestions on how to  parallel process all the merged data?</p>\n\n<p>Merging all data is causing memory issues. </p>",
      "rawMarkdown": "Anyone have any suggestions on how to  parallel process all the merged data?\n\nMerging all data is causing memory issues.",
      "votes": null
    },
    {
      "id": "559203",
      "postDate": "06/23/2019 16:33:20",
      "content": "<p>I am having same issues, I think just getting clever with (if in python) using melt/pivot etc. is the only way to go</p>",
      "rawMarkdown": "I am having same issues, I think just getting clever with (if in python) using melt/pivot etc. is the only way to go",
      "votes": null
    },
    {
      "id": "559400",
      "postDate": "06/24/2019 04:43:55",
      "content": "<p>My PC will run 200 features with a training set of 20.9 GB (pickled file size in windows), but it dies a horrible death with 350 features.   Since the file contains 643 features at present it's probably time to kill some of them off.</p>\n\n<p>So my method is to add a file save of the importance results and run my kernel three times - each time with a different group of 215.  The saved importance file is appended to for each fold.  Building a decent set of oof features so it's pretty slow going.</p>\n\n<p>I have JMP stat package and its a quick file read, summary report, sort and I got the list of 643 features in order of importance.  That sorted list than used for my good_columns list.  It's a pain to have to run three times so will not have my sort done for 24 to 36 hours - should have implemented this when the number of features was 200 but I am a hoarder of features - never saw one I wanted to eliminate.</p>",
      "rawMarkdown": "My PC will run 200 features with a training set of 20.9 GB (pickled file size in windows), but it dies a horrible death with 350 features.   Since the file contains 643 features at present it's probably time to kill some of them off.\n\nSo my method is to add a file save of the importance results and run my kernel three times - each time with a different group of 215.  The saved importance file is appended to for each fold.  Building a decent set of oof features so it's pretty slow going.\n\nI have JMP stat package and its a quick file read, summary report, sort and I got the list of 643 features in order of importance.  That sorted list than used for my good_columns list.  It's a pain to have to run three times so will not have my sort done for 24 to 36 hours - should have implemented this when the number of features was 200 but I am a hoarder of features - never saw one I wanted to eliminate.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 559203,
      "author_name": "michaelj8",
      "author_url": "",
      "post_date": "06/23/2019 16:33:20",
      "content": "<p>I am having same issues, I think just getting clever with (if in python) using melt/pivot etc. is the only way to go</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 559400,
      "author_name": "pcjimmmy",
      "author_url": "",
      "post_date": "06/24/2019 04:43:55",
      "content": "<p>My PC will run 200 features with a training set of 20.9 GB (pickled file size in windows), but it dies a horrible death with 350 features.   Since the file contains 643 features at present it's probably time to kill some of them off.</p>\n\n<p>So my method is to add a file save of the importance results and run my kernel three times - each time with a different group of 215.  The saved importance file is appended to for each fold.  Building a decent set of oof features so it's pretty slow going.</p>\n\n<p>I have JMP stat package and its a quick file read, summary report, sort and I got the list of 643 features in order of importance.  That sorted list than used for my good_columns list.  It's a pain to have to run three times so will not have my sort done for 24 to 36 hours - should have implemented this when the number of features was 200 but I am a hoarder of features - never saw one I wanted to eliminate.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "550699": "Anyone have any suggestions on how to  parallel process all the merged data?\n\nMerging all data is causing memory issues.",
    "559203": "I am having same issues, I think just getting clever with (if in python) using melt/pivot etc. is the only way to go",
    "559400": "My PC will run 200 features with a training set of 20.9 GB (pickled file size in windows), but it dies a horrible death with 350 features.   Since the file contains 643 features at present it's probably time to kill some of them off.\n\nSo my method is to add a file save of the importance results and run my kernel three times - each time with a different group of 215.  The saved importance file is appended to for each fold.  Building a decent set of oof features so it's pretty slow going.\n\nI have JMP stat package and its a quick file read, summary report, sort and I got the list of 643 features in order of importance.  That sorted list than used for my good_columns list.  It's a pain to have to run three times so will not have my sort done for 24 to 36 hours - should have implemented this when the number of features was 200 but I am a hoarder of features - never saw one I wanted to eliminate."
  },
  "source": "meta"
}