{
  "id": 23056,
  "title": "Shrinking the *date files",
  "url": "/competitions/bosch-production-line-performance/discussion/23056",
  "author_name": "Konrad Banachewicz",
  "post_date": "2016-08-23T18:20:59.923000",
  "votes": 6,
  "comment_count": 0,
  "views": 718,
  "content": "<p>I had a long hard look at the *date files and started feeling suspicious about identical entries in adjacent columns. This led me to suspect there might be redundancy due to columns being linearly dependent. When i tried doing a decomposition on the full data matrix 16gb proved insufficient (at least using the R::caret implementation), but doing it in an iterative fashion seems to do the job - no new columns are purged towards the end of the iteration (the choice of number of iterations, as well as size of each are rather arbitrary - i managed to get away with using those params if i wanted to run the script in the background while doing sth else). </p>\n\n<p>Comments / suggestions / upvotes most welcome :-)</p>",
  "messages": [
    {
      "id": 132225,
      "postDate": "2016-08-23T18:20:59.923Z",
      "content": "<p>I had a long hard look at the *date files and started feeling suspicious about identical entries in adjacent columns. This led me to suspect there might be redundancy due to columns being linearly dependent. When i tried doing a decomposition on the full data matrix 16gb proved insufficient (at least using the R::caret implementation), but doing it in an iterative fashion seems to do the job - no new columns are purged towards the end of the iteration (the choice of number of iterations, as well as size of each are rather arbitrary - i managed to get away with using those params if i wanted to run the script in the background while doing sth else). </p>\n\n<p>Comments / suggestions / upvotes most welcome :-)</p>",
      "rawMarkdown": "I had a long hard look at the *date files and started feeling suspicious about identical entries in adjacent columns. This led me to suspect there might be redundancy due to columns being linearly dependent. When i tried doing a decomposition on the full data matrix 16gb proved insufficient (at least using the R::caret implementation), but doing it in an iterative fashion seems to do the job - no new columns are purged towards the end of the iteration (the choice of number of iterations, as well as size of each are rather arbitrary - i managed to get away with using those params if i wanted to run the script in the background while doing sth else). \r\n\r\nComments / suggestions / upvotes most welcome :-)",
      "votes": 6
    }
  ],
  "comments": [],
  "raw_markdown_by_id": {
    "132225": "I had a long hard look at the *date files and started feeling suspicious about identical entries in adjacent columns. This led me to suspect there might be redundancy due to columns being linearly dependent. When i tried doing a decomposition on the full data matrix 16gb proved insufficient (at least using the R::caret implementation), but doing it in an iterative fashion seems to do the job - no new columns are purged towards the end of the iteration (the choice of number of iterations, as well as size of each are rather arbitrary - i managed to get away with using those params if i wanted to run the script in the background while doing sth else). \r\n\r\nComments / suggestions / upvotes most welcome :-)"
  }
}