{
  "id": 7243,
  "title": "Feature extraction times from training data",
  "url": "/competitions/galaxy-zoo-the-galaxy-challenge/discussion/7243",
  "author_name": "",
  "post_date": "2014-03-02T11:08:35.873Z",
  "votes": null,
  "comment_count": 2,
  "views": 1814,
  "content": "<p>Hmm I was wondering how long image feature extraction from training data took for others?</p>\n<p>Processing on my system takes 17 hours (on i7/920 with 16GB RAM and running R 3.0.2/64-bit). I am calculating mostly simple features and a few more complicated (Fourier, Fractals and shape of mass distribution). However my implementation is likely using only 1 core out of 4.</p>\n<p>I guess this is one reason why some are using those GPU libraries to speed-up processing significantly.</p>",
  "messages": [
    {
      "id": "39693",
      "postDate": "03/02/2014 11:08:35",
      "content": "<p>Hmm I was wondering how long image feature extraction from training data took for others?</p>\n<p>Processing on my system takes 17 hours (on i7/920 with 16GB RAM and running R 3.0.2/64-bit). I am calculating mostly simple features and a few more complicated (Fourier, Fractals and shape of mass distribution). However my implementation is likely using only 1 core out of 4.</p>\n<p>I guess this is one reason why some are using those GPU libraries to speed-up processing significantly.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "39760",
      "postDate": "03/03/2014 09:15:01",
      "content": "<p>I started yesterday. Extracting a single feature using Python + numpy + scikit-image took about 12 hours on my system&nbsp;(PCIe SSD, i5, 8GB RAM). I think the main bottleneck is the processing power needed to convert a jpg image to an array. CPU use was 300%, so I guess it was utilising some low-level libraries to take advantage of multiple cores.</p>\n<p>UPDATE:</p>\n<p>I decided to test the 'central pixel' feature, and it's extracting quite fast. (about 20000 images/minute).</p>\n<p>So I guess the algorithm I used for the previous feature is to blame (non-trivial thresholding, etc.).</p>\n<p>After running a new feature extract, turns, it's the convex-hull search algorithm that's to blame. Don't know about R vs Python speed differences, but perhaps the long feature extraction is because of a certain specific code part?</p>\n<p>And are you storing previously extracted features?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "39779",
      "postDate": "03/03/2014 15:14:46",
      "content": "<p>Hmm yeah, it seems that extracting blob (= &quot;thresholding and labeling B&amp;W image to segments&quot;) location and mass distribution takes most time. Fractal measures and fast fourier transform take second most of the time.</p>\n<p>I am storing things to memory only, and storing them only at the end of whole extraction proces. I also just noticed that my run time increased to ~72 hours for training data (and about 80 hours for test data feature extraction).</p>\n<p>Cannot say any specific R vs Python speed differences. But I would imagine that it is easier to write slow code with R than using Python :)</p>",
      "rawMarkdown": "",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 39760,
      "author_name": "kunstmord",
      "author_url": "",
      "post_date": "03/03/2014 09:15:01",
      "content": "<p>I started yesterday. Extracting a single feature using Python + numpy + scikit-image took about 12 hours on my system&nbsp;(PCIe SSD, i5, 8GB RAM). I think the main bottleneck is the processing power needed to convert a jpg image to an array. CPU use was 300%, so I guess it was utilising some low-level libraries to take advantage of multiple cores.</p>\n<p>UPDATE:</p>\n<p>I decided to test the 'central pixel' feature, and it's extracting quite fast. (about 20000 images/minute).</p>\n<p>So I guess the algorithm I used for the previous feature is to blame (non-trivial thresholding, etc.).</p>\n<p>After running a new feature extract, turns, it's the convex-hull search algorithm that's to blame. Don't know about R vs Python speed differences, but perhaps the long feature extraction is because of a certain specific code part?</p>\n<p>And are you storing previously extracted features?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 39779,
      "author_name": "sfin2015",
      "author_url": "",
      "post_date": "03/03/2014 15:14:46",
      "content": "<p>Hmm yeah, it seems that extracting blob (= &quot;thresholding and labeling B&amp;W image to segments&quot;) location and mass distribution takes most time. Fractal measures and fast fourier transform take second most of the time.</p>\n<p>I am storing things to memory only, and storing them only at the end of whole extraction proces. I also just noticed that my run time increased to ~72 hours for training data (and about 80 hours for test data feature extraction).</p>\n<p>Cannot say any specific R vs Python speed differences. But I would imagine that it is easier to write slow code with R than using Python :)</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "39693": "",
    "39760": "",
    "39779": ""
  },
  "source": "meta"
}