{ "id": 6827, "title": "Reading Image files", "url": "/competitions/galaxy-zoo-the-galaxy-challenge/discussion/6827", "author_name": "", "post_date": "2014-01-13T16:42:23.030Z", "votes": null, "comment_count": 4, "views": 2813, "content": "

Hi everyone,

\n

I am using this competition to get familiar with image processing but am stuck at the very start. I am using python to read the images and vectorize them but I get an error saying "too many open files" when I try stacking them. Am sure most of you here couldhelp me with this, is it a memory issue with my system or can I code efficiently to avoid this?

\n

 

\n

Help appreciated!!

\n

Thanks,

\n

MB

", "messages": [ { "id": "37353", "postDate": "01/13/2014 16:42:23", "content": "

Hi everyone,

\n

I am using this competition to get familiar with image processing but am stuck at the very start. I am using python to read the images and vectorize them but I get an error saying "too many open files" when I try stacking them. Am sure most of you here couldhelp me with this, is it a memory issue with my system or can I code efficiently to avoid this?

\n

 

\n

Help appreciated!!

\n

Thanks,

\n

MB

", "rawMarkdown": "", "votes": null }, { "id": "37355", "postDate": "01/13/2014 19:22:50", "content": "

What do you mean with stacking images?

\n

Here is some Python and PIL code to read images in a directory, one by one, cropping the images center-center to 140px * 140px, convert to grayscale, pixelate that image to 5px * 5px blocks (taking average of the 25 pixels that make up a block), then vectorizes these blocks to floats ranging between 0 and 1. For now it outputs a file with on every line: the image id followed by a list of tuples with feature id (0-783) and feature value (float between 0-1).

\n

I hope this gets you started vectorizing images without running into memory problems. This code is still fairly slow (about 100 images a second). If you are looking for increased speed, Numpy can vectorize images a lot faster.

\n

#imports
from PIL import Image
import glob
import os

\n

#main script variables
pixelSize = 5
crop_dimensions = (140, 140, 284, 284)
glob_files = "kaggle_space/images_test/*.jpg"
vector_file = "kaggle_space/imagevectors.txt" #will be created
nr_docs = len(glob.glob(glob_files))

\n

#open a file to store our vectors
with open(vector_file, "wb") as outfile:
  for enum, infile in enumerate( glob.glob(glob_files) ):

\n

    #open image and get the filename and extension
    image = Image.open(infile)
    filen, ext = os.path.splitext(infile)
    file_id = filen[-6:]

\n

    # take out interesting part in the center-center, leaves 140x140 image
    image = image.crop(crop_dimensions)

\n

    # convert to grayscale 0-256
    image = image.convert('L')

\n

    # pixelate the image with pixelSize
    image = image.resize((image.size[0]/pixelSize,

\n

image.size[1]/pixelSize), Image.NEAREST)
    image = image.resize((image.size[0]*pixelSize, image.size[1]*pixelSize), Image.NEAREST)

\n

    # load resulting pixel data
    pixel = image.load()

\n

    # convert every pixelated block to a 0-1 float
    u = 0
    vectors = []
    for i in xrange(0,image.size[0],pixelSize):
      for y in xrange(0,image.size[1],pixelSize):
        vectors.append( (u, round(pixel[y, i]/float(255),3)) )
        u += 1

\n

    #write vectors to file (or format as libSVM or whatever)
    outfile.write(str(file_id) + " " + str(vectors) + "\\n")

\n

    #status report
    if enum % 100 == 0:
      print enum

", "rawMarkdown": "", "votes": null }, { "id": "37379", "postDate": "01/14/2014 15:43:12", "content": "

I was using vstack to vertically stack images but this is more than helpful, thanks a lot Triskelion!!

", "rawMarkdown": "", "votes": null }, { "id": "37639", "postDate": "01/20/2014 14:52:32", "content": "

Thanks a lot Triskelion.  Managed to run your code and get the features for a couple of trial images.  Would be great if you can provide some pointers to code for the following:

\n

1. Is there someway to construct the images from the vectors - want to make sure that I am vectorizing correctly.  

\n

2. The file contains the feature id and the feature value in parenthesis. I want to load a matrix containing just the features to some tool like Octave/R.  Is there some ready code which will convert the generated text file to CSV or some other format which I can load using elementary commands.

\n

It would be great if you can provide pointers to code doing these.  Understand these might be simple... but I have never done image processing or coding in Python.  This is my first attempt after learning some basic stuff through coursera.

", "rawMarkdown": "", "votes": null }, { "id": "37719", "postDate": "01/22/2014 11:04:13", "content": "

1. You should be able to convert the vectors back to pixel values by multiplying them by 255. Then read up on PIL on how to save a list of pixel values to an image.

\n

If you just want to see the resulting pixelated image try image.save("test.jpg") just before #load resulting pixel data.

\n

2. Would be best to use a csv writer for that. Quick hack, change this line:

\n

vectors.append( (u, round(pixel[y, i]/float(255),3)) )

\n

to: 

\n

vectors.append( str(round(pixel[y, i]/float(255),3)) )

\n

Then change this line:

\n

outfile.write(str(file_id) + " " + str(vectors) + "\\n")

\n

to:

\n

outfile.write(str(file_id) + "," + ",".join(vectors) + "\\n")

\n

Not tested, but this should create a CSV-file without a header. Try if it works, if not, I'll test myself and provide proper code.

", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 37355, "author_name": "triskelion", "author_url": "", "post_date": "01/13/2014 19:22:50", "content": "

What do you mean with stacking images?

\n

Here is some Python and PIL code to read images in a directory, one by one, cropping the images center-center to 140px * 140px, convert to grayscale, pixelate that image to 5px * 5px blocks (taking average of the 25 pixels that make up a block), then vectorizes these blocks to floats ranging between 0 and 1. For now it outputs a file with on every line: the image id followed by a list of tuples with feature id (0-783) and feature value (float between 0-1).

\n

I hope this gets you started vectorizing images without running into memory problems. This code is still fairly slow (about 100 images a second). If you are looking for increased speed, Numpy can vectorize images a lot faster.

\n

#imports
from PIL import Image
import glob
import os

\n

#main script variables
pixelSize = 5
crop_dimensions = (140, 140, 284, 284)
glob_files = "kaggle_space/images_test/*.jpg"
vector_file = "kaggle_space/imagevectors.txt" #will be created
nr_docs = len(glob.glob(glob_files))

\n

#open a file to store our vectors
with open(vector_file, "wb") as outfile:
  for enum, infile in enumerate( glob.glob(glob_files) ):

\n

    #open image and get the filename and extension
    image = Image.open(infile)
    filen, ext = os.path.splitext(infile)
    file_id = filen[-6:]

\n

    # take out interesting part in the center-center, leaves 140x140 image
    image = image.crop(crop_dimensions)

\n

    # convert to grayscale 0-256
    image = image.convert('L')

\n

    # pixelate the image with pixelSize
    image = image.resize((image.size[0]/pixelSize,

\n

image.size[1]/pixelSize), Image.NEAREST)
    image = image.resize((image.size[0]*pixelSize, image.size[1]*pixelSize), Image.NEAREST)

\n

    # load resulting pixel data
    pixel = image.load()

\n

    # convert every pixelated block to a 0-1 float
    u = 0
    vectors = []
    for i in xrange(0,image.size[0],pixelSize):
      for y in xrange(0,image.size[1],pixelSize):
        vectors.append( (u, round(pixel[y, i]/float(255),3)) )
        u += 1

\n

    #write vectors to file (or format as libSVM or whatever)
    outfile.write(str(file_id) + " " + str(vectors) + "\\n")

\n

    #status report
    if enum % 100 == 0:
      print enum

", "votes": null, "replies": [] }, { "id": 37379, "author_name": "gauravtaneja", "author_url": "", "post_date": "01/14/2014 15:43:12", "content": "

I was using vstack to vertically stack images but this is more than helpful, thanks a lot Triskelion!!

", "votes": null, "replies": [] }, { "id": 37639, "author_name": "ratheesh", "author_url": "", "post_date": "01/20/2014 14:52:32", "content": "

Thanks a lot Triskelion.  Managed to run your code and get the features for a couple of trial images.  Would be great if you can provide some pointers to code for the following:

\n

1. Is there someway to construct the images from the vectors - want to make sure that I am vectorizing correctly.  

\n

2. The file contains the feature id and the feature value in parenthesis. I want to load a matrix containing just the features to some tool like Octave/R.  Is there some ready code which will convert the generated text file to CSV or some other format which I can load using elementary commands.

\n

It would be great if you can provide pointers to code doing these.  Understand these might be simple... but I have never done image processing or coding in Python.  This is my first attempt after learning some basic stuff through coursera.

", "votes": null, "replies": [] }, { "id": 37719, "author_name": "triskelion", "author_url": "", "post_date": "01/22/2014 11:04:13", "content": "

1. You should be able to convert the vectors back to pixel values by multiplying them by 255. Then read up on PIL on how to save a list of pixel values to an image.

\n

If you just want to see the resulting pixelated image try image.save("test.jpg") just before #load resulting pixel data.

\n

2. Would be best to use a csv writer for that. Quick hack, change this line:

\n

vectors.append( (u, round(pixel[y, i]/float(255),3)) )

\n

to: 

\n

vectors.append( str(round(pixel[y, i]/float(255),3)) )

\n

Then change this line:

\n

outfile.write(str(file_id) + " " + str(vectors) + "\\n")

\n

to:

\n

outfile.write(str(file_id) + "," + ",".join(vectors) + "\\n")

\n

Not tested, but this should create a CSV-file without a header. Try if it works, if not, I'll test myself and provide proper code.

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "37353": "", "37355": "", "37379": "", "37639": "", "37719": "" }, "source": "meta" }