{ "id": 12765, "title": "CNN + Caffe", "url": "/competitions/datasciencebowl/discussion/12765", "author_name": "", "post_date": "2015-03-10T21:46:50.277Z", "votes": null, "comment_count": 12, "views": 6952, "content": "

Hi 

\n

in the last few days I switched to use Caffe for CNN. The problem now is how to get predictions? I know there is another post about probabilities but it wasn't helpful for me.

\n

any one has easy to follow python code to get probabilities from caffe for this competition and like to share it?

\n

thanks in advance

", "messages": [ { "id": "65920", "postDate": "03/10/2015 21:46:50", "content": "

Hi 

\n

in the last few days I switched to use Caffe for CNN. The problem now is how to get predictions? I know there is another post about probabilities but it wasn't helpful for me.

\n

any one has easy to follow python code to get probabilities from caffe for this competition and like to share it?

\n

thanks in advance

", "rawMarkdown": "", "votes": null }, { "id": "65923", "postDate": "03/10/2015 22:57:00", "content": "

I do something like this:

\n

import caffe

\n

import os

\n

MODEL_FILE = 'plankton_full_deploy.prototxt'

\n

PRETRAINED =  'checkpoints/plankton_full_iter_1000000.caffemodel'

\n

TESTPATH = '../data/test/'

\n

blob = caffe.proto.caffe_pb2.BlobProto()

\n

data = open('../data/test_mean.binaryproto','rb').read()

\n

blob.ParseFromString(data)

\n

arr = np.array(caffe.io.blobproto_to_array(blob))

\n

net = caffe.Classifier(MODEL_FILE,               

\n

                       PRETRAINED,                                    

\n

                       mean=arr[0], 

\n

                       raw_scale=255,   

\n

                       image_dims=(70,70))

\n

net.set_phase_test()net.set_mode_gpu()

\n

for root, dirs, files in os.walk(TESTPATH):     

\n

    for name in files:          

\n

        if name.endswith((".jpg")):                

\n

            image = caffe.io.load_image(os.path.join(root,name))

\n

            predictions = net.predict(image)                

\n

            out = [p for p in predictions]

\n

out will then be a list containing the class probabilities.  This will run slowly though as I have missed out some of the details on how I do this in batches of images, but this should be enough to show the basic method.  Good luck!

", "rawMarkdown": "", "votes": null }, { "id": "65924", "postDate": "03/10/2015 23:04:07", "content": "

See

\n

http://nbviewer.ipython.org/github/BVLC/caffe/blob/master/examples/classification.ipynb

", "rawMarkdown": "", "votes": null }, { "id": "65925", "postDate": "03/10/2015 23:35:55", "content": "

Thanks guys for respond. I will try my best.

\n

Good Luck

", "rawMarkdown": "", "votes": null }, { "id": "65966", "postDate": "03/11/2015 16:53:15", "content": "

Senecaur

\n

in the deployment model: plankton_full_deploy.prototxt what is the value u are setting for the number of channels?

\n

Because our images are gray scale I am putting 1 but it is making an error. when I put 3 it worked

\n

is there any setting I should consider to overcome this problem?

\n

another question what is the rule of mean file? If I train the network without mean file mean_file: "average.png" is my solution is right. btw I tried the code to load average.png but didn't work

\n\n

please it is so important to get answer. I just one to do at least one submission using CNN

", "rawMarkdown": "", "votes": null }, { "id": "65973", "postDate": "03/11/2015 19:02:45", "content": "

Hi,

\n

I also had an error about channels not matching first. In https://github.com/BVLC/caffe/issues/462 you find a modification for the python wrapper to handle one-channel input. If you create your leveldb/lmdb database with the convert_imageset function, you can pass a "-gray" flag to treat everything as grayscale. Using those two solved the channel error for me.

\n

Best, Matthias

", "rawMarkdown": "", "votes": null }, { "id": "65975", "postDate": "03/11/2015 19:28:00", "content": "

Hi,

\n

Function  caffe.io.load_image(os.path.join(root,name)) return image with 3 channels by default. You can use parameter color=False in order to process grayscale. More details you can find in $CAFFE_ROOT/python/caffe/io.py

\n

If you didn't use mean file during training you can set mean=None. By the way. "None" is a default value for mean.

\n

Best regards, Oleksandr.

\n

[quote=Data-Lover;65966]

\n

Senecaur

\n

in the deployment model: plankton_full_deploy.prototxt what is the value u are setting for the number of channels?

\n

Because our images are gray scale I am putting 1 but it is making an error. when I put 3 it worked

\n

is there any setting I should consider to overcome this problem?

\n

another question what is the rule of mean file? If I train the network without mean file mean_file: "average.png" is my solution is right. btw I tried the code to load average.png but didn't work

\n

please it is so important to get answer. I just one to do at least one submission using CNN

\n

[/quote]

", "rawMarkdown": "", "votes": null }, { "id": "65982", "postDate": "03/11/2015 20:54:58", "content": "

Thanks guys for respond

\n

I dont have problem with reading images in gray scale. I have problem with building the net

\n

net = caffe.Classifier(model..., depl...)

\n

in the train test phase i read the images as type: IMAGE_DATA the training passed very well 

\n

but when I come to deployment I set the channels 

\n

input_dim: 10
input_dim: 1
input_dim: 48
input_dim: 48

\n

the problem now is caffe complain that channel number should be 3 when I changed input_dim: 3

\n

it worked. I think it is as you said changing data to leveldb will solve the problem

\n

if you can post the code u use for converting images to levelDB and compute image mean it would be great. learning caffe in few days is a big headach.

", "rawMarkdown": "", "votes": null }, { "id": "66002", "postDate": "03/12/2015 05:42:02", "content": "

try this:

\n

https://github.com/BVLC/caffe/issues/391

", "rawMarkdown": "", "votes": null }, { "id": "66078", "postDate": "03/13/2015 04:58:35", "content": "

Hi, may i know what is your last layer? Is it softmax, softmax seems gives 1s 0s only.

\n\n

[quote=Senecaur;65923]

\n

I do something like this:

\n

import caffe

\n

import os

\n

MODEL_FILE = 'plankton_full_deploy.prototxt'

\n

PRETRAINED =  'checkpoints/plankton_full_iter_1000000.caffemodel'

\n

TESTPATH = '../data/test/'

\n

blob = caffe.proto.caffe_pb2.BlobProto()

\n

data = open('../data/test_mean.binaryproto','rb').read()

\n

blob.ParseFromString(data)

\n

arr = np.array(caffe.io.blobproto_to_array(blob))

\n

net = caffe.Classifier(MODEL_FILE,               

\n

                       PRETRAINED,                                    

\n

                       mean=arr[0], 

\n

                       raw_scale=255,   

\n

                       image_dims=(70,70))

\n

net.set_phase_test()net.set_mode_gpu()

\n

for root, dirs, files in os.walk(TESTPATH):     

\n

    for name in files:          

\n

        if name.endswith((".jpg")):                

\n

            image = caffe.io.load_image(os.path.join(root,name))

\n

            predictions = net.predict(image)                

\n

            out = [p for p in predictions]

\n

out will then be a list containing the class probabilities.  This will run slowly though as I have missed out some of the details on how I do this in batches of images, but this should be enough to show the basic method.  Good luck!

\n

[/quote]

", "rawMarkdown": "", "votes": null }, { "id": "66080", "postDate": "03/13/2015 05:28:44", "content": "

I am using this code and it is working well but the problem my network architecture is not good so am getting high error rate but at least it is working

\n

in training file

\n

layers
{
name: "loss"
type: SOFTMAX_LOSS

bottom: "fully3"
bottom: "label"
top: "loss"
}



layers
{
name: "accuracy"
type: ACCURACY

bottom: "fully3"
bottom: "label"
top: "accuracy"

include: { phase: TEST }
}

\n\n

in deployment file

\n

layers
{
name: "prob"
type: SOFTMAX

bottom: "fully3"
top: "prob"
}

", "rawMarkdown": "", "votes": null }, { "id": "66083", "postDate": "03/13/2015 07:00:54", "content": "

Thanks, I am going to try it out.

\n\n

[quote=Data-Lover;66080]

\n

I am using this code and it is working well but the problem my network architecture is not good so am getting high error rate but at least it is working

\n

in training file

\n

layers
{
name: "loss"
type: SOFTMAX_LOSS

bottom: "fully3"
bottom: "label"
top: "loss"
}



layers
{
name: "accuracy"
type: ACCURACY

bottom: "fully3"
bottom: "label"
top: "accuracy"

include: { phase: TEST }
}

\n

in deployment file

\n

layers
{
name: "prob"
type: SOFTMAX

bottom: "fully3"
top: "prob"
}

\n

[/quote]

", "rawMarkdown": "", "votes": null }, { "id": "161361", "postDate": "02/13/2017 10:59:43", "content": "

I am getting very slow iteration rate.I can use only cpu mode please help.\n Iteration 0 (-5.62061e-42 iter/s, 6071.5s/50 iters), loss = 0.838223\nUsing 4cpu 23gbram google cloud server.

", "rawMarkdown": "I am getting very slow iteration rate.I can use only cpu mode please help.\n Iteration 0 (-5.62061e-42 iter/s, 6071.5s/50 iters), loss = 0.838223\nUsing 4cpu 23gbram google cloud server.", "votes": null } ], "comments": [ { "id": 65923, "author_name": "senecaur", "author_url": "", "post_date": "03/10/2015 22:57:00", "content": "

I do something like this:

\n

import caffe

\n

import os

\n

MODEL_FILE = 'plankton_full_deploy.prototxt'

\n

PRETRAINED =  'checkpoints/plankton_full_iter_1000000.caffemodel'

\n

TESTPATH = '../data/test/'

\n

blob = caffe.proto.caffe_pb2.BlobProto()

\n

data = open('../data/test_mean.binaryproto','rb').read()

\n

blob.ParseFromString(data)

\n

arr = np.array(caffe.io.blobproto_to_array(blob))

\n

net = caffe.Classifier(MODEL_FILE,               

\n

                       PRETRAINED,                                    

\n

                       mean=arr[0], 

\n

                       raw_scale=255,   

\n

                       image_dims=(70,70))

\n

net.set_phase_test()net.set_mode_gpu()

\n

for root, dirs, files in os.walk(TESTPATH):     

\n

    for name in files:          

\n

        if name.endswith((".jpg")):                

\n

            image = caffe.io.load_image(os.path.join(root,name))

\n

            predictions = net.predict(image)                

\n

            out = [p for p in predictions]

\n

out will then be a list containing the class probabilities.  This will run slowly though as I have missed out some of the details on how I do this in batches of images, but this should be enough to show the basic method.  Good luck!

", "votes": null, "replies": [] }, { "id": 65924, "author_name": "cydonia", "author_url": "", "post_date": "03/10/2015 23:04:07", "content": "

See

\n

http://nbviewer.ipython.org/github/BVLC/caffe/blob/master/examples/classification.ipynb

", "votes": null, "replies": [] }, { "id": 65925, "author_name": "", "author_url": "", "post_date": "03/10/2015 23:35:55", "content": "

Thanks guys for respond. I will try my best.

\n

Good Luck

", "votes": null, "replies": [] }, { "id": 65966, "author_name": "", "author_url": "", "post_date": "03/11/2015 16:53:15", "content": "

Senecaur

\n

in the deployment model: plankton_full_deploy.prototxt what is the value u are setting for the number of channels?

\n

Because our images are gray scale I am putting 1 but it is making an error. when I put 3 it worked

\n

is there any setting I should consider to overcome this problem?

\n

another question what is the rule of mean file? If I train the network without mean file mean_file: "average.png" is my solution is right. btw I tried the code to load average.png but didn't work

\n\n

please it is so important to get answer. I just one to do at least one submission using CNN

", "votes": null, "replies": [] }, { "id": 65973, "author_name": "", "author_url": "", "post_date": "03/11/2015 19:02:45", "content": "

Hi,

\n

I also had an error about channels not matching first. In https://github.com/BVLC/caffe/issues/462 you find a modification for the python wrapper to handle one-channel input. If you create your leveldb/lmdb database with the convert_imageset function, you can pass a "-gray" flag to treat everything as grayscale. Using those two solved the channel error for me.

\n

Best, Matthias

", "votes": null, "replies": [] }, { "id": 65975, "author_name": "obaiev", "author_url": "", "post_date": "03/11/2015 19:28:00", "content": "

Hi,

\n

Function  caffe.io.load_image(os.path.join(root,name)) return image with 3 channels by default. You can use parameter color=False in order to process grayscale. More details you can find in $CAFFE_ROOT/python/caffe/io.py

\n

If you didn't use mean file during training you can set mean=None. By the way. "None" is a default value for mean.

\n

Best regards, Oleksandr.

\n

[quote=Data-Lover;65966]

\n

Senecaur

\n

in the deployment model: plankton_full_deploy.prototxt what is the value u are setting for the number of channels?

\n

Because our images are gray scale I am putting 1 but it is making an error. when I put 3 it worked

\n

is there any setting I should consider to overcome this problem?

\n

another question what is the rule of mean file? If I train the network without mean file mean_file: "average.png" is my solution is right. btw I tried the code to load average.png but didn't work

\n

please it is so important to get answer. I just one to do at least one submission using CNN

\n

[/quote]

", "votes": null, "replies": [] }, { "id": 65982, "author_name": "", "author_url": "", "post_date": "03/11/2015 20:54:58", "content": "

Thanks guys for respond

\n

I dont have problem with reading images in gray scale. I have problem with building the net

\n

net = caffe.Classifier(model..., depl...)

\n

in the train test phase i read the images as type: IMAGE_DATA the training passed very well 

\n

but when I come to deployment I set the channels 

\n

input_dim: 10
input_dim: 1
input_dim: 48
input_dim: 48

\n

the problem now is caffe complain that channel number should be 3 when I changed input_dim: 3

\n

it worked. I think it is as you said changing data to leveldb will solve the problem

\n

if you can post the code u use for converting images to levelDB and compute image mean it would be great. learning caffe in few days is a big headach.

", "votes": null, "replies": [] }, { "id": 66002, "author_name": "zerozero", "author_url": "", "post_date": "03/12/2015 05:42:02", "content": "

try this:

\n

https://github.com/BVLC/caffe/issues/391

", "votes": null, "replies": [] }, { "id": 66078, "author_name": "stevendu", "author_url": "", "post_date": "03/13/2015 04:58:35", "content": "

Hi, may i know what is your last layer? Is it softmax, softmax seems gives 1s 0s only.

\n\n

[quote=Senecaur;65923]

\n

I do something like this:

\n

import caffe

\n

import os

\n

MODEL_FILE = 'plankton_full_deploy.prototxt'

\n

PRETRAINED =  'checkpoints/plankton_full_iter_1000000.caffemodel'

\n

TESTPATH = '../data/test/'

\n

blob = caffe.proto.caffe_pb2.BlobProto()

\n

data = open('../data/test_mean.binaryproto','rb').read()

\n

blob.ParseFromString(data)

\n

arr = np.array(caffe.io.blobproto_to_array(blob))

\n

net = caffe.Classifier(MODEL_FILE,               

\n

                       PRETRAINED,                                    

\n

                       mean=arr[0], 

\n

                       raw_scale=255,   

\n

                       image_dims=(70,70))

\n

net.set_phase_test()net.set_mode_gpu()

\n

for root, dirs, files in os.walk(TESTPATH):     

\n

    for name in files:          

\n

        if name.endswith((".jpg")):                

\n

            image = caffe.io.load_image(os.path.join(root,name))

\n

            predictions = net.predict(image)                

\n

            out = [p for p in predictions]

\n

out will then be a list containing the class probabilities.  This will run slowly though as I have missed out some of the details on how I do this in batches of images, but this should be enough to show the basic method.  Good luck!

\n

[/quote]

", "votes": null, "replies": [] }, { "id": 66080, "author_name": "", "author_url": "", "post_date": "03/13/2015 05:28:44", "content": "

I am using this code and it is working well but the problem my network architecture is not good so am getting high error rate but at least it is working

\n

in training file

\n

layers
{
name: "loss"
type: SOFTMAX_LOSS

bottom: "fully3"
bottom: "label"
top: "loss"
}



layers
{
name: "accuracy"
type: ACCURACY

bottom: "fully3"
bottom: "label"
top: "accuracy"

include: { phase: TEST }
}

\n\n

in deployment file

\n

layers
{
name: "prob"
type: SOFTMAX

bottom: "fully3"
top: "prob"
}

", "votes": null, "replies": [] }, { "id": 66083, "author_name": "stevendu", "author_url": "", "post_date": "03/13/2015 07:00:54", "content": "

Thanks, I am going to try it out.

\n\n

[quote=Data-Lover;66080]

\n

I am using this code and it is working well but the problem my network architecture is not good so am getting high error rate but at least it is working

\n

in training file

\n

layers
{
name: "loss"
type: SOFTMAX_LOSS

bottom: "fully3"
bottom: "label"
top: "loss"
}



layers
{
name: "accuracy"
type: ACCURACY

bottom: "fully3"
bottom: "label"
top: "accuracy"

include: { phase: TEST }
}

\n

in deployment file

\n

layers
{
name: "prob"
type: SOFTMAX

bottom: "fully3"
top: "prob"
}

\n

[/quote]

", "votes": null, "replies": [] }, { "id": 161361, "author_name": "karandeepdps", "author_url": "", "post_date": "02/13/2017 10:59:43", "content": "

I am getting very slow iteration rate.I can use only cpu mode please help.\n Iteration 0 (-5.62061e-42 iter/s, 6071.5s/50 iters), loss = 0.838223\nUsing 4cpu 23gbram google cloud server.

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "65920": "", "65923": "", "65924": "", "65925": "", "65966": "", "65973": "", "65975": "", "65982": "", "66002": "", "66078": "", "66080": "", "66083": "", "161361": "I am getting very slow iteration rate.I can use only cpu mode please help.\n Iteration 0 (-5.62061e-42 iter/s, 6071.5s/50 iters), loss = 0.838223\nUsing 4cpu 23gbram google cloud server." }, "source": "meta" }