{ "id": 8036, "title": "Py-oopsi: the python implementation of the fast-oopsi algorithm", "url": "/competitions/connectomics/discussion/8036", "author_name": "", "post_date": "2014-05-06T03:54:32.860Z", "votes": 1, "comment_count": 7, "views": 2556, "content": "

Hi, all,

\n

I implemented the fast oopsi algorithm (developed by J. Vogelstein in 2009), which was used to infer neuron spikes from noisy calcium fluorescence, in python! You may obtain the code on the github page:

\n

https://github.com/liubenyuan/py-oopsi

\n

I think it might be too late for this competition :(

\n

Any suggestions, comments on the algorithm, coding styles are all welcome.

\n

\"demo\"

\n

Liu Benyuan

", "messages": [ { "id": "43906", "postDate": "05/06/2014 03:54:32", "content": "

Hi, all,

\n

I implemented the fast oopsi algorithm (developed by J. Vogelstein in 2009), which was used to infer neuron spikes from noisy calcium fluorescence, in python! You may obtain the code on the github page:

\n

https://github.com/liubenyuan/py-oopsi

\n

I think it might be too late for this competition :(

\n

Any suggestions, comments on the algorithm, coding styles are all welcome.

\n

\"demo\"

\n

Liu Benyuan

", "rawMarkdown": "", "votes": null }, { "id": "43914", "postDate": "05/06/2014 04:39:02", "content": "

Below is a demo to extract spikes from calcium fluorescence signals. Data set is small_04, where the ground truth says that ([9,19,23,31,36,46,60,69]) connected to [34]. A segment data (from 152390:153190) is shown. Py-oopsi can also be used to extract the simulataneously bursts from the mean fluorescence trace of all neurons. 

\n

\"Connectomics

", "rawMarkdown": "", "votes": null }, { "id": "43988", "postDate": "05/06/2014 12:31:56", "content": "

Great !!

\n

The challenge just ended, so it might be a bit late to use it in there, but I will certainly take a look at the code.

\n

I think some participants also used spike inference methods in the challenge, it will be good to check and compare the different methods on the same datasets.

", "rawMarkdown": "", "votes": null }, { "id": "44214", "postDate": "05/07/2014 21:28:29", "content": "

Well, we also ended up writing a Python version of fast non-negative deconvolution, which is now freely available here. We found that we could get nice performance gains by wrapping LAPACK's `dgtsv` tridiagonal solver to compute the direction for each Newton step, rather than using standard LU factorization such as with scipy's `spsolve`.

", "rawMarkdown": "", "votes": null }, { "id": "44225", "postDate": "05/08/2014 01:52:55", "content": "

Great !

\n

Mime was a simply, bare bone implementation of the fast-oopsi algorithm, where only subroutine "initialize parameters", "MAP estimate", "estimate parameters" are used. I do not use dgtsv which might bring performance improvements. I found "test_trisolve" func in your code, have you test the ratio of speed up using trisolve and sparse.linalg.spsolve ?

\n

And one more question, how do you use the spikes inferred from the calcium trace ?

", "rawMarkdown": "", "votes": null }, { "id": "44282", "postDate": "05/09/2014 00:03:27", "content": "

I've added a simple benchmarking script to the _tridiag_solvers module. Here's the speed up for a range of different array sizes:

\n

\"Benchmarking:

\n

In practice the difference is even greater, since dgtsv allows us to avoid the extra overhead involved in constructing/updating the sparse matrices that are required by spsolve (dgtsv just takes three 1D arrays for the matrix diagonals), and we can also solve in-place to avoid creating additional copies.

\n

As for your second question, we just thresholded the continuous estimate of spike probability that FNND returns. We did this based on the average expected firing rate for a neuron in this network model (we ended up running some spiking network simulations ourselves, using the code that the organisers provided here, so we had comparable datasets where we had access to the actual spike trains). We then basically just threw these discrete estimated spike trains at GTE, more or less as described in the Stetter paper.

", "rawMarkdown": "", "votes": null }, { "id": "44283", "postDate": "05/09/2014 00:15:31", "content": "

Cool !! I will surely learn more python coding from your implementation :)

\n

I do not think GTE might be a better choice, as the firings are sparse and the same-bin effect is not evident between simulataneously bursts. 

", "rawMarkdown": "", "votes": null }, { "id": "44284", "postDate": "05/09/2014 00:32:50", "content": "

True, I don't think that there's a great deal of mileage in GTE, but it was the most robust method we were able to implement in time. We were playing around with a few other information theoretic measures which consider multiple nodes rather than just pairs, but it's very tricky to get these to work robustly, since the dimensionality over which you have to compute conditional probabilities tends to explode. As far as the bursts are concerned, I think that there's really not much information that can be obtained during these time bins (at least not without a much higher sample rate).

", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 43914, "author_name": "liubenyuan", "author_url": "", "post_date": "05/06/2014 04:39:02", "content": "

Below is a demo to extract spikes from calcium fluorescence signals. Data set is small_04, where the ground truth says that ([9,19,23,31,36,46,60,69]) connected to [34]. A segment data (from 152390:153190) is shown. Py-oopsi can also be used to extract the simulataneously bursts from the mean fluorescence trace of all neurons. 

\n

\"Connectomics

", "votes": null, "replies": [] }, { "id": 43988, "author_name": "orlandi", "author_url": "", "post_date": "05/06/2014 12:31:56", "content": "

Great !!

\n

The challenge just ended, so it might be a bit late to use it in there, but I will certainly take a look at the code.

\n

I think some participants also used spike inference methods in the challenge, it will be good to check and compare the different methods on the same datasets.

", "votes": null, "replies": [] }, { "id": 44214, "author_name": "alistairmuldal", "author_url": "", "post_date": "05/07/2014 21:28:29", "content": "

Well, we also ended up writing a Python version of fast non-negative deconvolution, which is now freely available here. We found that we could get nice performance gains by wrapping LAPACK's `dgtsv` tridiagonal solver to compute the direction for each Newton step, rather than using standard LU factorization such as with scipy's `spsolve`.

", "votes": null, "replies": [] }, { "id": 44225, "author_name": "liubenyuan", "author_url": "", "post_date": "05/08/2014 01:52:55", "content": "

Great !

\n

Mime was a simply, bare bone implementation of the fast-oopsi algorithm, where only subroutine "initialize parameters", "MAP estimate", "estimate parameters" are used. I do not use dgtsv which might bring performance improvements. I found "test_trisolve" func in your code, have you test the ratio of speed up using trisolve and sparse.linalg.spsolve ?

\n

And one more question, how do you use the spikes inferred from the calcium trace ?

", "votes": null, "replies": [] }, { "id": 44282, "author_name": "alistairmuldal", "author_url": "", "post_date": "05/09/2014 00:03:27", "content": "

I've added a simple benchmarking script to the _tridiag_solvers module. Here's the speed up for a range of different array sizes:

\n

\"Benchmarking:

\n

In practice the difference is even greater, since dgtsv allows us to avoid the extra overhead involved in constructing/updating the sparse matrices that are required by spsolve (dgtsv just takes three 1D arrays for the matrix diagonals), and we can also solve in-place to avoid creating additional copies.

\n

As for your second question, we just thresholded the continuous estimate of spike probability that FNND returns. We did this based on the average expected firing rate for a neuron in this network model (we ended up running some spiking network simulations ourselves, using the code that the organisers provided here, so we had comparable datasets where we had access to the actual spike trains). We then basically just threw these discrete estimated spike trains at GTE, more or less as described in the Stetter paper.

", "votes": null, "replies": [] }, { "id": 44283, "author_name": "liubenyuan", "author_url": "", "post_date": "05/09/2014 00:15:31", "content": "

Cool !! I will surely learn more python coding from your implementation :)

\n

I do not think GTE might be a better choice, as the firings are sparse and the same-bin effect is not evident between simulataneously bursts. 

", "votes": null, "replies": [] }, { "id": 44284, "author_name": "alistairmuldal", "author_url": "", "post_date": "05/09/2014 00:32:50", "content": "

True, I don't think that there's a great deal of mileage in GTE, but it was the most robust method we were able to implement in time. We were playing around with a few other information theoretic measures which consider multiple nodes rather than just pairs, but it's very tricky to get these to work robustly, since the dimensionality over which you have to compute conditional probabilities tends to explode. As far as the bursts are concerned, I think that there's really not much information that can be obtained during these time bins (at least not without a much higher sample rate).

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "43906": "", "43914": "", "43988": "", "44214": "", "44225": "", "44282": "", "44283": "", "44284": "" }, "source": "meta" }