{ "id": 7011, "title": "Data and code forum wiki", "url": "/competitions/connectomics/discussion/7011", "author_name": "", "post_date": "2014-02-06T22:19:55.293Z", "votes": 7, "comment_count": 2, "views": 1924, "content": "

This competition has both a lot of data and seems high on the "requires domain expertise" scale. For these reasons I think it is a good idea to keep information and insights about data in one central place. I usually make this competition documentation for myself, but my first experiment to publish a public wiki on the forums was appreciated. I usually aim to compress as much of the posts, but since there is little information on the forums yet, I am going to rant a sparse bag-of-words and edit it out later.

\n

I don't know Matlab (yet) nor do I have a license. So I am in the process of porting the example Correlation script to Python. I saw similar efforts to port to Python were planned, but not yet available, on the competition website. Since I do not know Matlab (and probably suck at Python) any help there is appreciated, so Pythonista's could get started. The SampleCode.zip is 80Mb alone and we have but 2 months to go. If you want to contribute, please do share code or insights (no matter the language (ok, please no regular expressions or foul language!)). 

\n

I think I also understood that there are more accurate (more computationally intensive) similarity measures hinted at in the example code. Is anyone planning to run the benchmark script with different, more demanding, settings?

\n

Neuron Activity Browser

\n

I've started work on plotting the neuron activity in Python (so you could visually browse neurons and their activities). This graph is an example of the progress.

\n

\n

If you want to build your own neuron browser, you could take a column (not a row) from "fluorescence_iNet1_Size100_CC03inh.txt" (found in small.tgz). The column index is the id of the neuron, so fifth column is fifth neuron, and its values are activity. The rows are the timeseries, so fifth row is the 5th point in time (i think 20ms times 5).

\n

Then (partially) plot these neurons on a graph, where y-axis equals neuron activity and x-axis equals the timeseries points. I plotted the first 2000 points. I am using matplotlib for this.

\n

Since we also have a position file, I wonder if it is possible to animate an entire net. I am trying this with PyGame now. Update! All glory to the hypnonetworks!:

\n

\n

showing activity for 100 neurons 30 time steps of 20ms interval, 5 x slowed. Real-time: http://i.imgur.com/bBegkvM.gif

\n

ROC AUC Evaluation Metric

\n

The competition's scoring metric is Area Under the ROC Curve.

\n

http://www.kaggle.com/c/connectomics/details/evaluation

\n

https://www.kaggle.com/wiki/AreaUnderCurve

\n

http://metaoptimize.com/qa/questions/988/simple-explanation-of-area-under-the-roc-curve

\n

http://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_auc_score.html

\n

In fact, you could get relatively easily AUCs of the order of 0.9. However, an AUC between 0.85 and 0.9 is still deceiving. Assume that you know what the average number of connections per neuron is and that you use this information to decide how many links to include in your reconstructed network. So you select a specific point on the ROC curve, associated to specific true and false positive ratios (TPR and FPR = 1-TPR).

\n

Let's suppose you have a network with N=100 neurons and k=12 connections per neuron on average. So, excluding diagonal terms you have 9900 possible entries in the connectivity matrix, and only ~1200 correspond to real connections. If you select a point with a sensitivity=TPR=0.7 for a (1-specificity)=FPR=0.1 you are looking at a subset of 1830 links, of which 840 are correctly identified and 990 wrong! So there is still a lot of room for improvement, since more than 50% of the included links is still wrong!

\n

The larger the AUC you will get, the less inference mistakes you will make. But the overall performance is not the only aspect that matters. You might have methods which have a bad overall performance, because of some systematic error, but that still make a good job in detecting specific features, such as the level of clustering of the network, or the presence of certain graph motifs, etc.

\n

All these things matter for neuroscience.

\n

Correlation

\n

I am trying http://docs.scipy.org/doc/numpy/reference/generated/numpy.correlate.html with good results. The average of neuron activities with a connection has a substantially higher (or lower) correlation than neuron activities without a connection. The competition website lists specific and more advanced correlation measures. 

\n

Contributors/sourcing (use Google to find the posts and threads):

\n

http://www.kaggle.com/users/163679/sagredo

\n

http://www.kaggle.com/users/163673/salviati

", "messages": [ { "id": "38458", "postDate": "02/06/2014 22:19:55", "content": "

This competition has both a lot of data and seems high on the "requires domain expertise" scale. For these reasons I think it is a good idea to keep information and insights about data in one central place. I usually make this competition documentation for myself, but my first experiment to publish a public wiki on the forums was appreciated. I usually aim to compress as much of the posts, but since there is little information on the forums yet, I am going to rant a sparse bag-of-words and edit it out later.

\n

I don't know Matlab (yet) nor do I have a license. So I am in the process of porting the example Correlation script to Python. I saw similar efforts to port to Python were planned, but not yet available, on the competition website. Since I do not know Matlab (and probably suck at Python) any help there is appreciated, so Pythonista's could get started. The SampleCode.zip is 80Mb alone and we have but 2 months to go. If you want to contribute, please do share code or insights (no matter the language (ok, please no regular expressions or foul language!)). 

\n

I think I also understood that there are more accurate (more computationally intensive) similarity measures hinted at in the example code. Is anyone planning to run the benchmark script with different, more demanding, settings?

\n

Neuron Activity Browser

\n

I've started work on plotting the neuron activity in Python (so you could visually browse neurons and their activities). This graph is an example of the progress.

\n

\n

If you want to build your own neuron browser, you could take a column (not a row) from "fluorescence_iNet1_Size100_CC03inh.txt" (found in small.tgz). The column index is the id of the neuron, so fifth column is fifth neuron, and its values are activity. The rows are the timeseries, so fifth row is the 5th point in time (i think 20ms times 5).

\n

Then (partially) plot these neurons on a graph, where y-axis equals neuron activity and x-axis equals the timeseries points. I plotted the first 2000 points. I am using matplotlib for this.

\n

Since we also have a position file, I wonder if it is possible to animate an entire net. I am trying this with PyGame now. Update! All glory to the hypnonetworks!:

\n

\n

showing activity for 100 neurons 30 time steps of 20ms interval, 5 x slowed. Real-time: http://i.imgur.com/bBegkvM.gif

\n

ROC AUC Evaluation Metric

\n

The competition's scoring metric is Area Under the ROC Curve.

\n

http://www.kaggle.com/c/connectomics/details/evaluation

\n

https://www.kaggle.com/wiki/AreaUnderCurve

\n

http://metaoptimize.com/qa/questions/988/simple-explanation-of-area-under-the-roc-curve

\n

http://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_auc_score.html

\n

In fact, you could get relatively easily AUCs of the order of 0.9. However, an AUC between 0.85 and 0.9 is still deceiving. Assume that you know what the average number of connections per neuron is and that you use this information to decide how many links to include in your reconstructed network. So you select a specific point on the ROC curve, associated to specific true and false positive ratios (TPR and FPR = 1-TPR).

\n

Let's suppose you have a network with N=100 neurons and k=12 connections per neuron on average. So, excluding diagonal terms you have 9900 possible entries in the connectivity matrix, and only ~1200 correspond to real connections. If you select a point with a sensitivity=TPR=0.7 for a (1-specificity)=FPR=0.1 you are looking at a subset of 1830 links, of which 840 are correctly identified and 990 wrong! So there is still a lot of room for improvement, since more than 50% of the included links is still wrong!

\n

The larger the AUC you will get, the less inference mistakes you will make. But the overall performance is not the only aspect that matters. You might have methods which have a bad overall performance, because of some systematic error, but that still make a good job in detecting specific features, such as the level of clustering of the network, or the presence of certain graph motifs, etc.

\n

All these things matter for neuroscience.

\n

Correlation

\n

I am trying http://docs.scipy.org/doc/numpy/reference/generated/numpy.correlate.html with good results. The average of neuron activities with a connection has a substantially higher (or lower) correlation than neuron activities without a connection. The competition website lists specific and more advanced correlation measures. 

\n

Contributors/sourcing (use Google to find the posts and threads):

\n

http://www.kaggle.com/users/163679/sagredo

\n

http://www.kaggle.com/users/163673/salviati

", "rawMarkdown": "", "votes": null }, { "id": "38559", "postDate": "02/10/2014 15:33:58", "content": "

HI,

\n

Looks great!

\n

One of the project advisors is porting the Matlab code to python, but no ETA yet.

\n

As for the visualization, yes it is possible. In fact, one of the Matlab codes does just that, but on a coarse-grained representation, not on the network itself; simulating what you obtain from the real experiments.

", "rawMarkdown": "", "votes": null }, { "id": "41807", "postDate": "04/12/2014 18:24:15", "content": "

I am not very good at Matlab,is there any R code for this challenge.

\n

Deepak Chaturvedi

", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 38559, "author_name": "orlandi", "author_url": "", "post_date": "02/10/2014 15:33:58", "content": "

HI,

\n

Looks great!

\n

One of the project advisors is porting the Matlab code to python, but no ETA yet.

\n

As for the visualization, yes it is possible. In fact, one of the Matlab codes does just that, but on a coarse-grained representation, not on the network itself; simulating what you obtain from the real experiments.

", "votes": null, "replies": [] }, { "id": 41807, "author_name": "deepakchaturvedi1", "author_url": "", "post_date": "04/12/2014 18:24:15", "content": "

I am not very good at Matlab,is there any R code for this challenge.

\n

Deepak Chaturvedi

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "38458": "", "38559": "", "41807": "" }, "source": "meta" }