{ "id": 7317, "title": "Beat the Correlation Benchmark ~0.87337 AUC [Python]", "url": "/competitions/connectomics/discussion/7317", "author_name": "", "post_date": "2014-03-08T19:00:28.227Z", "votes": 6, "comment_count": 9, "views": 2615, "content": "

I wrote Python scripts to parse the data (brainparse.py) and to make predictions (model.py) based on Pearson Correlation with Discretization.

\n

It scores ~0.87337 AUC (13th) on the leaderboard and thus beats the provided Correlation Benchmark.

\n

You can find full model code and a description at my new site: http://mlwave.com/kaggle-connectomics-python-benchmark-code/

\n

If something is unclear, please ask. The code runs in under 6 hours 38 minutes 30 minutes on the validation set (1000 neurons) on my laptop.

\n

Next to the benchmark two scripts are provided to visualize and animate the neural networks.

\n

", "messages": [ { "id": "40025", "postDate": "03/08/2014 19:00:28", "content": "

I wrote Python scripts to parse the data (brainparse.py) and to make predictions (model.py) based on Pearson Correlation with Discretization.

\n

It scores ~0.87337 AUC (13th) on the leaderboard and thus beats the provided Correlation Benchmark.

\n

You can find full model code and a description at my new site: http://mlwave.com/kaggle-connectomics-python-benchmark-code/

\n

If something is unclear, please ask. The code runs in under 6 hours 38 minutes 30 minutes on the validation set (1000 neurons) on my laptop.

\n

Next to the benchmark two scripts are provided to visualize and animate the neural networks.

\n

", "rawMarkdown": "", "votes": null }, { "id": "40029", "postDate": "03/09/2014 00:29:46", "content": "

Excellent stuff, Triskelion! Great blog and code, thanks!

", "rawMarkdown": "", "votes": null }, { "id": "40065", "postDate": "03/10/2014 00:27:27", "content": "

I tried to improve the score with PyEntropy. Ran it with the pt-method (Panzeri-Treves bias correction). Took under 20 hours (10 hours wrapping code in function) and did not improve the Correlation Benchmark (but close).

\n

Looks like an interesting and related package: PyEntropy project page and I may try other methods/approaches with this to get something more similar to the GTE benchmark in Python.

\n

s = DiscreteSystem(neuron_activities[neuron_i_id],(1,100), neuron_activities[neuron_j_id],(1,100))
s.calculate_entropies(method='pt', calc=['HX', 'HXY'])
corr = str(s.I())

\n

I tried (1,1000) but execution time would be multiple days.

", "rawMarkdown": "", "votes": null }, { "id": "40072", "postDate": "03/10/2014 08:42:49", "content": "

Created tremendous speed-up by using numpy arrays over dictionaries/lists.

\n

The Python correlation benchmark now runs in 38 minutes on a laptop over 1000 neurons (1 million connection estimations). It uses less than 1 GB of memory. 

\n

To quickly update brainparse.py change:

\n

discretized[k] = [1 if f > threshold else 0 for f in list(np.diff(v,axis=0))]

\n

to:

\n

discretized[k] = np.array([1 if f > threshold else 0 for f in list(np.diff(v,axis=0))])

\n

or download the updated version at https://github.com/MLWave/Kaggle_Connectomics

\n

I'm still looking for further speedups. Also to make the code run on multiple machines at once..

", "rawMarkdown": "", "votes": null }, { "id": "40106", "postDate": "03/10/2014 17:33:34", "content": "

I tried using information-dynamics-toolkit , which is written in Java, via JPype, but it's just too slow to be practical. I suspect plenty of time is lost on converting Python (numpy) arrays to Java. I guess rewriting the main program in Java would make much more sense in order to use the library but I'm personally not very keen on it.

\n

Later I found bctpy which is a Python port of the Brain Connectivity Toolbox (Matlab library). I haven't had time to try to use it yet, but it looks promising. It's a beta quality software, according to its author, and still under active development, apparently.

", "rawMarkdown": "", "votes": null }, { "id": "40116", "postDate": "03/10/2014 19:59:56", "content": "

Running a XOR on the descretized time series you can beat the plain correlation benchmark in a few minutes ~0.83686.

\n

a = [1,0,0,1]

\n

b = [1,0,1,1]

\n

XOR score a_b = 3/4 = 0.75

", "rawMarkdown": "", "votes": null }, { "id": "40126", "postDate": "03/10/2014 23:42:16", "content": "

This is an interesting remark.

\n

If I understand it correctly you use the (complement of the) XOR on the binarized signal as an ultra-fast implementation of a coincidence detector. In this sense you expect to fall close to correlation with discretization, which is also very related to a count of coincidences, taking into account however the actual rates of different neurons:

\n

CC (S_i, S_j) = <S_i S_j> - <S_i><S_j>

\n

The XOR detector lacks this correction term, therefore I am not surprised that it performs WORSE than correlation with discretization. Is performance is largely due just to the discretization itself, in the same way in which discretization boosts performance of Pearson correlation with respect to the non-discretized case.

", "rawMarkdown": "", "votes": null }, { "id": "40128", "postDate": "03/11/2014 00:09:50", "content": "

Please also note: the goal of the challenge is not only to improve on correlation and evaluate statistical dependencies that hint on non-directed connectivity. The goal is also to find the directions of the connections. Solving the orientation problem will probably differentiate the top ranking participants.

", "rawMarkdown": "", "votes": null }, { "id": "40158", "postDate": "03/11/2014 10:19:22", "content": "

Using stats.pointbiserialr over stats.pearsonr you get the same results, but in 30 minutes.

\n

Twiddling with the binary threshold can improve the benchmark score.

\n

Using np.correlate the script runs in 5 minutes (from raw data files to submission) and gives ~0.83 AUC. XOR was faster and performed a little better.

\n

Now working on local validation.

", "rawMarkdown": "", "votes": null }, { "id": "40305", "postDate": "03/13/2014 21:45:00", "content": "

If you change the threshold to 0.127 you can increase the benchmark score. Locally you can find this value by a brute force search, but every set/neural net seems to give a higher AUC with a different threshold. There is no guarantee that the 0.127 threshold will also increase the private leaderboard score. I am trying to find a good average threshold that will work for most neural nets. I suspect it is somewhere near 0.12.

\n

Tried smoothing/denoising the signal with numpy.convolve (moving average) and http://wiki.scipy.org/Cookbook/SignalSmooth. Was able to produce reasonable graphs like below, but didn't improve the correlation score yet. Are any other competitors trying smoothing methods over binary discretization? Is this path leading anywhere?

\n

\"denoised

", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 40029, "author_name": "piotrkuchta", "author_url": "", "post_date": "03/09/2014 00:29:46", "content": "

Excellent stuff, Triskelion! Great blog and code, thanks!

", "votes": null, "replies": [] }, { "id": 40065, "author_name": "triskelion", "author_url": "", "post_date": "03/10/2014 00:27:27", "content": "

I tried to improve the score with PyEntropy. Ran it with the pt-method (Panzeri-Treves bias correction). Took under 20 hours (10 hours wrapping code in function) and did not improve the Correlation Benchmark (but close).

\n

Looks like an interesting and related package: PyEntropy project page and I may try other methods/approaches with this to get something more similar to the GTE benchmark in Python.

\n

s = DiscreteSystem(neuron_activities[neuron_i_id],(1,100), neuron_activities[neuron_j_id],(1,100))
s.calculate_entropies(method='pt', calc=['HX', 'HXY'])
corr = str(s.I())

\n

I tried (1,1000) but execution time would be multiple days.

", "votes": null, "replies": [] }, { "id": 40072, "author_name": "triskelion", "author_url": "", "post_date": "03/10/2014 08:42:49", "content": "

Created tremendous speed-up by using numpy arrays over dictionaries/lists.

\n

The Python correlation benchmark now runs in 38 minutes on a laptop over 1000 neurons (1 million connection estimations). It uses less than 1 GB of memory. 

\n

To quickly update brainparse.py change:

\n

discretized[k] = [1 if f > threshold else 0 for f in list(np.diff(v,axis=0))]

\n

to:

\n

discretized[k] = np.array([1 if f > threshold else 0 for f in list(np.diff(v,axis=0))])

\n

or download the updated version at https://github.com/MLWave/Kaggle_Connectomics

\n

I'm still looking for further speedups. Also to make the code run on multiple machines at once..

", "votes": null, "replies": [] }, { "id": 40106, "author_name": "piotrkuchta", "author_url": "", "post_date": "03/10/2014 17:33:34", "content": "

I tried using information-dynamics-toolkit , which is written in Java, via JPype, but it's just too slow to be practical. I suspect plenty of time is lost on converting Python (numpy) arrays to Java. I guess rewriting the main program in Java would make much more sense in order to use the library but I'm personally not very keen on it.

\n

Later I found bctpy which is a Python port of the Brain Connectivity Toolbox (Matlab library). I haven't had time to try to use it yet, but it looks promising. It's a beta quality software, according to its author, and still under active development, apparently.

", "votes": null, "replies": [] }, { "id": 40116, "author_name": "triskelion", "author_url": "", "post_date": "03/10/2014 19:59:56", "content": "

Running a XOR on the descretized time series you can beat the plain correlation benchmark in a few minutes ~0.83686.

\n

a = [1,0,0,1]

\n

b = [1,0,1,1]

\n

XOR score a_b = 3/4 = 0.75

", "votes": null, "replies": [] }, { "id": 40126, "author_name": "salviati1", "author_url": "", "post_date": "03/10/2014 23:42:16", "content": "

This is an interesting remark.

\n

If I understand it correctly you use the (complement of the) XOR on the binarized signal as an ultra-fast implementation of a coincidence detector. In this sense you expect to fall close to correlation with discretization, which is also very related to a count of coincidences, taking into account however the actual rates of different neurons:

\n

CC (S_i, S_j) = <S_i S_j> - <S_i><S_j>

\n

The XOR detector lacks this correction term, therefore I am not surprised that it performs WORSE than correlation with discretization. Is performance is largely due just to the discretization itself, in the same way in which discretization boosts performance of Pearson correlation with respect to the non-discretized case.

", "votes": null, "replies": [] }, { "id": 40128, "author_name": "iguyon", "author_url": "", "post_date": "03/11/2014 00:09:50", "content": "

Please also note: the goal of the challenge is not only to improve on correlation and evaluate statistical dependencies that hint on non-directed connectivity. The goal is also to find the directions of the connections. Solving the orientation problem will probably differentiate the top ranking participants.

", "votes": null, "replies": [] }, { "id": 40158, "author_name": "triskelion", "author_url": "", "post_date": "03/11/2014 10:19:22", "content": "

Using stats.pointbiserialr over stats.pearsonr you get the same results, but in 30 minutes.

\n

Twiddling with the binary threshold can improve the benchmark score.

\n

Using np.correlate the script runs in 5 minutes (from raw data files to submission) and gives ~0.83 AUC. XOR was faster and performed a little better.

\n

Now working on local validation.

", "votes": null, "replies": [] }, { "id": 40305, "author_name": "triskelion", "author_url": "", "post_date": "03/13/2014 21:45:00", "content": "

If you change the threshold to 0.127 you can increase the benchmark score. Locally you can find this value by a brute force search, but every set/neural net seems to give a higher AUC with a different threshold. There is no guarantee that the 0.127 threshold will also increase the private leaderboard score. I am trying to find a good average threshold that will work for most neural nets. I suspect it is somewhere near 0.12.

\n

Tried smoothing/denoising the signal with numpy.convolve (moving average) and http://wiki.scipy.org/Cookbook/SignalSmooth. Was able to produce reasonable graphs like below, but didn't improve the correlation score yet. Are any other competitors trying smoothing methods over binary discretization? Is this path leading anywhere?

\n

\"denoised

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "40025": "", "40029": "", "40065": "", "40072": "", "40106": "", "40116": "", "40126": "", "40128": "", "40158": "", "40305": "" }, "source": "meta" }