Inverse cross variance seemed to be a really good feature. Here's a small Matlab script to get a decent score pretty fast.
", "messages": [ { "id": "44045", "postDate": "05/06/2014 15:03:52", "content": "Inverse cross variance seemed to be a really good feature. Here's a small Matlab script to get a decent score pretty fast.
", "rawMarkdown": "", "votes": null }, { "id": "44047", "postDate": "05/06/2014 15:08:05", "content": "Thanks for the code!
\n\nWhen you say feature does that imply you extracted a bunch of features for each possible connection and framed this as a classification problem?
", "rawMarkdown": "", "votes": null }, { "id": "44048", "postDate": "05/06/2014 15:13:13", "content": "I computed multiple features by evaluating the cross variance at different time lags and spike thresholds. So yes, this created a number of features per possible connection. Then I used standard ensemble methods for classification.
", "rawMarkdown": "", "votes": null }, { "id": "44051", "postDate": "05/06/2014 15:21:07", "content": "Great ! Have you inferred any model-free law from this single feature ? for example, when cov_inv > thd or cov_inv < thd, it can be inferred as a connection ?
", "rawMarkdown": "", "votes": null }, { "id": "44053", "postDate": "05/06/2014 15:30:02", "content": "Well, I tried using inverse covariance matrix when I just joined challenge since it's linked to partial correlation. But I gave up after a few trials and looked for other alternatives because I could not find significant improvement out of it. It seems that I should pay more time on this direction... I'll take a look at your script.
", "rawMarkdown": "", "votes": null }, { "id": "44065", "postDate": "05/06/2014 16:53:03", "content": "Inverse covariances are regularly used to estimate the structure of (multivariate Gaussian) undirected graphical models. By corollary to the classical Hammersley–Clifford theorem, it is well known theoretical result that zeros in the inverse matrix indicate absent edges in the network. However, this result can't be (directly) used to learn the directions of the edges. But I guess it wasn't that important for this competition?
\nOne widely used technique based on this theory is graphical lasso. Did somebody try that algorithm for this competition?
", "rawMarkdown": "", "votes": null }, { "id": "44066", "postDate": "05/06/2014 16:55:15", "content": "Yes, I first tried graphical lasso. But it actually had a worse score (at least for me). My guess for the reason is that the metric was AUC, and therefore you want every possible connection to have a score (rather than most getting a zero).
", "rawMarkdown": "", "votes": null }, { "id": "44068", "postDate": "05/06/2014 16:59:01", "content": "[quote=gaucho81;44066]
\nYes, I first tried graphical lasso. But it actually had a worse score (at least for me). My guess for the reason is that the metric was AUC, and therefore you want every possible connection to have a score (rather than most getting a zero).
\n[/quote]
\nYeah, I think that makes sense. Maybe some kind of bootstrapping could have worked? Ie. split the original dataset to many smaller ones and then run graphical lasso for each one of them and average the results.
", "rawMarkdown": "", "votes": null }, { "id": "44071", "postDate": "05/06/2014 17:04:39", "content": "You read my mind! I tried that too, by splitting the time series into chunks, running graphical lasso on each, and then taking mean or max over the chunks. Didn't have much luck with that, but it was towards the end of the competition and I didn't spend much time trying different parameters.
", "rawMarkdown": "", "votes": null }, { "id": "44072", "postDate": "05/06/2014 17:14:13", "content": "[quote=gaucho81;44071]
\nYou read my mind! I tried that too, by splitting the time series into chunks, running graphical lasso on each, and then taking mean or max over the chunks. Didn't have much luck with that, but it was towards the end of the competition and I didn't spend much time trying different parameters.
\n[/quote]
\nI agree. I also tried the same thing without much luck. I spent tons of time looking for a sparse solution but I did't achieve to improve my solution.
My approach was to use ICA-based LiNGAM
http://www.cs.helsinki.fi/group/neuroinf/lingam/
", "rawMarkdown": "", "votes": null }, { "id": "44074", "postDate": "05/06/2014 17:34:07", "content": "I guess that basically means that the data is not really (or even that close to) Gaussian and then (as far as I know) there isn't any theoretical guarantees for that inverse covariance / graphical lasso method.
\nLately there have been some studies using copulas for non-Gaussian data. (for example: http://repository.cmu.edu/cgi/viewcontent.cgi?article=1061&context=statistics) Have to read that about that ICA based approach as well.
\nDamn, too bad that I missed this one out. Seems to be a fun problem.
", "rawMarkdown": "", "votes": null }, { "id": "44075", "postDate": "05/06/2014 17:46:28", "content": "There was Gaussian noise added to the data, but the signal was discrete spikes where the probability of a spike depends on previous spikes at other nodes. So in that sense a Bernoulli model makes more sense. This paper discusses how to solve the graphical lasso problem for that type of data,
\nhttp://jmlr.org/papers/volume9/banerjee08a/banerjee08a.pdf
\nbut there was no open source software I could find and didn't have the time to try implementing it myself.
", "rawMarkdown": "", "votes": null }, { "id": "44079", "postDate": "05/06/2014 18:16:27", "content": "We will give more details later, but we basically did that as well. Lots of signal processing + estimates of the inverse of the covariance matrix to retrieve the direct connections.
", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 44047, "author_name": "dhammack", "author_url": "", "post_date": "05/06/2014 15:08:05", "content": "Thanks for the code!
\n\nWhen you say feature does that imply you extracted a bunch of features for each possible connection and framed this as a classification problem?
", "votes": null, "replies": [] }, { "id": 44048, "author_name": "george04", "author_url": "", "post_date": "05/06/2014 15:13:13", "content": "I computed multiple features by evaluating the cross variance at different time lags and spike thresholds. So yes, this created a number of features per possible connection. Then I used standard ensemble methods for classification.
", "votes": null, "replies": [] }, { "id": 44051, "author_name": "liubenyuan", "author_url": "", "post_date": "05/06/2014 15:21:07", "content": "Great ! Have you inferred any model-free law from this single feature ? for example, when cov_inv > thd or cov_inv < thd, it can be inferred as a connection ?
", "votes": null, "replies": [] }, { "id": 44053, "author_name": "killertom", "author_url": "", "post_date": "05/06/2014 15:30:02", "content": "Well, I tried using inverse covariance matrix when I just joined challenge since it's linked to partial correlation. But I gave up after a few trials and looked for other alternatives because I could not find significant improvement out of it. It seems that I should pay more time on this direction... I'll take a look at your script.
", "votes": null, "replies": [] }, { "id": 44065, "author_name": "herrahuu", "author_url": "", "post_date": "05/06/2014 16:53:03", "content": "Inverse covariances are regularly used to estimate the structure of (multivariate Gaussian) undirected graphical models. By corollary to the classical Hammersley–Clifford theorem, it is well known theoretical result that zeros in the inverse matrix indicate absent edges in the network. However, this result can't be (directly) used to learn the directions of the edges. But I guess it wasn't that important for this competition?
\nOne widely used technique based on this theory is graphical lasso. Did somebody try that algorithm for this competition?
", "votes": null, "replies": [] }, { "id": 44066, "author_name": "george04", "author_url": "", "post_date": "05/06/2014 16:55:15", "content": "Yes, I first tried graphical lasso. But it actually had a worse score (at least for me). My guess for the reason is that the metric was AUC, and therefore you want every possible connection to have a score (rather than most getting a zero).
", "votes": null, "replies": [] }, { "id": 44068, "author_name": "herrahuu", "author_url": "", "post_date": "05/06/2014 16:59:01", "content": "[quote=gaucho81;44066]
\nYes, I first tried graphical lasso. But it actually had a worse score (at least for me). My guess for the reason is that the metric was AUC, and therefore you want every possible connection to have a score (rather than most getting a zero).
\n[/quote]
\nYeah, I think that makes sense. Maybe some kind of bootstrapping could have worked? Ie. split the original dataset to many smaller ones and then run graphical lasso for each one of them and average the results.
", "votes": null, "replies": [] }, { "id": 44071, "author_name": "george04", "author_url": "", "post_date": "05/06/2014 17:04:39", "content": "You read my mind! I tried that too, by splitting the time series into chunks, running graphical lasso on each, and then taking mean or max over the chunks. Didn't have much luck with that, but it was towards the end of the competition and I didn't spend much time trying different parameters.
", "votes": null, "replies": [] }, { "id": 44072, "author_name": "lnicalo", "author_url": "", "post_date": "05/06/2014 17:14:13", "content": "[quote=gaucho81;44071]
\nYou read my mind! I tried that too, by splitting the time series into chunks, running graphical lasso on each, and then taking mean or max over the chunks. Didn't have much luck with that, but it was towards the end of the competition and I didn't spend much time trying different parameters.
\n[/quote]
\nI agree. I also tried the same thing without much luck. I spent tons of time looking for a sparse solution but I did't achieve to improve my solution.
My approach was to use ICA-based LiNGAM
http://www.cs.helsinki.fi/group/neuroinf/lingam/
", "votes": null, "replies": [] }, { "id": 44074, "author_name": "herrahuu", "author_url": "", "post_date": "05/06/2014 17:34:07", "content": "I guess that basically means that the data is not really (or even that close to) Gaussian and then (as far as I know) there isn't any theoretical guarantees for that inverse covariance / graphical lasso method.
\nLately there have been some studies using copulas for non-Gaussian data. (for example: http://repository.cmu.edu/cgi/viewcontent.cgi?article=1061&context=statistics) Have to read that about that ICA based approach as well.
\nDamn, too bad that I missed this one out. Seems to be a fun problem.
", "votes": null, "replies": [] }, { "id": 44075, "author_name": "george04", "author_url": "", "post_date": "05/06/2014 17:46:28", "content": "There was Gaussian noise added to the data, but the signal was discrete spikes where the probability of a spike depends on previous spikes at other nodes. So in that sense a Bernoulli model makes more sense. This paper discusses how to solve the graphical lasso problem for that type of data,
\nhttp://jmlr.org/papers/volume9/banerjee08a/banerjee08a.pdf
\nbut there was no open source software I could find and didn't have the time to try implementing it myself.
", "votes": null, "replies": [] }, { "id": 44079, "author_name": "glouppe", "author_url": "", "post_date": "05/06/2014 18:16:27", "content": "We will give more details later, but we basically did that as well. Lots of signal processing + estimates of the inverse of the covariance matrix to retrieve the direct connections.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "44045": "", "44047": "", "44048": "", "44051": "", "44053": "", "44065": "", "44066": "", "44068": "", "44071": "", "44072": "", "44074": "", "44075": "", "44079": "" }, "source": "meta" }