{
  "id": 8150,
  "title": "Deep CNN for Time Series Correlation Measurement ",
  "url": "/competitions/connectomics/writeups/lukasz-8000-deep-cnn-for-time-series-correlation-m",
  "author_name": "",
  "post_date": "2014-05-14T14:04:54.100Z",
  "votes": 6,
  "comment_count": 7,
  "views": 5508,
  "content": "<p>Hello,</p>\n<p>here is a brief description of my solution:</p>\n<p>The solution is a deep convolutional neural network (4th place, 93.96% AUC score), and is a novelty in terms of application of CNN to time series correlation measurement. It takes as input two fragments of activity recordings (derivative) and detects patterns indicating correlations between them. One significant improvement was developed by providing one additional row to these two - an overall brain network activity increase in each timeframe, allowing the CNN to learn the level of noise and different behavior depending on the network activity level. Moreover, the recordings are firstly frame-filtered based on optimal threshold of minimum activity increase in a given time frame, keeping fragments of high activity and decreasing the input size to around 1% of the initial input length.</p>\n<p>Training uses the same number of positive and negative examples, total 1.2 million. The code was developed in Python with Theano and is designed for running on a GPU. The initial ideas of the network structure were based on Lenet5 Network for digit classification (by Y. LeCun). The network behavior has been inspected deeply, including exporting activation values of Theano hidden layer units and classifying them with SVM (libSVM, libLinear), allowing to adopt the network architecture specifically for the task of correlation detection.</p>\n<p>Network architecture:</p>\n<p><br>Input [3x330]<br>&nbsp;&nbsp; = [Fragment_cell1,<br>&nbsp;&nbsp; &nbsp; &nbsp; Fragment_cell2,<br>&nbsp;&nbsp; &nbsp; &nbsp; Network_activity]</p>\n<p>&nbsp;&nbsp; Please note that a length of 330 covers around 20% of total simulation time before frame-filtering.</p>\n<p><br>Convolutional, 18 units (= feature maps for Convolutional layers), tanh, filter shape = [2x5]<br>Convolutional, 40 units, ReLU, filter shape = [2x5]<br>Convolutional, 15 units, ReLU, filter shape = [1x1], max pooling [1x10]<br>Fully Connected, 100 units, tanh<br>Softmax</p>\n<p>Best regards,<br>Lukasz Romaszko</p>",
  "messages": [
    {
      "id": "44531",
      "postDate": "05/14/2014 14:04:54",
      "content": "<p>Hello,</p>\n<p>here is a brief description of my solution:</p>\n<p>The solution is a deep convolutional neural network (4th place, 93.96% AUC score), and is a novelty in terms of application of CNN to time series correlation measurement. It takes as input two fragments of activity recordings (derivative) and detects patterns indicating correlations between them. One significant improvement was developed by providing one additional row to these two - an overall brain network activity increase in each timeframe, allowing the CNN to learn the level of noise and different behavior depending on the network activity level. Moreover, the recordings are firstly frame-filtered based on optimal threshold of minimum activity increase in a given time frame, keeping fragments of high activity and decreasing the input size to around 1% of the initial input length.</p>\n<p>Training uses the same number of positive and negative examples, total 1.2 million. The code was developed in Python with Theano and is designed for running on a GPU. The initial ideas of the network structure were based on Lenet5 Network for digit classification (by Y. LeCun). The network behavior has been inspected deeply, including exporting activation values of Theano hidden layer units and classifying them with SVM (libSVM, libLinear), allowing to adopt the network architecture specifically for the task of correlation detection.</p>\n<p>Network architecture:</p>\n<p><br>Input [3x330]<br>&nbsp;&nbsp; = [Fragment_cell1,<br>&nbsp;&nbsp; &nbsp; &nbsp; Fragment_cell2,<br>&nbsp;&nbsp; &nbsp; &nbsp; Network_activity]</p>\n<p>&nbsp;&nbsp; Please note that a length of 330 covers around 20% of total simulation time before frame-filtering.</p>\n<p><br>Convolutional, 18 units (= feature maps for Convolutional layers), tanh, filter shape = [2x5]<br>Convolutional, 40 units, ReLU, filter shape = [2x5]<br>Convolutional, 15 units, ReLU, filter shape = [1x1], max pooling [1x10]<br>Fully Connected, 100 units, tanh<br>Softmax</p>\n<p>Best regards,<br>Lukasz Romaszko</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "44655",
      "postDate": "05/16/2014 09:07:47",
      "content": "<p>Hello,</p>\n<p>Very interesting and of course very impressive.<br>I was wondering if NN's could be applied but the causality theory was pretty complex. I thought&nbsp;that a NN might not be able to &quot;learn&quot; it.</p>\n<p>Few&nbsp;things that I could not read between the lines..<br><br>Is the 3x330 a 1-channel 2d input like a black-white image ?<br>Or is it a 3 channel x 330 1d input ? &nbsp;</p>\n<p>Given the filter I assume it's 2d. Assuming the filter is also 2d.<br>If the filter = 2x5&nbsp;so wouldn't you miss applying the filter to&nbsp;[Fragment_cell1 x&nbsp;Network_activity] ?<br>Example:&nbsp;when you apply the filter in the rows direction, the input has 3 rows and the filter has 2 rows.<br>The first application would apply the filter to row 1,2 [Fragment_cell1 x&nbsp;Fragment_cell2].<br>The second&nbsp;application would apply the filter to row 2,3 [Fragment_cell2 x Network_activity].<br>Then it is not applied to&nbsp;&nbsp;[Fragment_cell1 x Network_activity].<br><br></p>\n<p>I hope you follow my question.</p>\n<p><br>Last question.. Why the variation with the activations. I see tanh and relu applied at different layers.<br>Did this give better results ?</p>\n<p>Thanks.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "44676",
      "postDate": "05/16/2014 17:58:40",
      "content": "<p>Dear Julian,</p>\n<p>thank you very much for your comment. I have drawn a figure to explain it better. In this figure you can see subsequent layers details and their dimensions. Numbers in bold font are the CNN parameters, and the other ones are inferred from them.</p>\n<p>Since in CNN we use feature maps, I wrote the dimensions of a filter applied to one feature map, however, the feature maps are fully-connected, i.e. a single value in a particular feature map is computed based on NUM_FEATURE_MAPS_PREVIOUS_LAYER x FILTER_SHAPE.</p>\n<p>In the figure there are also some extra visualisations, e.g. the small green square represents a single value in a single feature map, calculated by applying a filter of 18 x 2 x 5 weights (which are this feature map's parameters to be learnt during training) to all of the feature maps in the previous layer.</p>\n<p>The total number of parameters in convolutional layers to be learnt is therefore:<br>2x18x1x2x5 = 360 <br>2x40x18x2x5 = 14400<br>2x15x40x1x1 = 1200</p>\n<p>Total 15960 parameters (first '2x' stands for an edge weight and bias) + without last layer's, but time spent for learning parameters in this usual layer is negligible.</p>\n<p>Below link to the picture. Yes, there is 1 input of 3 x 330&nbsp; (2D) dimension.</p>\n<p>https://drive.google.com/file/d/0B98iXdxQRL0TRF9xbW1fUE8tSm8</p>\n<p>Regarding your question, when the activity row was added also as adjacent to Fragment_cell1, the result was almost the same. Much higher accuracy gain was achieved by increasing the training size by 33% than the input itself (assuming limited memory). It seems that the CNN was already able to learn properly, since that row was already given to it. Regarding activation functions, firstly I used only tanh as a default, and then tested other ones, since positive indications of correlation should be additive it occurred that this improved the result, it is said that ReLU is &quot;argued to be more biologically plausible&quot;, furthermore I don't want to let any accidental negative correlation indication pattern to decrease the correlation confidence.</p>\n<p>Best regards,</p>\n<p>Lukasz</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "44826",
      "postDate": "05/19/2014 13:25:53",
      "content": "<p>Thanks !</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "107549",
      "postDate": "02/11/2016 00:42:26",
      "content": "<p>[quote=Lukasz 8000;44676]</p>\n<p>Dear Julian,</p>\n<p>thank you very much for your comment. I have drawn a figure to explain it better. In this figure you can see subsequent layers details and their dimensions. Numbers in bold font are the CNN parameters, and the other ones are inferred from them.</p>\n<p>Since in CNN we use feature maps, I wrote the dimensions of a filter applied to one feature map, however, the feature maps are fully-connected, i.e. a single value in a particular feature map is computed based on NUM_FEATURE_MAPS_PREVIOUS_LAYER x FILTER_SHAPE.</p>\n<p>In the figure there are also some extra visualisations, e.g. the small green square represents a single value in a single feature map, calculated by applying a filter of 18 x 2 x 5 weights (which are this feature map's parameters to be learnt during training) to all of the feature maps in the previous layer.</p>\n<p>The total number of parameters in convolutional layers to be learnt is therefore:<br>2x18x1x2x5 = 360 <br>2x40x18x2x5 = 14400<br>2x15x40x1x1 = 1200</p>\n<p>Total 15960 parameters (first '2x' stands for an edge weight and bias) + without last layer's, but time spent for learning parameters in this usual layer is negligible.</p>\n<p>Below link to the picture. Yes, there is 1 input of 3 x 330&nbsp; (2D) dimension.</p>\n<p>https://drive.google.com/file/d/0B98iXdxQRL0TRF9xbW1fUE8tSm8</p>\n<p>Regarding your question, when the activity row was added also as adjacent to Fragment_cell1, the result was almost the same. Much higher accuracy gain was achieved by increasing the training size by 33% than the input itself (assuming limited memory). It seems that the CNN was already able to learn properly, since that row was already given to it. Regarding activation functions, firstly I used only tanh as a default, and then tested other ones, since positive indications of correlation should be additive it occurred that this improved the result, it is said that ReLU is &quot;argued to be more biologically plausible&quot;, furthermore I don't want to let any accidental negative correlation indication pattern to decrease the correlation confidence.</p>\n<p>Best regards,</p>\n<p>Lukasz</p>\n<p>[/quote]</p>\n\n<p>Lukas, hi!</p>\n<p>I was wondering if you would share this code so i can study it and make changes and see how it works!</p>\n<p>Thanks in advance :)</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "121181",
      "postDate": "05/24/2016 19:12:58",
      "content": "<p>Can someone post a link to the problem statement?</p>",
      "rawMarkdown": "Can someone post a link to the problem statement?",
      "votes": null
    },
    {
      "id": "121185",
      "postDate": "05/24/2016 19:20:28",
      "content": "<p>See <a href=\"http://connectomics.chalearn.org/help/setting\">http://connectomics.chalearn.org/help/setting</a>\nGiven time series of variable (representing neural activities) reconstruct the network topology.</p>",
      "rawMarkdown": "See http://connectomics.chalearn.org/help/setting\r\nGiven time series of variable (representing neural activities) reconstruct the network topology.",
      "votes": null
    },
    {
      "id": "135838",
      "postDate": "09/17/2016 14:20:23",
      "content": "<p>Can someone guide me how to input a time series data namely pamap2 activity recognition dataset for classification through convolutional neural networks</p>",
      "rawMarkdown": "Can someone guide me how to input a time series data namely pamap2 activity recognition dataset for classification through convolutional neural networks",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 44655,
      "author_name": "juliandewit",
      "author_url": "",
      "post_date": "05/16/2014 09:07:47",
      "content": "<p>Hello,</p>\n<p>Very interesting and of course very impressive.<br>I was wondering if NN's could be applied but the causality theory was pretty complex. I thought&nbsp;that a NN might not be able to &quot;learn&quot; it.</p>\n<p>Few&nbsp;things that I could not read between the lines..<br><br>Is the 3x330 a 1-channel 2d input like a black-white image ?<br>Or is it a 3 channel x 330 1d input ? &nbsp;</p>\n<p>Given the filter I assume it's 2d. Assuming the filter is also 2d.<br>If the filter = 2x5&nbsp;so wouldn't you miss applying the filter to&nbsp;[Fragment_cell1 x&nbsp;Network_activity] ?<br>Example:&nbsp;when you apply the filter in the rows direction, the input has 3 rows and the filter has 2 rows.<br>The first application would apply the filter to row 1,2 [Fragment_cell1 x&nbsp;Fragment_cell2].<br>The second&nbsp;application would apply the filter to row 2,3 [Fragment_cell2 x Network_activity].<br>Then it is not applied to&nbsp;&nbsp;[Fragment_cell1 x Network_activity].<br><br></p>\n<p>I hope you follow my question.</p>\n<p><br>Last question.. Why the variation with the activations. I see tanh and relu applied at different layers.<br>Did this give better results ?</p>\n<p>Thanks.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 44676,
      "author_name": "lukasz8000",
      "author_url": "",
      "post_date": "05/16/2014 17:58:40",
      "content": "<p>Dear Julian,</p>\n<p>thank you very much for your comment. I have drawn a figure to explain it better. In this figure you can see subsequent layers details and their dimensions. Numbers in bold font are the CNN parameters, and the other ones are inferred from them.</p>\n<p>Since in CNN we use feature maps, I wrote the dimensions of a filter applied to one feature map, however, the feature maps are fully-connected, i.e. a single value in a particular feature map is computed based on NUM_FEATURE_MAPS_PREVIOUS_LAYER x FILTER_SHAPE.</p>\n<p>In the figure there are also some extra visualisations, e.g. the small green square represents a single value in a single feature map, calculated by applying a filter of 18 x 2 x 5 weights (which are this feature map's parameters to be learnt during training) to all of the feature maps in the previous layer.</p>\n<p>The total number of parameters in convolutional layers to be learnt is therefore:<br>2x18x1x2x5 = 360 <br>2x40x18x2x5 = 14400<br>2x15x40x1x1 = 1200</p>\n<p>Total 15960 parameters (first '2x' stands for an edge weight and bias) + without last layer's, but time spent for learning parameters in this usual layer is negligible.</p>\n<p>Below link to the picture. Yes, there is 1 input of 3 x 330&nbsp; (2D) dimension.</p>\n<p>https://drive.google.com/file/d/0B98iXdxQRL0TRF9xbW1fUE8tSm8</p>\n<p>Regarding your question, when the activity row was added also as adjacent to Fragment_cell1, the result was almost the same. Much higher accuracy gain was achieved by increasing the training size by 33% than the input itself (assuming limited memory). It seems that the CNN was already able to learn properly, since that row was already given to it. Regarding activation functions, firstly I used only tanh as a default, and then tested other ones, since positive indications of correlation should be additive it occurred that this improved the result, it is said that ReLU is &quot;argued to be more biologically plausible&quot;, furthermore I don't want to let any accidental negative correlation indication pattern to decrease the correlation confidence.</p>\n<p>Best regards,</p>\n<p>Lukasz</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 44826,
      "author_name": "juliandewit",
      "author_url": "",
      "post_date": "05/19/2014 13:25:53",
      "content": "<p>Thanks !</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 107549,
      "author_name": "andrelopes1705",
      "author_url": "",
      "post_date": "02/11/2016 00:42:26",
      "content": "<p>[quote=Lukasz 8000;44676]</p>\n<p>Dear Julian,</p>\n<p>thank you very much for your comment. I have drawn a figure to explain it better. In this figure you can see subsequent layers details and their dimensions. Numbers in bold font are the CNN parameters, and the other ones are inferred from them.</p>\n<p>Since in CNN we use feature maps, I wrote the dimensions of a filter applied to one feature map, however, the feature maps are fully-connected, i.e. a single value in a particular feature map is computed based on NUM_FEATURE_MAPS_PREVIOUS_LAYER x FILTER_SHAPE.</p>\n<p>In the figure there are also some extra visualisations, e.g. the small green square represents a single value in a single feature map, calculated by applying a filter of 18 x 2 x 5 weights (which are this feature map's parameters to be learnt during training) to all of the feature maps in the previous layer.</p>\n<p>The total number of parameters in convolutional layers to be learnt is therefore:<br>2x18x1x2x5 = 360 <br>2x40x18x2x5 = 14400<br>2x15x40x1x1 = 1200</p>\n<p>Total 15960 parameters (first '2x' stands for an edge weight and bias) + without last layer's, but time spent for learning parameters in this usual layer is negligible.</p>\n<p>Below link to the picture. Yes, there is 1 input of 3 x 330&nbsp; (2D) dimension.</p>\n<p>https://drive.google.com/file/d/0B98iXdxQRL0TRF9xbW1fUE8tSm8</p>\n<p>Regarding your question, when the activity row was added also as adjacent to Fragment_cell1, the result was almost the same. Much higher accuracy gain was achieved by increasing the training size by 33% than the input itself (assuming limited memory). It seems that the CNN was already able to learn properly, since that row was already given to it. Regarding activation functions, firstly I used only tanh as a default, and then tested other ones, since positive indications of correlation should be additive it occurred that this improved the result, it is said that ReLU is &quot;argued to be more biologically plausible&quot;, furthermore I don't want to let any accidental negative correlation indication pattern to decrease the correlation confidence.</p>\n<p>Best regards,</p>\n<p>Lukasz</p>\n<p>[/quote]</p>\n\n<p>Lukas, hi!</p>\n<p>I was wondering if you would share this code so i can study it and make changes and see how it works!</p>\n<p>Thanks in advance :)</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 121181,
      "author_name": "siddhu95",
      "author_url": "",
      "post_date": "05/24/2016 19:12:58",
      "content": "<p>Can someone post a link to the problem statement?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 121185,
      "author_name": "iguyon",
      "author_url": "",
      "post_date": "05/24/2016 19:20:28",
      "content": "<p>See <a href=\"http://connectomics.chalearn.org/help/setting\">http://connectomics.chalearn.org/help/setting</a>\nGiven time series of variable (representing neural activities) reconstruct the network topology.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 135838,
      "author_name": "hannaj",
      "author_url": "",
      "post_date": "09/17/2016 14:20:23",
      "content": "<p>Can someone guide me how to input a time series data namely pamap2 activity recognition dataset for classification through convolutional neural networks</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "44531": "",
    "44655": "",
    "44676": "",
    "44826": "",
    "107549": "",
    "121181": "Can someone post a link to the problem statement?",
    "121185": "See http://connectomics.chalearn.org/help/setting\r\nGiven time series of variable (representing neural activities) reconstruct the network topology.",
    "135838": "Can someone guide me how to input a time series data namely pamap2 activity recognition dataset for classification through convolutional neural networks"
  },
  "source": "meta"
}