We plan on writing this up next week and the code will be released in a few days, but in the meantime, if anyone is interested in the structure of our best net (0.97088 on the private LB), here's sketch I put together. It's possible I still have some mistakes in it, so if you see anything odd let me know. And feel free to ask questions, it will help me focus the write up.
\n\nThe two key sections are the linear pre-filtering at the front and the portion where the data corresponding to recent events skips the final, drastic maxpooling. Linear elements are not typical in convents in my experience, but using linear elements to filter and downsample the data worked better for us than starting with nonlinear (ReLU) units.
\n\nThe drastic maxpooling at the end was used to reduce overfitting, but had the unfortunate side effect of making the temporal location of the signal fuzzy to dense portions of the net. By bypassing the maxpooling with a few of the most recent time points, we managed to keep the overfitting fairly low while removing the ambiguity.
\n\nAt least that's what I think is going on, it makes a good story, and it did improve our results, but who knows what the net was actually doing internally!
\n\nVLReLU stands for "Very Leaky ReLU" (and ReLU stands for "Rectified Linear Unit" but it's usually written ReLU). In this case this is a nonlinearity of the form max(0.3*a,a). I think the other terminology is standard, but I'll be happy to explain anything you find ambiguous.
\n\n[EDIT] I forgot to mention, that untied biases means that each filters has a separate bias parameter for each output location rather having one bias parameter per filter. This makes the filter more expressive at the expense of possibly overfitting. I found that using untied biases on some, but not all of the convolutional layers worked best: using them on all layers led to overfitting.
We plan on writing this up next week and the code will be released in a few days, but in the meantime, if anyone is interested in the structure of our best net (0.97088 on the private LB), here's sketch I put together. It's possible I still have some mistakes in it, so if you see anything odd let me know. And feel free to ask questions, it will help me focus the write up.
\n\nThe two key sections are the linear pre-filtering at the front and the portion where the data corresponding to recent events skips the final, drastic maxpooling. Linear elements are not typical in convents in my experience, but using linear elements to filter and downsample the data worked better for us than starting with nonlinear (ReLU) units.
\n\nThe drastic maxpooling at the end was used to reduce overfitting, but had the unfortunate side effect of making the temporal location of the signal fuzzy to dense portions of the net. By bypassing the maxpooling with a few of the most recent time points, we managed to keep the overfitting fairly low while removing the ambiguity.
\n\nAt least that's what I think is going on, it makes a good story, and it did improve our results, but who knows what the net was actually doing internally!
\n\nVLReLU stands for "Very Leaky ReLU" (and ReLU stands for "Rectified Linear Unit" but it's usually written ReLU). In this case this is a nonlinearity of the form max(0.3*a,a). I think the other terminology is standard, but I'll be happy to explain anything you find ambiguous.
\n\n[EDIT] I forgot to mention, that untied biases means that each filters has a separate bias parameter for each output location rather having one bias parameter per filter. This makes the filter more expressive at the expense of possibly overfitting. I found that using untied biases on some, but not all of the convolutional layers worked best: using them on all layers led to overfitting.
Hi Tim, thanks for posting this. I tried using Anil Thomas' NN script for this competition, and this is my first experience training Neural Nets to do anything.
\n\nI'm interested in how you get to a structure like the one you've posted here, which looks fairly complex to me. Do you automate any of the process of searching for a good network structure? Are there useful measurements besides cross-validation AUC that you use to decide what you might want to try, or what was a good change to your network?
", "rawMarkdown": "Hi Tim, thanks for posting this. I tried using Anil Thomas' NN script for this competition, and this is my first experience training Neural Nets to do anything. \r\n\r\nI'm interested in how you get to a structure like the one you've posted here, which looks fairly complex to me. Do you automate any of the process of searching for a good network structure? Are there useful measurements besides cross-validation AUC that you use to decide what you might want to try, or what was a good change to your network?", "votes": null }, { "id": "91862", "postDate": "09/08/2015 15:47:02", "content": "@David Johnston: unfortunately, I have no secret recipe for you. We were just using AUC as a validation metric. Typically we were running this with series [3,6] used for validation. This was OK, but not great. Near the end we ran some nets with other validation schemes, one of which was somewhat better, but we didn't have time to explore them.
\n\nAs for coming up with the structure. It was just an iterative process, where I'd try things, keep the things that worked and throw out those that didn't. For smaller nets, sometimes people use a grid search to explore the parameter space, but these were too slow to train for that. There were two cases where thinking about the structure of the problem and how other people approached it yielded big dividends. Those were the linear prefiltering stage and the part where I route recent features around the maxpooling.
\n\nExperience helps, since as you get familiar with different layers you can turn ideas into networks. I also took big chunks of this net directly from the standard structure for image recognition nets. This is my third neural net competition entry and I played around more with the structure this time than I have before because I'm getting more comfortable with the way the layers work and interact.
\n\nI know that's not terribly helpful, but the only way I know how to learn this stuff is to just start trying it out, even if you don't fully understand what you're doing, until eventually you develop some intuition about what's going to work. Essentially training a real neural net to work on neural nets;-)
", "rawMarkdown": "David Johnston: unfortunately, I have no secret recipe for you. We were just using AUC as a validation metric. Typically we were running this with series [3,6] used for validation. This was OK, but not great. Near the end we ran some nets with other validation schemes, one of which was somewhat better, but we didn't have time to explore them.\r\n\r\nAs for coming up with the structure. It was just an iterative process, where I'd try things, keep the things that worked and throw out those that didn't. For smaller nets, sometimes people use a grid search to explore the parameter space, but these were too slow to train for that. There were two cases where thinking about the structure of the problem and how other people approached it yielded big dividends. Those were the linear prefiltering stage and the part where I route recent features around the maxpooling. \r\n\r\nExperience helps, since as you get familiar with different layers you can turn ideas into networks. I also took big chunks of this net directly from the standard structure for image recognition nets. This is my third neural net competition entry and I played around more with the structure this time than I have before because I'm getting more comfortable with the way the layers work and interact.\r\n\r\nI know that's not terribly helpful, but the only way I know how to learn this stuff is to just start trying it out, even if you don't fully understand what you're doing, until eventually you develop some intuition about what's going to work. Essentially training a real neural net to work on neural nets;-)", "votes": null } ], "comments": [ { "id": 91823, "author_name": "davidoj", "author_url": "", "post_date": "09/08/2015 04:59:14", "content": "Hi Tim, thanks for posting this. I tried using Anil Thomas' NN script for this competition, and this is my first experience training Neural Nets to do anything.
\n\nI'm interested in how you get to a structure like the one you've posted here, which looks fairly complex to me. Do you automate any of the process of searching for a good network structure? Are there useful measurements besides cross-validation AUC that you use to decide what you might want to try, or what was a good change to your network?
", "votes": null, "replies": [] }, { "id": 91862, "author_name": "bitsofbits", "author_url": "", "post_date": "09/08/2015 15:47:02", "content": "@David Johnston: unfortunately, I have no secret recipe for you. We were just using AUC as a validation metric. Typically we were running this with series [3,6] used for validation. This was OK, but not great. Near the end we ran some nets with other validation schemes, one of which was somewhat better, but we didn't have time to explore them.
\n\nAs for coming up with the structure. It was just an iterative process, where I'd try things, keep the things that worked and throw out those that didn't. For smaller nets, sometimes people use a grid search to explore the parameter space, but these were too slow to train for that. There were two cases where thinking about the structure of the problem and how other people approached it yielded big dividends. Those were the linear prefiltering stage and the part where I route recent features around the maxpooling.
\n\nExperience helps, since as you get familiar with different layers you can turn ideas into networks. I also took big chunks of this net directly from the standard structure for image recognition nets. This is my third neural net competition entry and I played around more with the structure this time than I have before because I'm getting more comfortable with the way the layers work and interact.
\n\nI know that's not terribly helpful, but the only way I know how to learn this stuff is to just start trying it out, even if you don't fully understand what you're doing, until eventually you develop some intuition about what's going to work. Essentially training a real neural net to work on neural nets;-)
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "91366": "We plan on writing this up next week and the code will be released in a few days, but in the meantime, if anyone is interested in the structure of our best net (0.97088 on the private LB), here's sketch I put together. It's possible I still have some mistakes in it, so if you see anything odd let me know. And feel free to ask questions, it will help me focus the write up.\r\n\r\nThe two key sections are the linear pre-filtering at the front and the portion where the data corresponding to recent events skips the final, drastic maxpooling. Linear elements are not typical in convents in my experience, but using linear elements to filter and downsample the data worked better for us than starting with nonlinear (ReLU) units. \r\n\r\nThe drastic maxpooling at the end was used to reduce overfitting, but had the unfortunate side effect of making the temporal location of the signal fuzzy to dense portions of the net. By bypassing the maxpooling with a few of the most recent time points, we managed to keep the overfitting fairly low while removing the ambiguity. \r\n\r\nAt least that's what I think is going on, it makes a good story, and it did improve our results, but who knows what the net was actually doing internally!\r\n\r\nVLReLU stands for \"Very Leaky ReLU\" (and ReLU stands for \"Rectified Linear Unit\" but it's usually written ReLU). In this case this is a nonlinearity of the form max(0.3*a,a). I think the other terminology is standard, but I'll be happy to explain anything you find ambiguous.\r\n\r\n[EDIT] I forgot to mention, that `untied biases` means that each filters has a separate bias parameter for each output location rather having one bias parameter per filter. This makes the filter more expressive at the expense of possibly overfitting. I found that using untied biases on some, but not all of the convolutional layers worked best: using them on all layers led to overfitting.", "91823": "Hi Tim, thanks for posting this. I tried using Anil Thomas' NN script for this competition, and this is my first experience training Neural Nets to do anything. \r\n\r\nI'm interested in how you get to a structure like the one you've posted here, which looks fairly complex to me. Do you automate any of the process of searching for a good network structure? Are there useful measurements besides cross-validation AUC that you use to decide what you might want to try, or what was a good change to your network?", "91862": "David Johnston: unfortunately, I have no secret recipe for you. We were just using AUC as a validation metric. Typically we were running this with series [3,6] used for validation. This was OK, but not great. Near the end we ran some nets with other validation schemes, one of which was somewhat better, but we didn't have time to explore them.\r\n\r\nAs for coming up with the structure. It was just an iterative process, where I'd try things, keep the things that worked and throw out those that didn't. For smaller nets, sometimes people use a grid search to explore the parameter space, but these were too slow to train for that. There were two cases where thinking about the structure of the problem and how other people approached it yielded big dividends. Those were the linear prefiltering stage and the part where I route recent features around the maxpooling. \r\n\r\nExperience helps, since as you get familiar with different layers you can turn ideas into networks. I also took big chunks of this net directly from the standard structure for image recognition nets. This is my third neural net competition entry and I played around more with the structure this time than I have before because I'm getting more comfortable with the way the layers work and interact.\r\n\r\nI know that's not terribly helpful, but the only way I know how to learn this stuff is to just start trying it out, even if you don't fully understand what you're doing, until eventually you develop some intuition about what's going to work. Essentially training a real neural net to work on neural nets;-)" }, "source": "meta" }