Generic question, how do people come up with new deep learning architectures? Do we have a scientific methodology to design a new arch that is specific to the given problem?
\n\nAlso, it would be interesting to know how the past authors came up with their archs (eg: how unet authors came up with their arch). Please share your thoughts and any references/links.
\n\nThanks.
", "messages": [ { "id": "301421", "postDate": "03/22/2018 18:05:20", "content": "Generic question, how do people come up with new deep learning architectures? Do we have a scientific methodology to design a new arch that is specific to the given problem?
\n\nAlso, it would be interesting to know how the past authors came up with their archs (eg: how unet authors came up with their arch). Please share your thoughts and any references/links.
\n\nThanks.
", "rawMarkdown": "Generic question, how do people come up with new deep learning architectures? Do we have a scientific methodology to design a new arch that is specific to the given problem?\n\nAlso, it would be interesting to know how the past authors came up with their archs (eg: how unet authors came up with their arch). Please share your thoughts and any references/links.\n\nThanks.", "votes": null }, { "id": "301430", "postDate": "03/22/2018 18:18:03", "content": "A generic answer for your generic question is to read a lot of papers on the topic (and perhaps tangent to it) and run your own experiments. It should sharpen your intuition and theoretical understanding that may help towards new discoveries.
", "rawMarkdown": "A generic answer for your generic question is to read a lot of papers on the topic (and perhaps tangent to it) and run your own experiments. It should sharpen your intuition and theoretical understanding that may help towards new discoveries.", "votes": null }, { "id": "301465", "postDate": "03/22/2018 19:42:35", "content": "Here is my experience with modifying architecture. Get some common representation of architecture(like the graph visualization in tensorboard). Then compare architectures that exist. What components are similar which are different. Many many papers use nearly identical architecture some small(or larger) modification. This leads to one method of creating new architecture that is modifying an existing one. When reading papers, I make sure I pay attention to what problem they are solving, and then try to find similarities between the problem they are solving and one I am solving. If some similarity is found then maybe I will cut out a piece of a network and try adding it to what I am working on.
\n\nAs an example: for this competition most people are using mask-rcnn. There are two reasons I would not want to use mask-rcnn, 1: it is difficult to train from scratch(at least according to some comments I have found in various github repos). 2: As far as I know the batch size is fixed to 1, and as someone with hardware on the lower end(gtx 1070 + some cloud credits) I cannot rapidly experiment with mask-rcnn(this problem is made worse by training difficulty(are restarts common?)).
\n\nSo I would have to find some network that does not have the above problems. Or figure out how to modify mask-rcnn so that I don't have those issues.
\n\nOther method is to imagine the problem you are solving and imagine what a good set of convolutional filters(in the example of conv net) would look like(how many layers, are needed etc). then build an architecture that would solve the problem. This is not easy. But you can do it if you have an idea about what each module in a network is doing.
", "rawMarkdown": "Here is my experience with modifying architecture. Get some common representation of architecture(like the graph visualization in tensorboard). Then compare architectures that exist. What components are similar which are different. Many many papers use nearly identical architecture some small(or larger) modification. This leads to one method of creating new architecture that is modifying an existing one. When reading papers, I make sure I pay attention to what problem they are solving, and then try to find similarities between the problem they are solving and one I am solving. If some similarity is found then maybe I will cut out a piece of a network and try adding it to what I am working on. \n\nAs an example: for this competition most people are using mask-rcnn. There are two reasons I would not want to use mask-rcnn, 1: it is difficult to train from scratch(at least according to some comments I have found in various github repos). 2: As far as I know the batch size is fixed to 1, and as someone with hardware on the lower end(gtx 1070 + some cloud credits) I cannot rapidly experiment with mask-rcnn(this problem is made worse by training difficulty(are restarts common?)).\n\nSo I would have to find some network that does not have the above problems. Or figure out how to modify mask-rcnn so that I don't have those issues. \n\nOther method is to imagine the problem you are solving and imagine what a good set of convolutional filters(in the example of conv net) would look like(how many layers, are needed etc). then build an architecture that would solve the problem. This is not easy. But you can do it if you have an idea about what each module in a network is doing.", "votes": null }, { "id": "301470", "postDate": "03/22/2018 19:55:50", "content": "thanks Devin! for the detailed response.
", "rawMarkdown": "thanks Devin! for the detailed response.", "votes": null }, { "id": "303202", "postDate": "03/25/2018 18:35:03", "content": "I find this article interesting http://www.argmin.net/2018/01/25/optics/.
", "rawMarkdown": "I find this article interesting http://www.argmin.net/2018/01/25/optics/.", "votes": null }, { "id": "303383", "postDate": "03/26/2018 04:58:54", "content": "interesting! thanks for sharing!
", "rawMarkdown": "interesting! thanks for sharing!", "votes": null } ], "comments": [ { "id": 301430, "author_name": "pliptor", "author_url": "", "post_date": "03/22/2018 18:18:03", "content": "A generic answer for your generic question is to read a lot of papers on the topic (and perhaps tangent to it) and run your own experiments. It should sharpen your intuition and theoretical understanding that may help towards new discoveries.
", "votes": null, "replies": [] }, { "id": 301465, "author_name": "devinanzelmo", "author_url": "", "post_date": "03/22/2018 19:42:35", "content": "Here is my experience with modifying architecture. Get some common representation of architecture(like the graph visualization in tensorboard). Then compare architectures that exist. What components are similar which are different. Many many papers use nearly identical architecture some small(or larger) modification. This leads to one method of creating new architecture that is modifying an existing one. When reading papers, I make sure I pay attention to what problem they are solving, and then try to find similarities between the problem they are solving and one I am solving. If some similarity is found then maybe I will cut out a piece of a network and try adding it to what I am working on.
\n\nAs an example: for this competition most people are using mask-rcnn. There are two reasons I would not want to use mask-rcnn, 1: it is difficult to train from scratch(at least according to some comments I have found in various github repos). 2: As far as I know the batch size is fixed to 1, and as someone with hardware on the lower end(gtx 1070 + some cloud credits) I cannot rapidly experiment with mask-rcnn(this problem is made worse by training difficulty(are restarts common?)).
\n\nSo I would have to find some network that does not have the above problems. Or figure out how to modify mask-rcnn so that I don't have those issues.
\n\nOther method is to imagine the problem you are solving and imagine what a good set of convolutional filters(in the example of conv net) would look like(how many layers, are needed etc). then build an architecture that would solve the problem. This is not easy. But you can do it if you have an idea about what each module in a network is doing.
", "votes": null, "replies": [ { "id": 301470, "author_name": "ravivadapalli", "author_url": "", "post_date": "03/22/2018 19:55:50", "content": "thanks Devin! for the detailed response.
", "votes": null, "replies": [] } ] }, { "id": 303202, "author_name": "mrtzmilani", "author_url": "", "post_date": "03/25/2018 18:35:03", "content": "I find this article interesting http://www.argmin.net/2018/01/25/optics/.
", "votes": null, "replies": [ { "id": 303383, "author_name": "ravivadapalli", "author_url": "", "post_date": "03/26/2018 04:58:54", "content": "interesting! thanks for sharing!
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "301421": "Generic question, how do people come up with new deep learning architectures? Do we have a scientific methodology to design a new arch that is specific to the given problem?\n\nAlso, it would be interesting to know how the past authors came up with their archs (eg: how unet authors came up with their arch). Please share your thoughts and any references/links.\n\nThanks.", "301430": "A generic answer for your generic question is to read a lot of papers on the topic (and perhaps tangent to it) and run your own experiments. It should sharpen your intuition and theoretical understanding that may help towards new discoveries.", "301465": "Here is my experience with modifying architecture. Get some common representation of architecture(like the graph visualization in tensorboard). Then compare architectures that exist. What components are similar which are different. Many many papers use nearly identical architecture some small(or larger) modification. This leads to one method of creating new architecture that is modifying an existing one. When reading papers, I make sure I pay attention to what problem they are solving, and then try to find similarities between the problem they are solving and one I am solving. If some similarity is found then maybe I will cut out a piece of a network and try adding it to what I am working on. \n\nAs an example: for this competition most people are using mask-rcnn. There are two reasons I would not want to use mask-rcnn, 1: it is difficult to train from scratch(at least according to some comments I have found in various github repos). 2: As far as I know the batch size is fixed to 1, and as someone with hardware on the lower end(gtx 1070 + some cloud credits) I cannot rapidly experiment with mask-rcnn(this problem is made worse by training difficulty(are restarts common?)).\n\nSo I would have to find some network that does not have the above problems. Or figure out how to modify mask-rcnn so that I don't have those issues. \n\nOther method is to imagine the problem you are solving and imagine what a good set of convolutional filters(in the example of conv net) would look like(how many layers, are needed etc). then build an architecture that would solve the problem. This is not easy. But you can do it if you have an idea about what each module in a network is doing.", "301470": "thanks Devin! for the detailed response.", "303202": "I find this article interesting http://www.argmin.net/2018/01/25/optics/.", "303383": "interesting! thanks for sharing!" }, "source": "meta" }