After many rounds of experiments, I have proved that convolution on multiple embedded features such as T5 is indeed not as good as the fitting of linear neural networks. How should the model be selected? Or is there a problem with my data features? 🤔Thanks!
\nHappy kaggling to everyone.🎉
", "messages": [ { "id": "2278305", "postDate": "05/28/2023 14:52:53", "content": "After many rounds of experiments, I have proved that convolution on multiple embedded features such as T5 is indeed not as good as the fitting of linear neural networks. How should the model be selected? Or is there a problem with my data features? 🤔Thanks!
\nHappy kaggling to everyone.🎉
", "rawMarkdown": "After many rounds of experiments, I have proved that convolution on multiple embedded features such as T5 is indeed not as good as the fitting of linear neural networks. How should the model be selected? Or is there a problem with my data features? 🤔Thanks!\n\nHappy kaggling to everyone.🎉", "votes": null }, { "id": "2278566", "postDate": "05/28/2023 19:43:17", "content": "CNNs are great for learning local patterns (say in an image) - perhaps you might want to consider GNNs instead?
", "rawMarkdown": "CNNs are great for learning local patterns (say in an image) - perhaps you might want to consider GNNs instead?", "votes": null }, { "id": "2279724", "postDate": "05/29/2023 15:05:15", "content": "Hi! It depend on what type of data your are currently working on. If you work with embeddings vectors, CNN are not really good for this kind of inputs as it is made to capture dependencies among local structures (close pixels of an image for example). For embedding vectors it’s like you want to catch dependencies between close vector coordinates.
", "rawMarkdown": "Hi! It depend on what type of data your are currently working on. If you work with embeddings vectors, CNN are not really good for this kind of inputs as it is made to capture dependencies among local structures (close pixels of an image for example). For embedding vectors it’s like you want to catch dependencies between close vector coordinates.", "votes": null }, { "id": "2280363", "postDate": "05/30/2023 04:40:49", "content": "Thx! it's useful, I find that simple linear models score a little higher, is this normal? Is this an overfitting like the simple ridge model?
\nthanks!
It can be over fitting indeed. There could be other reasons like for example the architecture. Perhaps you could compare the nb of learnable parameters from both models. Don’t know also if CNN are weak to multilabel classification with a high number of classes (with imbalance also)
", "rawMarkdown": "It can be over fitting indeed. There could be other reasons like for example the architecture. Perhaps you could compare the nb of learnable parameters from both models. Don’t know also if CNN are weak to multilabel classification with a high number of classes (with imbalance also)", "votes": null }, { "id": "2342906", "postDate": "07/13/2023 08:39:14", "content": "We can actually obtain a decent score using henris pipeline with CNNs (for now my best is 0.5 on LB) but we need a small trick. As others pointed out the embeddings are not spatially ordered. So we need to order them before applying cnn
", "rawMarkdown": "We can actually obtain a decent score using henris pipeline with CNNs (for now my best is 0.5 on LB) but we need a small trick. As others pointed out the embeddings are not spatially ordered. So we need to order them before applying cnn", "votes": null }, { "id": "2343223", "postDate": "07/13/2023 13:45:52", "content": "does spatial ordering of embeddings improve performance with non-CNN models as well? I am also confused if this is something that can be done with sequence based embeddings or would you require the embeddings from each residue?
", "rawMarkdown": "does spatial ordering of embeddings improve performance with non-CNN models as well? I am also confused if this is something that can be done with sequence based embeddings or would you require the embeddings from each residue?", "votes": null }, { "id": "2343315", "postDate": "07/13/2023 15:13:58", "content": "\n\ndoes spatial ordering of embeddings improve performance with non-CNN models as well? I am also confused if this is something that can be done with sequence based embeddings or would you require the embeddings from each residue?
\n
I don't know what you have in mind but right now I simply project in higher dimensional space, reshape and feed into cnn1d. I guess this way the model learns by itself a spatial ordering. Maybe you have something better in mind or are familiar with more sophisticated approaches. I would be happy to hear about them :)
", "rawMarkdown": "> does spatial ordering of embeddings improve performance with non-CNN models as well? I am also confused if this is something that can be done with sequence based embeddings or would you require the embeddings from each residue?\n\nI don't know what you have in mind but right now I simply project in higher dimensional space, reshape and feed into cnn1d. I guess this way the model learns by itself a spatial ordering. Maybe you have something better in mind or are familiar with more sophisticated approaches. I would be happy to hear about them :)", "votes": null }, { "id": "2343393", "postDate": "07/13/2023 16:17:56", "content": "I have not had the chance to try such approaches but its interesting to know! Mainly I am sticking to Dense NNs
", "rawMarkdown": "I have not had the chance to try such approaches but its interesting to know! Mainly I am sticking to Dense NNs", "votes": null }, { "id": "2343399", "postDate": "07/13/2023 16:24:01", "content": "I mainly tried out this approach to get diverse model. I think that maybe that can make good impact in ensembles as uts different approach but who knows :))
", "rawMarkdown": "I mainly tried out this approach to get diverse model. I think that maybe that can make good impact in ensembles as uts different approach but who knows :))", "votes": null } ], "comments": [ { "id": 2278566, "author_name": "zmcxjt", "author_url": "", "post_date": "05/28/2023 19:43:17", "content": "CNNs are great for learning local patterns (say in an image) - perhaps you might want to consider GNNs instead?
", "votes": null, "replies": [] }, { "id": 2279724, "author_name": "henriupton", "author_url": "", "post_date": "05/29/2023 15:05:15", "content": "Hi! It depend on what type of data your are currently working on. If you work with embeddings vectors, CNN are not really good for this kind of inputs as it is made to capture dependencies among local structures (close pixels of an image for example). For embedding vectors it’s like you want to catch dependencies between close vector coordinates.
", "votes": null, "replies": [ { "id": 2280363, "author_name": "lau01b", "author_url": "", "post_date": "05/30/2023 04:40:49", "content": "Thx! it's useful, I find that simple linear models score a little higher, is this normal? Is this an overfitting like the simple ridge model?
\nthanks!
It can be over fitting indeed. There could be other reasons like for example the architecture. Perhaps you could compare the nb of learnable parameters from both models. Don’t know also if CNN are weak to multilabel classification with a high number of classes (with imbalance also)
", "votes": null, "replies": [] } ] } ] }, { "id": 2342906, "author_name": "simonveitner", "author_url": "", "post_date": "07/13/2023 08:39:14", "content": "We can actually obtain a decent score using henris pipeline with CNNs (for now my best is 0.5 on LB) but we need a small trick. As others pointed out the embeddings are not spatially ordered. So we need to order them before applying cnn
", "votes": null, "replies": [ { "id": 2343223, "author_name": "zmcxjt", "author_url": "", "post_date": "07/13/2023 13:45:52", "content": "does spatial ordering of embeddings improve performance with non-CNN models as well? I am also confused if this is something that can be done with sequence based embeddings or would you require the embeddings from each residue?
", "votes": null, "replies": [ { "id": 2343315, "author_name": "simonveitner", "author_url": "", "post_date": "07/13/2023 15:13:58", "content": "\n\ndoes spatial ordering of embeddings improve performance with non-CNN models as well? I am also confused if this is something that can be done with sequence based embeddings or would you require the embeddings from each residue?
\n
I don't know what you have in mind but right now I simply project in higher dimensional space, reshape and feed into cnn1d. I guess this way the model learns by itself a spatial ordering. Maybe you have something better in mind or are familiar with more sophisticated approaches. I would be happy to hear about them :)
", "votes": null, "replies": [ { "id": 2343393, "author_name": "zmcxjt", "author_url": "", "post_date": "07/13/2023 16:17:56", "content": "I have not had the chance to try such approaches but its interesting to know! Mainly I am sticking to Dense NNs
", "votes": null, "replies": [ { "id": 2343399, "author_name": "simonveitner", "author_url": "", "post_date": "07/13/2023 16:24:01", "content": "I mainly tried out this approach to get diverse model. I think that maybe that can make good impact in ensembles as uts different approach but who knows :))
", "votes": null, "replies": [] } ] } ] } ] } ] } ], "raw_markdown_by_id": { "2278305": "After many rounds of experiments, I have proved that convolution on multiple embedded features such as T5 is indeed not as good as the fitting of linear neural networks. How should the model be selected? Or is there a problem with my data features? 🤔Thanks!\n\nHappy kaggling to everyone.🎉", "2278566": "CNNs are great for learning local patterns (say in an image) - perhaps you might want to consider GNNs instead?", "2279724": "Hi! It depend on what type of data your are currently working on. If you work with embeddings vectors, CNN are not really good for this kind of inputs as it is made to capture dependencies among local structures (close pixels of an image for example). For embedding vectors it’s like you want to catch dependencies between close vector coordinates.", "2280363": "Thx! it's useful, I find that simple linear models score a little higher, is this normal? Is this an overfitting like the simple ridge model?\nthanks!", "2280495": "It can be over fitting indeed. There could be other reasons like for example the architecture. Perhaps you could compare the nb of learnable parameters from both models. Don’t know also if CNN are weak to multilabel classification with a high number of classes (with imbalance also)", "2342906": "We can actually obtain a decent score using henris pipeline with CNNs (for now my best is 0.5 on LB) but we need a small trick. As others pointed out the embeddings are not spatially ordered. So we need to order them before applying cnn", "2343223": "does spatial ordering of embeddings improve performance with non-CNN models as well? I am also confused if this is something that can be done with sequence based embeddings or would you require the embeddings from each residue?", "2343315": "> does spatial ordering of embeddings improve performance with non-CNN models as well? I am also confused if this is something that can be done with sequence based embeddings or would you require the embeddings from each residue?\n\nI don't know what you have in mind but right now I simply project in higher dimensional space, reshape and feed into cnn1d. I guess this way the model learns by itself a spatial ordering. Maybe you have something better in mind or are familiar with more sophisticated approaches. I would be happy to hear about them :)", "2343393": "I have not had the chance to try such approaches but its interesting to know! Mainly I am sticking to Dense NNs", "2343399": "I mainly tried out this approach to get diverse model. I think that maybe that can make good impact in ensembles as uts different approach but who knows :))" }, "source": "meta" }