Based on the limits of the modelling I have done, I hope to not be totally wrong in suggesting that ~0.53 is the highest score achievable simply based on the publicly available sequence derived embeddings (T5/EMS2/ProtBert). I would be quite surprised to learn that there are more ways to wring out a higher score based simply on these embeddings and if so will be most delighted to be contradicted by those higher on the leaderboard! This post is thus also a sign to those that are working on this problem using embeddings that ~0.53 is achievable based solely on that data. Good luck all!
\nEdit: The limit is significantly higher, my apologies.
", "messages": [ { "id": 2290914, "postDate": "2023-06-07T07:48:27.750Z", "content": "Based on the limits of the modelling I have done, I hope to not be totally wrong in suggesting that ~0.53 is the highest score achievable simply based on the publicly available sequence derived embeddings (T5/EMS2/ProtBert). I would be quite surprised to learn that there are more ways to wring out a higher score based simply on these embeddings and if so will be most delighted to be contradicted by those higher on the leaderboard! This post is thus also a sign to those that are working on this problem using embeddings that ~0.53 is achievable based solely on that data. Good luck all!
\nEdit: The limit is significantly higher, my apologies.
", "rawMarkdown": "Based on the limits of the modelling I have done, I hope to not be totally wrong in suggesting that ~0.53 is the highest score achievable simply based on the publicly available sequence derived embeddings (T5/EMS2/ProtBert). I would be quite surprised to learn that there are more ways to wring out a higher score based simply on these embeddings and if so will be most delighted to be contradicted by those higher on the leaderboard! This post is thus also a sign to those that are working on this problem using embeddings that ~0.53 is achievable based solely on that data. Good luck all! \n\nEdit: The limit is significantly higher, my apologies.", "votes": 3 }, { "id": 2290930, "postDate": "2023-06-07T08:01:16.213Z", "content": "I think you are right in your assumption, maybe the top limit I bet is 0.56-0.57 if you are very very good in statistics and linear regression, but you need something else. Also with kaggle resources I think that you have to reduce a lot your dimensionality in order to train something.
\nPeople that will achieve better resources in this comp. (mainly labs that have participated in past editions) they'll create their own models and they'll use their own resources to create some successful model.
", "rawMarkdown": "I think you are right in your assumption, maybe the top limit I bet is 0.56-0.57 if you are very very good in statistics and linear regression, but you need something else. Also with kaggle resources I think that you have to reduce a lot your dimensionality in order to train something. \n\nPeople that will achieve better resources in this comp. (mainly labs that have participated in past editions) they'll create their own models and they'll use their own resources to create some successful model.", "votes": 2, "replies": [ { "id": 2291308, "postDate": "2023-06-07T13:14:33.250Z", "content": "There are tricks to save ram, I have been able to train with all 3 embeddings easily below the max RAM limit on kaggle with GPU. Good to know the limit is likely higher than I assumed!
", "rawMarkdown": "There are tricks to save ram, I have been able to train with all 3 embeddings easily below the max RAM limit on kaggle with GPU. Good to know the limit is likely higher than I assumed! ", "votes": 1, "replies": [ { "id": 2291339, "postDate": "2023-06-07T13:43:00.813Z", "content": "The tricks that i know less parameters and low dimensionality…
", "rawMarkdown": "The tricks that i know less parameters and low dimensionality...", "votes": 1 }, { "id": 2291423, "postDate": "2023-06-07T14:43:23.177Z", "content": "Change the way you load in data and data type will get you far.. :)
", "rawMarkdown": "Change the way you load in data and data type will get you far.. :) ", "votes": 1 } ] } ] }, { "id": 2291428, "postDate": "2023-06-07T14:45:19.210Z", "content": "You might be right, however I feel this approach should be properly investigated as a baseline, before starting something really complicated. So far my best score based purely on the public embeddings is 0.54. An interesting question though how much it overfits the leaderboard. I am hoping to get a properly crossvalidated embedding version that gets 0.55.
", "rawMarkdown": "You might be right, however I feel this approach should be properly investigated as a baseline, before starting something really complicated. So far my best score based purely on the public embeddings is 0.54. An interesting question though how much it overfits the leaderboard. I am hoping to get a properly crossvalidated embedding version that gets 0.55.", "votes": 2, "isDeleted": true, "replies": [ { "id": 2291444, "postDate": "2023-06-07T14:53:05.897Z", "content": "My feeling is that overfitting is not too big a problem as my cross validation models individually agree quite well with one another but it is good to have another perspective on baseline - my team is adding new datasets to our model and having an idea of how far embeddings can take you is important imo.
", "rawMarkdown": "My feeling is that overfitting is not too big a problem as my cross validation models individually agree quite well with one another but it is good to have another perspective on baseline - my team is adding new datasets to our model and having an idea of how far embeddings can take you is important imo. ", "votes": 1 } ] } ], "comments": [ { "id": 2290930, "author_name": "Victor Fernandez Albor", "author_url": "", "post_date": "2023-06-07T08:01:16.213000", "content": "I think you are right in your assumption, maybe the top limit I bet is 0.56-0.57 if you are very very good in statistics and linear regression, but you need something else. Also with kaggle resources I think that you have to reduce a lot your dimensionality in order to train something.
\nPeople that will achieve better resources in this comp. (mainly labs that have participated in past editions) they'll create their own models and they'll use their own resources to create some successful model.
", "votes": 2, "replies": [ { "id": 2291308, "author_name": "ZMC", "author_url": "", "post_date": "2023-06-07T13:14:33.250000", "content": "There are tricks to save ram, I have been able to train with all 3 embeddings easily below the max RAM limit on kaggle with GPU. Good to know the limit is likely higher than I assumed!
", "votes": 1, "replies": [ { "id": 2291339, "author_name": "Victor Fernandez Albor", "author_url": "", "post_date": "2023-06-07T13:43:00.813000", "content": "The tricks that i know less parameters and low dimensionality…
", "votes": 1, "replies": [] }, { "id": 2291423, "author_name": "ZMC", "author_url": "", "post_date": "2023-06-07T14:43:23.177000", "content": "Change the way you load in data and data type will get you far.. :)
", "votes": 1, "replies": [] } ] } ] }, { "id": 2291428, "author_name": "", "author_url": "", "post_date": "2023-06-07T14:45:19.210000", "content": "You might be right, however I feel this approach should be properly investigated as a baseline, before starting something really complicated. So far my best score based purely on the public embeddings is 0.54. An interesting question though how much it overfits the leaderboard. I am hoping to get a properly crossvalidated embedding version that gets 0.55.
", "votes": 2, "replies": [ { "id": 2291444, "author_name": "ZMC", "author_url": "", "post_date": "2023-06-07T14:53:05.897000", "content": "My feeling is that overfitting is not too big a problem as my cross validation models individually agree quite well with one another but it is good to have another perspective on baseline - my team is adding new datasets to our model and having an idea of how far embeddings can take you is important imo.
", "votes": 1, "replies": [] } ] } ], "raw_markdown_by_id": { "2290914": "Based on the limits of the modelling I have done, I hope to not be totally wrong in suggesting that ~0.53 is the highest score achievable simply based on the publicly available sequence derived embeddings (T5/EMS2/ProtBert). I would be quite surprised to learn that there are more ways to wring out a higher score based simply on these embeddings and if so will be most delighted to be contradicted by those higher on the leaderboard! This post is thus also a sign to those that are working on this problem using embeddings that ~0.53 is achievable based solely on that data. Good luck all! \n\nEdit: The limit is significantly higher, my apologies.", "2290930": "I think you are right in your assumption, maybe the top limit I bet is 0.56-0.57 if you are very very good in statistics and linear regression, but you need something else. Also with kaggle resources I think that you have to reduce a lot your dimensionality in order to train something. \n\nPeople that will achieve better resources in this comp. (mainly labs that have participated in past editions) they'll create their own models and they'll use their own resources to create some successful model.", "2291428": "You might be right, however I feel this approach should be properly investigated as a baseline, before starting something really complicated. So far my best score based purely on the public embeddings is 0.54. An interesting question though how much it overfits the leaderboard. I am hoping to get a properly crossvalidated embedding version that gets 0.55." } }