First of all, I want to thank the organizers for this very important and interesting competition. It was very difficult and unusual (it's not every day you have to do multilabel classification for such a large number of classes), but that makes it even more interesting.
\nIn my solution, I used data from embedding models such as the T5 and ESM2 T33 650M. BERT showed much worse results, ESM2 3B gave more overfitting.
\nI've used this in the following variations:
\nAlso I have used data extracted with ProteinAnalysis from Biopython module, and amino acid count information.
\nThe models were trained on different target sets selected and sorted by frequency.
\nNo model has been trained on more than 2000 classes at a time if it is a neural network, and on 3100 classes if it is a Ridge model (due to RAM limitations on the :D). But NNs showed better results with fewer classes (1000-1500).
\nAlso as we have a lot of classes, the clustering of targets also performed well. I used K-means.
First I tried many sklearn, tensorflow and pytorch models, but after that I left only Ridge from sklearn, and these pytorch architectures:
\nI tried using a metamodel for stacking, but it didn't work out well.
\nI tried BCE/Focal losses, but F1 performed much better for me (especially after thresholds calculation).
\nAlso I wrote custom SampledF1Loss. The main idea is to select a random set of targets (of a given range, e.g. from 20 to 80%) from each batches, and compute the loss only on them. This increases the speed, and also increases the variety of gradients.
\nLion optimizer, Sophia, augmentations also didn't work good enough.
Taking the N-top values is important, because I've noticed that models don't realize how many terms a protein actually has. I've tried to make regression on the terms count, and the results were awful.
", "messages": [ { "id": 2411640, "postDate": "2023-08-27T18:53:59.913Z", "content": "First of all, I want to thank the organizers for this very important and interesting competition. It was very difficult and unusual (it's not every day you have to do multilabel classification for such a large number of classes), but that makes it even more interesting.
\nIn my solution, I used data from embedding models such as the T5 and ESM2 T33 650M. BERT showed much worse results, ESM2 3B gave more overfitting.
\nI've used this in the following variations:
\nAlso I have used data extracted with ProteinAnalysis from Biopython module, and amino acid count information.
\nThe models were trained on different target sets selected and sorted by frequency.
\nNo model has been trained on more than 2000 classes at a time if it is a neural network, and on 3100 classes if it is a Ridge model (due to RAM limitations on the :D). But NNs showed better results with fewer classes (1000-1500).
\nAlso as we have a lot of classes, the clustering of targets also performed well. I used K-means.
First I tried many sklearn, tensorflow and pytorch models, but after that I left only Ridge from sklearn, and these pytorch architectures:
\nI tried using a metamodel for stacking, but it didn't work out well.
\nI tried BCE/Focal losses, but F1 performed much better for me (especially after thresholds calculation).
\nAlso I wrote custom SampledF1Loss. The main idea is to select a random set of targets (of a given range, e.g. from 20 to 80%) from each batches, and compute the loss only on them. This increases the speed, and also increases the variety of gradients.
\nLion optimizer, Sophia, augmentations also didn't work good enough.
Taking the N-top values is important, because I've noticed that models don't realize how many terms a protein actually has. I've tried to make regression on the terms count, and the results were awful.
", "rawMarkdown": "First of all, I want to thank the organizers for this very important and interesting competition. It was very difficult and unusual (it's not every day you have to do multilabel classification for such a large number of classes), but that makes it even more interesting.\n\n# Data \n\n### What worked for me\n\nIn my solution, I used data from embedding models such as the **T5** and **ESM2 T33 650M**. **BERT** showed much worse results, **ESM2 3B** gave more overfitting. \n\nI've used this in the following variations:\n- Separately (only T5/only ESM)\n- Combined (simple concatenation)\n- Combined after PCA512\n- Combined after feature selection based on RF features importances\n\nAlso I have used data extracted with *ProteinAnalysis* from *Biopython* module, and amino acid count information.\n\n### What didn't worked for me\n\n- K-MER data\n- TF-IDF\n- tSVD for embeddings\n- tSVD for labels\n\n### Targets\n\nThe models were trained on different target sets selected and sorted by frequency. \nNo model has been trained on more than 2000 classes at a time if it is a neural network, and on 3100 classes if it is a **Ridge** model (due to RAM limitations on the :D). But NNs showed better results with fewer classes (1000-1500).\nAlso as we have a lot of classes, the clustering of targets also performed well. I used K-means.\n\n# Modeling part\n\n### Models\n\nFirst I tried many *sklearn*, *tensorflow* and *pytorch* models, but after that I left only **Ridge** from *sklearn*, and these *pytorch* architectures:\n- FCNN\n- NN with both conv and FC encoders\n- Residual FCNN\n- Residual CNN\n- All variations mentioned above, but with 3 heads (one head for one subontology)\n\nI tried using a metamodel for stacking, but it didn't work out well.\n\n### Training\n\n- Training only on samples containing all subontologies\n- For data splitting I used **MultilabelStratifiedKFold** with 4/5/6 folds\n- **Quantile Transformer** & **MinMax Scaler** (0-1) from *sklearn* as preprocessing\n- Adam optimizer\n- MacroF1 metric for best model choosing\n- MacroF1Loss\n- Pseudo-labeling\n\nI tried BCE/Focal losses, but F1 performed much better for me (especially after thresholds calculation).\nAlso I wrote custom SampledF1Loss. The main idea is to select a random set of targets (of a given range, e.g. from 20 to 80%) from each batches, and compute the loss only on them. This increases the speed, and also increases the variety of gradients.\nLion optimizer, Sophia, augmentations also didn't work good enough.\n\n### Postprocessing\n\n1. For each model I calculated best thresholds based on F1 metric (I have calculated it for each class, not for each subontology).\n2. For submission I used values only exceeding the threshold for each trained class.\n3. Blended all submissions with pandas (group by ID & class, taking an averaged value, taking the top-50 highest values for each protein ID).\n\nTaking the N-top values is important, because I've noticed that models don't realize how many terms a protein actually has. I've tried to make regression on the terms count, and the results were awful.", "votes": 19 } ], "comments": [], "raw_markdown_by_id": { "2411640": "First of all, I want to thank the organizers for this very important and interesting competition. It was very difficult and unusual (it's not every day you have to do multilabel classification for such a large number of classes), but that makes it even more interesting.\n\n# Data \n\n### What worked for me\n\nIn my solution, I used data from embedding models such as the **T5** and **ESM2 T33 650M**. **BERT** showed much worse results, **ESM2 3B** gave more overfitting. \n\nI've used this in the following variations:\n- Separately (only T5/only ESM)\n- Combined (simple concatenation)\n- Combined after PCA512\n- Combined after feature selection based on RF features importances\n\nAlso I have used data extracted with *ProteinAnalysis* from *Biopython* module, and amino acid count information.\n\n### What didn't worked for me\n\n- K-MER data\n- TF-IDF\n- tSVD for embeddings\n- tSVD for labels\n\n### Targets\n\nThe models were trained on different target sets selected and sorted by frequency. \nNo model has been trained on more than 2000 classes at a time if it is a neural network, and on 3100 classes if it is a **Ridge** model (due to RAM limitations on the :D). But NNs showed better results with fewer classes (1000-1500).\nAlso as we have a lot of classes, the clustering of targets also performed well. I used K-means.\n\n# Modeling part\n\n### Models\n\nFirst I tried many *sklearn*, *tensorflow* and *pytorch* models, but after that I left only **Ridge** from *sklearn*, and these *pytorch* architectures:\n- FCNN\n- NN with both conv and FC encoders\n- Residual FCNN\n- Residual CNN\n- All variations mentioned above, but with 3 heads (one head for one subontology)\n\nI tried using a metamodel for stacking, but it didn't work out well.\n\n### Training\n\n- Training only on samples containing all subontologies\n- For data splitting I used **MultilabelStratifiedKFold** with 4/5/6 folds\n- **Quantile Transformer** & **MinMax Scaler** (0-1) from *sklearn* as preprocessing\n- Adam optimizer\n- MacroF1 metric for best model choosing\n- MacroF1Loss\n- Pseudo-labeling\n\nI tried BCE/Focal losses, but F1 performed much better for me (especially after thresholds calculation).\nAlso I wrote custom SampledF1Loss. The main idea is to select a random set of targets (of a given range, e.g. from 20 to 80%) from each batches, and compute the loss only on them. This increases the speed, and also increases the variety of gradients.\nLion optimizer, Sophia, augmentations also didn't work good enough.\n\n### Postprocessing\n\n1. For each model I calculated best thresholds based on F1 metric (I have calculated it for each class, not for each subontology).\n2. For submission I used values only exceeding the threshold for each trained class.\n3. Blended all submissions with pandas (group by ID & class, taking an averaged value, taking the top-50 highest values for each protein ID).\n\nTaking the N-top values is important, because I've noticed that models don't realize how many terms a protein actually has. I've tried to make regression on the terms count, and the results were awful." } }