Hi Kagglers.
\nThere are different ways to approach this competition, ranging from RNNs and CNNs, or using GBTs without NNs.
\nThe approach I am considering now is Transformer, as the topic title suggests.
\n@cdeotte, one of the best Kaggler, shows that transformers' CV and LB are superior to RNNs and CNNs on the topic of CV vs LB scores.
\nHis post reminded me of the Google Brain - Ventilator Pressure Prediction competition.
\nHe won the gold medal in 13th place in the competition and stated the adoption of Transformers in the solution.
\nThere was a leak in the competition, but he won without using it.
\nHis Transformer implementation can be seen here.
\nWe can assume that he is also using a transformer for this competition.
\nAs I mentioned earlier, I am now thinking of using a Transformer, but at the same time I have a problem. Specifically, they are as follows.
Training does not converge.
\nIt is difficult to train a transformer from the random init. And this competition does not distribute enough training data for convergence. I have used several pretrained models from the HuggingFace hub. But they are inferior to CNNs' loss.
The length of the series is too long.
\nBecause of the attention mechanism, the computational complexity of the transformer increases as the square of the series length. In the ventilator competition, the series length was constant and short and could be ignored, but in this competition the series length is not constant and very long.
What model is suitable?
\nThe fundamental problem is that I do not know what of the transformers to use. There are many kinds of transformers. Moreover, Take BERT for example, The scale and structure of each model is different from one another(e.g. num_hidden_layers, hidden_size). Pretrained models also differ in what they are optimized for(e.g. Language, Task).
Chris has solved these problems and is sitting the Gold Zone. The same may be true for other gold medal candidates.
\nIs anyone else attempting to train in Transformers as I am?
\nDo you have any ideas for successful training in Transformers?
\nPlease let me know what you think!
Hi Kagglers.
\nThere are different ways to approach this competition, ranging from RNNs and CNNs, or using GBTs without NNs.
\nThe approach I am considering now is Transformer, as the topic title suggests.
\n@cdeotte, one of the best Kaggler, shows that transformers' CV and LB are superior to RNNs and CNNs on the topic of CV vs LB scores.
\nHis post reminded me of the Google Brain - Ventilator Pressure Prediction competition.
\nHe won the gold medal in 13th place in the competition and stated the adoption of Transformers in the solution.
\nThere was a leak in the competition, but he won without using it.
\nHis Transformer implementation can be seen here.
\nWe can assume that he is also using a transformer for this competition.
\nAs I mentioned earlier, I am now thinking of using a Transformer, but at the same time I have a problem. Specifically, they are as follows.
Training does not converge.
\nIt is difficult to train a transformer from the random init. And this competition does not distribute enough training data for convergence. I have used several pretrained models from the HuggingFace hub. But they are inferior to CNNs' loss.
The length of the series is too long.
\nBecause of the attention mechanism, the computational complexity of the transformer increases as the square of the series length. In the ventilator competition, the series length was constant and short and could be ignored, but in this competition the series length is not constant and very long.
What model is suitable?
\nThe fundamental problem is that I do not know what of the transformers to use. There are many kinds of transformers. Moreover, Take BERT for example, The scale and structure of each model is different from one another(e.g. num_hidden_layers, hidden_size). Pretrained models also differ in what they are optimized for(e.g. Language, Task).
Chris has solved these problems and is sitting the Gold Zone. The same may be true for other gold medal candidates.
\nIs anyone else attempting to train in Transformers as I am?
\nDo you have any ideas for successful training in Transformers?
\nPlease let me know what you think!
My first transformer submission scored .635 despite training for a single epoch on a small portion of the available data. The model was relatively small (~700k parameters) and trained from scratch.
\nI recommend starting small so you can quickly try many different designs.
", "rawMarkdown": "My first transformer submission scored .635 despite training for a single epoch on a small portion of the available data. The model was relatively small (~700k parameters) and trained from scratch. \n\nI recommend starting small so you can quickly try many different designs.", "votes": 8, "replies": [ { "id": 2503648, "postDate": "2023-10-29T10:22:26.057Z", "content": "Very interesting post. It's great that you were able to successfully train the transformers with a small data set. I have tried BERTs of various sizes from 2 to 12 layers and they all did not work. I use the same preprocessing as the my public notebook, but it may differ from your method. I look forward to the publication of your solution.
", "rawMarkdown": "Very interesting post. It's great that you were able to successfully train the transformers with a small data set. I have tried BERTs of various sizes from 2 to 12 layers and they all did not work. I use the same preprocessing as the my public notebook, but it may differ from your method. I look forward to the publication of your solution.", "votes": 3, "replies": [ { "id": 2508574, "postDate": "2023-11-01T19:52:12.330Z", "content": "I tried some transformer models, but I'm having trouble training them at all. It seems I miss something.
", "rawMarkdown": "I tried some transformer models, but I'm having trouble training them at all. It seems I miss something.", "votes": 1 } ] } ] }, { "id": 2503393, "postDate": "2023-10-29T03:55:38.850Z", "content": "my best score (0.685) is a single fold transformer, although I am pretty sure, its far from a good transformer. Transformers do have potential
", "rawMarkdown": "my best score (0.685) is a single fold transformer, although I am pretty sure, its far from a good transformer. Transformers do have potential", "votes": 5 }, { "id": 2503564, "postDate": "2023-10-29T07:58:59.870Z", "content": "As for the series, I believe you should split them to pieces by using window of length L and stride of L//2, like it was done here. I'm using this approach (after I aggregate the series to 1 min step (factor of 12)), and I'm getting quite good results with very small time required to train the model. But I haven't tried transformers yet - only RNN and CNN+RNN.
", "rawMarkdown": "As for the series, I believe you should split them to pieces by using window of length L and stride of L//2, like it was done [here](https://www.kaggle.com/competitions/tlvmc-parkinsons-freezing-gait-prediction/discussion/416057). I'm using this approach (after I aggregate the series to 1 min step (factor of 12)), and I'm getting quite good results with very small time required to train the model. But I haven't tried transformers yet - only RNN and CNN+RNN.", "votes": 6, "replies": [ { "id": 2503658, "postDate": "2023-10-29T10:33:12.560Z", "content": "Compressing the time to minutes, as you suggest, is a reasonable measure to alleviate the problem of the series length of the Transformer. If the model is not large, we can train the model using chunks for each day (1440 minutes and points), for example.
\nI have already tried this but have yet to successfully train.
Do you need to make sure that there is only one event in every window. I mean if there are two or three events in a window, how to deal with it?
", "rawMarkdown": "Do you need to make sure that there is only one event in every window. I mean if there are two or three events in a window, how to deal with it?", "replies": [ { "id": 2504384, "postDate": "2023-10-29T19:51:41.483Z", "content": "No, I haven't introduced any such limitations. As for details you asked about - see the link in my post.
", "rawMarkdown": "No, I haven't introduced any such limitations. As for details you asked about - see the link in my post." }, { "id": 2504395, "postDate": "2023-10-29T19:59:19.423Z", "content": "Ok, thanks for your great share.
", "rawMarkdown": "Ok, thanks for your great share." } ] }, { "id": 2505753, "postDate": "2023-10-30T19:19:44.690Z", "content": "Are you training the models in Kaggle's notebooks? What is small time? In my case training few epochs takes more than 1 hour :/
", "rawMarkdown": "Are you training the models in Kaggle's notebooks? What is small time? In my case training few epochs takes more than 1 hour :/", "votes": 1, "replies": [ { "id": 2508575, "postDate": "2023-11-01T19:53:21.353Z", "content": "Yes, I do. As for the training time, you probably need to downsample and/or split the series to a number of blocks. In my case 1 epoch takes ~ 1min.
", "rawMarkdown": "Yes, I do. As for the training time, you probably need to downsample and/or split the series to a number of blocks. In my case 1 epoch takes ~ 1min." }, { "id": 2508576, "postDate": "2023-11-01T19:55:07.407Z", "content": "Also remember, that looking for a specific series in a large pandas dataframe each time you call your pytorch dataset is extremely slow, so you might want to use numpy arrays instead.
", "rawMarkdown": "Also remember, that looking for a specific series in a large pandas dataframe each time you call your pytorch dataset is extremely slow, so you might want to use numpy arrays instead.", "votes": 1 } ] } ] }, { "id": 2502920, "postDate": "2023-10-28T15:07:24.053Z", "content": "I've tried training transformers too with various setups, but no luck so far! :D
", "rawMarkdown": "I've tried training transformers too with various setups, but no luck so far! :D", "votes": 6 }, { "id": 2503780, "postDate": "2023-10-29T12:47:05.627Z", "content": "@itsuki9180 To what extent can your local training loss and valid loss converge? I use transformer locally (currently training). The training loss has now dropped steadily to 0.739610, and the valid loss has dropped steadily to 0.698006. I observed the previous training set and validation set. Loss, the loss on the validation set is basically lower than the loss on the training set. Although it decreases slowly, the direction of convergence seems to be very good. I have not used the evaluation method to check the AP of each verification.
", "rawMarkdown": "@itsuki9180 To what extent can your local training loss and valid loss converge? I use transformer locally (currently training). The training loss has now dropped steadily to 0.739610, and the valid loss has dropped steadily to 0.698006. I observed the previous training set and validation set. Loss, the loss on the validation set is basically lower than the loss on the training set. Although it decreases slowly, the direction of convergence seems to be very good. I have not used the evaluation method to check the AP of each verification.", "votes": 2, "replies": [ { "id": 2503849, "postDate": "2023-10-29T14:02:23.613Z", "content": "Which loss are you using? My losses are much smaller (I use binary cross entropy).
", "rawMarkdown": "Which loss are you using? My losses are much smaller (I use binary cross entropy).", "replies": [ { "id": 2503917, "postDate": "2023-10-29T14:41:33.827Z", "content": "I use MSELoss
", "rawMarkdown": "I use MSELoss", "votes": 1 } ] }, { "id": 2504998, "postDate": "2023-10-30T09:52:56.160Z", "content": "Based on my public notebook, when the label shape is Gaussian distribution and the loss function is Focal Loss, the valid loss is in the range of 0.003800~0.004800. If the loss is less than this value with trained transformers, we can consider the training is successful. In reality, however, the valid loss will stop at around 0.020000.
", "rawMarkdown": "Based on [my public notebook](https://www.kaggle.com/code/itsuki9180/detect-sleep-states-train), when the label shape is Gaussian distribution and the loss function is Focal Loss, the valid loss is in the range of 0.003800~0.004800. If the loss is less than this value with trained transformers, we can consider the training is successful. In reality, however, the valid loss will stop at around 0.020000.", "votes": 1, "replies": [ { "id": 2505026, "postDate": "2023-10-30T10:12:43.510Z", "content": "Do you think using FocalLoss is a good choice? In fact, when I first saw the imbalance between positive and negative samples, I wanted to use Focalloss like you. The effect of using focalloss is not as good as mseloss. What is the reason for this?
", "rawMarkdown": "Do you think using FocalLoss is a good choice? In fact, when I first saw the imbalance between positive and negative samples, I wanted to use Focalloss like you. The effect of using focalloss is not as good as mseloss. What is the reason for this?", "votes": 1 }, { "id": 2505098, "postDate": "2023-10-30T11:10:52.367Z", "content": "I'm having quite close loss values, although I use simple binary cross entropy.
", "rawMarkdown": "I'm having quite close loss values, although I use simple binary cross entropy.", "votes": 1 }, { "id": 2506043, "postDate": "2023-10-31T04:03:35.847Z", "content": "I think that the appropriate loss function depends on how the preprocessing, especially the labeling of event points, is dealt with. Focal Loss is better than MSE, at least in my public baseline. I don't know what preprocessing you are using, but sometimes MSE is a good loss function, as in @werus23's RNN baseline.
", "rawMarkdown": "I think that the appropriate loss function depends on how the preprocessing, especially the labeling of event points, is dealt with. Focal Loss is better than MSE, at least in my public baseline. I don't know what preprocessing you are using, but sometimes MSE is a good loss function, as in @werus23's [RNN baseline](https://www.kaggle.com/code/werus23/sleep-critical-point-train).", "votes": 1 }, { "id": 2506044, "postDate": "2023-10-31T04:06:58.070Z", "content": "If the valid loss in BCE is around 0.004000 it can be considered very low. Unfortunately, however, different loss functions are calculated in different ways, so it does not make sense to compare them. We need to discuss this using the AP, which is a measure of competition metrics.
", "rawMarkdown": "If the valid loss in BCE is around 0.004000 it can be considered very low. Unfortunately, however, different loss functions are calculated in different ways, so it does not make sense to compare them. We need to discuss this using the AP, which is a measure of competition metrics.", "votes": 1 }, { "id": 2508572, "postDate": "2023-11-01T19:49:57.580Z", "content": "Focal loss is a good idea, and I plan to try it. As for AP - I'm having a problem of large gap between CV and LB. The LB in my case heavily relies on windows size selected during the inference. I noticed that larger window size than used during the training (but not too large) leads to better result, but the gap is still large (~0.07-0.1).
", "rawMarkdown": "Focal loss is a good idea, and I plan to try it. As for AP - I'm having a problem of large gap between CV and LB. The LB in my case heavily relies on windows size selected during the inference. I noticed that larger window size than used during the training (but not too large) leads to better result, but the gap is still large (~0.07-0.1).", "votes": 1 }, { "id": 2508573, "postDate": "2023-11-01T19:50:34.410Z", "content": "Since I treat this problem as a multi-label classification, I don't use MSE in my pipeline.
", "rawMarkdown": "Since I treat this problem as a multi-label classification, I don't use MSE in my pipeline." } ] } ] }, { "id": 2510485, "postDate": "2023-11-03T03:23:11.717Z", "content": "Transformers might not be feasible for such long sequences. To even get a reasonable sequence length for training you'd have to slice step the sequences down by a couple factors, but then you'd lose precision. LSTM is better in these scenarios. However, maybe if you really cut down the amount of steps, and make each step have features representing the steps around it that were sliced out, it might work if you also couple other methods. For example, say you take the training data, divide it into steps of 30 minutes, and feature engineer each step so that it represents the 15 minutes worth of time steps before and after it such at a center rolling average of period 300 5 sec steps. This would result in guesses every 30 min in the sequence. Then take the steps that were guessed as either onset or awake by the model, and run them through another model that takes the 300 5 second time steps(15min) before and after the guess to make a precise prediction of where the onset/awake happened.
", "rawMarkdown": "Transformers might not be feasible for such long sequences. To even get a reasonable sequence length for training you'd have to slice step the sequences down by a couple factors, but then you'd lose precision. LSTM is better in these scenarios. However, maybe if you really cut down the amount of steps, and make each step have features representing the steps around it that were sliced out, it might work if you also couple other methods. For example, say you take the training data, divide it into steps of 30 minutes, and feature engineer each step so that it represents the 15 minutes worth of time steps before and after it such at a center rolling average of period 300 5 sec steps. This would result in guesses every 30 min in the sequence. Then take the steps that were guessed as either onset or awake by the model, and run them through another model that takes the 300 5 second time steps(15min) before and after the guess to make a precise prediction of where the onset/awake happened. " }, { "id": 2508419, "postDate": "2023-11-01T17:40:57.617Z", "content": "Does anyone have any ideas on how to train a transformer with one day as input (17280 steps) and the output as a Gaussian of the same size? Which structure to choose (encoder, decoder, both) ?
", "rawMarkdown": "Does anyone have any ideas on how to train a transformer with one day as input (17280 steps) and the output as a Gaussian of the same size? Which structure to choose (encoder, decoder, both) ?" } ], "comments": [ { "id": 2502950, "author_name": "Quillen", "author_url": "", "post_date": "2023-10-28T15:47:19.750000", "content": "My first transformer submission scored .635 despite training for a single epoch on a small portion of the available data. The model was relatively small (~700k parameters) and trained from scratch.
\nI recommend starting small so you can quickly try many different designs.
", "votes": 8, "replies": [ { "id": 2503648, "author_name": "ITK8191", "author_url": "", "post_date": "2023-10-29T10:22:26.057000", "content": "Very interesting post. It's great that you were able to successfully train the transformers with a small data set. I have tried BERTs of various sizes from 2 to 12 layers and they all did not work. I use the same preprocessing as the my public notebook, but it may differ from your method. I look forward to the publication of your solution.
", "votes": 3, "replies": [ { "id": 2508574, "author_name": "Araik Tamazian", "author_url": "", "post_date": "2023-11-01T19:52:12.330000", "content": "I tried some transformer models, but I'm having trouble training them at all. It seems I miss something.
", "votes": 1, "replies": [] } ] } ] }, { "id": 2503393, "author_name": "NikhilMishra", "author_url": "", "post_date": "2023-10-29T03:55:38.850000", "content": "my best score (0.685) is a single fold transformer, although I am pretty sure, its far from a good transformer. Transformers do have potential
", "votes": 5, "replies": [] }, { "id": 2503564, "author_name": "Araik Tamazian", "author_url": "", "post_date": "2023-10-29T07:58:59.870000", "content": "As for the series, I believe you should split them to pieces by using window of length L and stride of L//2, like it was done here. I'm using this approach (after I aggregate the series to 1 min step (factor of 12)), and I'm getting quite good results with very small time required to train the model. But I haven't tried transformers yet - only RNN and CNN+RNN.
", "votes": 6, "replies": [ { "id": 2503658, "author_name": "ITK8191", "author_url": "", "post_date": "2023-10-29T10:33:12.560000", "content": "Compressing the time to minutes, as you suggest, is a reasonable measure to alleviate the problem of the series length of the Transformer. If the model is not large, we can train the model using chunks for each day (1440 minutes and points), for example.
\nI have already tried this but have yet to successfully train.
Do you need to make sure that there is only one event in every window. I mean if there are two or three events in a window, how to deal with it?
", "votes": 0, "replies": [ { "id": 2504384, "author_name": "Araik Tamazian", "author_url": "", "post_date": "2023-10-29T19:51:41.483000", "content": "No, I haven't introduced any such limitations. As for details you asked about - see the link in my post.
", "votes": 0, "replies": [] }, { "id": 2504395, "author_name": "Zhecheng Li", "author_url": "", "post_date": "2023-10-29T19:59:19.423000", "content": "Ok, thanks for your great share.
", "votes": 0, "replies": [] } ] }, { "id": 2505753, "author_name": "Andreu Arderiu", "author_url": "", "post_date": "2023-10-30T19:19:44.690000", "content": "Are you training the models in Kaggle's notebooks? What is small time? In my case training few epochs takes more than 1 hour :/
", "votes": 1, "replies": [ { "id": 2508575, "author_name": "Araik Tamazian", "author_url": "", "post_date": "2023-11-01T19:53:21.353000", "content": "Yes, I do. As for the training time, you probably need to downsample and/or split the series to a number of blocks. In my case 1 epoch takes ~ 1min.
", "votes": 0, "replies": [] }, { "id": 2508576, "author_name": "Araik Tamazian", "author_url": "", "post_date": "2023-11-01T19:55:07.407000", "content": "Also remember, that looking for a specific series in a large pandas dataframe each time you call your pytorch dataset is extremely slow, so you might want to use numpy arrays instead.
", "votes": 1, "replies": [] } ] } ] }, { "id": 2502920, "author_name": "slime", "author_url": "", "post_date": "2023-10-28T15:07:24.053000", "content": "I've tried training transformers too with various setups, but no luck so far! :D
", "votes": 6, "replies": [] }, { "id": 2503780, "author_name": "Donghui Zhang", "author_url": "", "post_date": "2023-10-29T12:47:05.627000", "content": "@itsuki9180 To what extent can your local training loss and valid loss converge? I use transformer locally (currently training). The training loss has now dropped steadily to 0.739610, and the valid loss has dropped steadily to 0.698006. I observed the previous training set and validation set. Loss, the loss on the validation set is basically lower than the loss on the training set. Although it decreases slowly, the direction of convergence seems to be very good. I have not used the evaluation method to check the AP of each verification.
", "votes": 2, "replies": [ { "id": 2503849, "author_name": "Araik Tamazian", "author_url": "", "post_date": "2023-10-29T14:02:23.613000", "content": "Which loss are you using? My losses are much smaller (I use binary cross entropy).
", "votes": 0, "replies": [ { "id": 2503917, "author_name": "Donghui Zhang", "author_url": "", "post_date": "2023-10-29T14:41:33.827000", "content": "I use MSELoss
", "votes": 1, "replies": [] } ] }, { "id": 2504998, "author_name": "ITK8191", "author_url": "", "post_date": "2023-10-30T09:52:56.160000", "content": "Based on my public notebook, when the label shape is Gaussian distribution and the loss function is Focal Loss, the valid loss is in the range of 0.003800~0.004800. If the loss is less than this value with trained transformers, we can consider the training is successful. In reality, however, the valid loss will stop at around 0.020000.
", "votes": 1, "replies": [ { "id": 2505026, "author_name": "Donghui Zhang", "author_url": "", "post_date": "2023-10-30T10:12:43.510000", "content": "Do you think using FocalLoss is a good choice? In fact, when I first saw the imbalance between positive and negative samples, I wanted to use Focalloss like you. The effect of using focalloss is not as good as mseloss. What is the reason for this?
", "votes": 1, "replies": [] }, { "id": 2505098, "author_name": "Araik Tamazian", "author_url": "", "post_date": "2023-10-30T11:10:52.367000", "content": "I'm having quite close loss values, although I use simple binary cross entropy.
", "votes": 1, "replies": [] }, { "id": 2506043, "author_name": "ITK8191", "author_url": "", "post_date": "2023-10-31T04:03:35.847000", "content": "I think that the appropriate loss function depends on how the preprocessing, especially the labeling of event points, is dealt with. Focal Loss is better than MSE, at least in my public baseline. I don't know what preprocessing you are using, but sometimes MSE is a good loss function, as in @werus23's RNN baseline.
", "votes": 1, "replies": [] }, { "id": 2506044, "author_name": "ITK8191", "author_url": "", "post_date": "2023-10-31T04:06:58.070000", "content": "If the valid loss in BCE is around 0.004000 it can be considered very low. Unfortunately, however, different loss functions are calculated in different ways, so it does not make sense to compare them. We need to discuss this using the AP, which is a measure of competition metrics.
", "votes": 1, "replies": [] }, { "id": 2508572, "author_name": "Araik Tamazian", "author_url": "", "post_date": "2023-11-01T19:49:57.580000", "content": "Focal loss is a good idea, and I plan to try it. As for AP - I'm having a problem of large gap between CV and LB. The LB in my case heavily relies on windows size selected during the inference. I noticed that larger window size than used during the training (but not too large) leads to better result, but the gap is still large (~0.07-0.1).
", "votes": 1, "replies": [] }, { "id": 2508573, "author_name": "Araik Tamazian", "author_url": "", "post_date": "2023-11-01T19:50:34.410000", "content": "Since I treat this problem as a multi-label classification, I don't use MSE in my pipeline.
", "votes": 0, "replies": [] } ] } ] }, { "id": 2510485, "author_name": "Aleksander Klimczyk", "author_url": "", "post_date": "2023-11-03T03:23:11.717000", "content": "Transformers might not be feasible for such long sequences. To even get a reasonable sequence length for training you'd have to slice step the sequences down by a couple factors, but then you'd lose precision. LSTM is better in these scenarios. However, maybe if you really cut down the amount of steps, and make each step have features representing the steps around it that were sliced out, it might work if you also couple other methods. For example, say you take the training data, divide it into steps of 30 minutes, and feature engineer each step so that it represents the 15 minutes worth of time steps before and after it such at a center rolling average of period 300 5 sec steps. This would result in guesses every 30 min in the sequence. Then take the steps that were guessed as either onset or awake by the model, and run them through another model that takes the 300 5 second time steps(15min) before and after the guess to make a precise prediction of where the onset/awake happened.
", "votes": 0, "replies": [] }, { "id": 2508419, "author_name": "Florian Delaplace", "author_url": "", "post_date": "2023-11-01T17:40:57.617000", "content": "Does anyone have any ideas on how to train a transformer with one day as input (17280 steps) and the output as a Gaussian of the same size? Which structure to choose (encoder, decoder, both) ?
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "2502725": "Hi Kagglers.\nThere are different ways to approach this competition, ranging from RNNs and CNNs, or using GBTs without NNs.\nThe approach I am considering now is **Transformer**, as the topic title suggests.\n@cdeotte, one of the best Kaggler, [shows that transformers' CV and LB are superior to RNNs and CNNs](https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/447882#2491744) on the topic of [CV vs LB scores](https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/447882).\nHis post reminded me of the [Google Brain - Ventilator Pressure Prediction](https://www.kaggle.com/competitions/ventilator-pressure-prediction) competition.\n[He won the gold medal in 13th place in the competition](https://www.kaggle.com/competitions/ventilator-pressure-prediction/leaderboard) and [stated the adoption of Transformers in the solution](https://www.kaggle.com/competitions/ventilator-pressure-prediction/discussion/285435).\nThere was a leak in the competition, but he won without using it.\nHis Transformer implementation can be seen [here](https://www.kaggle.com/code/cdeotte/tensorflow-transformer-0-112).\nWe can assume that he is also using a transformer for this competition.\nAs I mentioned earlier, I am now thinking of using a Transformer, but at the same time I have a problem. Specifically, they are as follows.\n\n- **Training does not converge.**\nIt is difficult to train a transformer from the random init. And this competition does not distribute enough training data for convergence. I have used several pretrained models from the HuggingFace hub. But they are inferior to CNNs' loss.\n\n- **The length of the series is too long.**\nBecause of the attention mechanism, the computational complexity of the transformer increases as the square of the series length. In the ventilator competition, the series length was constant and short and could be ignored, but in this competition the series length is not constant and very long.\n\n- **What model is suitable?**\nThe fundamental problem is that I do not know what of the transformers to use. There are many kinds of transformers. Moreover, Take BERT for example, The scale and structure of each model is different from one another(e.g. num_hidden_layers, hidden_size). Pretrained models also differ in what they are optimized for(e.g. Language, Task).\n\nChris has solved these problems and is sitting the Gold Zone. The same may be true for other gold medal candidates.\nIs anyone else attempting to train in Transformers as I am?\nDo you have any ideas for successful training in Transformers?\nPlease let me know what you think!\n", "2502950": "My first transformer submission scored .635 despite training for a single epoch on a small portion of the available data. The model was relatively small (~700k parameters) and trained from scratch. \n\nI recommend starting small so you can quickly try many different designs.", "2503393": "my best score (0.685) is a single fold transformer, although I am pretty sure, its far from a good transformer. Transformers do have potential", "2503564": "As for the series, I believe you should split them to pieces by using window of length L and stride of L//2, like it was done [here](https://www.kaggle.com/competitions/tlvmc-parkinsons-freezing-gait-prediction/discussion/416057). I'm using this approach (after I aggregate the series to 1 min step (factor of 12)), and I'm getting quite good results with very small time required to train the model. But I haven't tried transformers yet - only RNN and CNN+RNN.", "2502920": "I've tried training transformers too with various setups, but no luck so far! :D", "2503780": "@itsuki9180 To what extent can your local training loss and valid loss converge? I use transformer locally (currently training). The training loss has now dropped steadily to 0.739610, and the valid loss has dropped steadily to 0.698006. I observed the previous training set and validation set. Loss, the loss on the validation set is basically lower than the loss on the training set. Although it decreases slowly, the direction of convergence seems to be very good. I have not used the evaluation method to check the AP of each verification.", "2510485": "Transformers might not be feasible for such long sequences. To even get a reasonable sequence length for training you'd have to slice step the sequences down by a couple factors, but then you'd lose precision. LSTM is better in these scenarios. However, maybe if you really cut down the amount of steps, and make each step have features representing the steps around it that were sliced out, it might work if you also couple other methods. For example, say you take the training data, divide it into steps of 30 minutes, and feature engineer each step so that it represents the 15 minutes worth of time steps before and after it such at a center rolling average of period 300 5 sec steps. This would result in guesses every 30 min in the sequence. Then take the steps that were guessed as either onset or awake by the model, and run them through another model that takes the 300 5 second time steps(15min) before and after the guess to make a precise prediction of where the onset/awake happened. ", "2508419": "Does anyone have any ideas on how to train a transformer with one day as input (17280 steps) and the output as a Gaussian of the same size? Which structure to choose (encoder, decoder, both) ?" } }