I noticed that CodeBert's returns 2 outputs. The first one of dimension [1, len(input), 768] and the second [1, 768]. What does each one represent?
", "messages": [ { "id": "1858273", "postDate": "07/16/2022 19:23:26", "content": "I noticed that CodeBert's returns 2 outputs. The first one of dimension [1, len(input), 768] and the second [1, 768]. What does each one represent?
", "rawMarkdown": "I noticed that CodeBert's returns 2 outputs. The first one of dimension [1, len(input), 768] and the second [1, 768]. What does each one represent?", "votes": null }, { "id": "1863330", "postDate": "07/20/2022 09:49:44", "content": "(I guess that..)
\nThe first output is the encoded vector of the input sequence and the second output is the vector representation of the [CLS] token. The second vector is often used for classification tasks.
Probably a little late. I am assuming you are just using a base model with no specific head on top of the model from hugging face. The first output is called the last hidden state output and contains the hidden states of all your input ids as each input id will have a hidden state output associated with it.
\nIn this case it is given with shape (batch_size, len(input), 768). The 768 representing the amount of hidden units (3rd dimension) in the model. If you have the [CLS] token in your input you will access it like so \"transformer(input_ids, attention_mask)[0][:,0,:]\". Where the first [0] indexes to the last hidden state output then the [:,0,:] collects all the [CLS] token embeddings from the batch and from each hidden unit. The idea being that the first [CLS] token and its hidden units basically hold the overall sentiment for the given input. Both BERT and Roberta were pre-trained like that so the model will know that the [CLS] token is used for classification and is used for the majoirty of the notebooks in the tutorials here as you can train your own head on it to collect your final result.
\nThe second output is called the pooler output / layer. From huggingface's desciption it uses the same output as the [CLS] token, so with shape (batch_size, 768). However it is additionally processed through a linear layer with a tanh activation function. The weights set on it were trained from the next sentence prediction classification objective during pretraining. See model outputs on huggingface.
\nHowever, these outputs are based on the way BERT and Roberta were pretrained and the way you aggregate your outputs can change your results. If you have enough training data you don't have to just use the [CLS] token and you can start training for / utilizing other representations to help your final classification. Some ways are to try enabling (output_hidden_states=True) to get all hidden states at each layer and aggregating those as well or using a mean/max pool on your final output with your [CLS] token.
\nFor more information and ideas on how to use your model outputs please read Utilizing Transformer Representations Efficiently. Although how succesfully or good these options are, in my opinon they are more for pushing for gains and time could be spent elsewhere such as ensembling / hyops / data transforms, but might be worth trying as mean/max pooling have won competitions before.
", "rawMarkdown": "Probably a little late. I am assuming you are just using a base model with no specific head on top of the model from hugging face. The first output is called the last hidden state output and contains the hidden states of all your input ids as each input id will have a hidden state output associated with it.\n\nIn this case it is given with shape (batch_size, len(input), 768). The 768 representing the amount of hidden units (3rd dimension) in the model. If you have the [CLS] token in your input you will access it like so \"transformer(input_ids, attention_mask)[0][:,0,:]\". Where the first [0] indexes to the last hidden state output then the [:,0,:] collects all the [CLS] token embeddings from the batch and from each hidden unit. The idea being that the first [CLS] token and its hidden units basically hold the overall sentiment for the given input. Both BERT and Roberta were pre-trained like that so the model will know that the [CLS] token is used for classification and is used for the majoirty of the notebooks in the tutorials here as you can train your own head on it to collect your final result.\n\nThe second output is called the pooler output / layer. From huggingface's desciption it uses the same output as the [CLS] token, so with shape (batch_size, 768). However it is additionally processed through a linear layer with a tanh activation function. The weights set on it were trained from the next sentence prediction classification objective during pretraining. See [model outputs on huggingface](https://huggingface.co/docs/transformers/main_classes/output#model-outputs).\n\nHowever, these outputs are based on the way BERT and Roberta were pretrained and the way you aggregate your outputs can change your results. If you have enough training data you don't have to just use the [CLS] token and you can start training for / utilizing other representations to help your final classification. Some ways are to try enabling (output_hidden_states=True) to get all hidden states at each layer and aggregating those as well or using a mean/max pool on your final output with your [CLS] token.\n\nFor more information and ideas on how to use your model outputs please read [Utilizing Transformer Representations Efficiently](https://www.kaggle.com/code/rhtsingh/utilizing-transformer-representations-efficiently/notebook). Although how succesfully or good these options are, in my opinon they are more for pushing for gains and time could be spent elsewhere such as ensembling / hyops / data transforms, but might be worth trying as mean/max pooling have won competitions before.", "votes": null } ], "comments": [ { "id": 1863330, "author_name": "thedevastator", "author_url": "", "post_date": "07/20/2022 09:49:44", "content": "(I guess that..)
\nThe first output is the encoded vector of the input sequence and the second output is the vector representation of the [CLS] token. The second vector is often used for classification tasks.
Probably a little late. I am assuming you are just using a base model with no specific head on top of the model from hugging face. The first output is called the last hidden state output and contains the hidden states of all your input ids as each input id will have a hidden state output associated with it.
\nIn this case it is given with shape (batch_size, len(input), 768). The 768 representing the amount of hidden units (3rd dimension) in the model. If you have the [CLS] token in your input you will access it like so \"transformer(input_ids, attention_mask)[0][:,0,:]\". Where the first [0] indexes to the last hidden state output then the [:,0,:] collects all the [CLS] token embeddings from the batch and from each hidden unit. The idea being that the first [CLS] token and its hidden units basically hold the overall sentiment for the given input. Both BERT and Roberta were pre-trained like that so the model will know that the [CLS] token is used for classification and is used for the majoirty of the notebooks in the tutorials here as you can train your own head on it to collect your final result.
\nThe second output is called the pooler output / layer. From huggingface's desciption it uses the same output as the [CLS] token, so with shape (batch_size, 768). However it is additionally processed through a linear layer with a tanh activation function. The weights set on it were trained from the next sentence prediction classification objective during pretraining. See model outputs on huggingface.
\nHowever, these outputs are based on the way BERT and Roberta were pretrained and the way you aggregate your outputs can change your results. If you have enough training data you don't have to just use the [CLS] token and you can start training for / utilizing other representations to help your final classification. Some ways are to try enabling (output_hidden_states=True) to get all hidden states at each layer and aggregating those as well or using a mean/max pool on your final output with your [CLS] token.
\nFor more information and ideas on how to use your model outputs please read Utilizing Transformer Representations Efficiently. Although how succesfully or good these options are, in my opinon they are more for pushing for gains and time could be spent elsewhere such as ensembling / hyops / data transforms, but might be worth trying as mean/max pooling have won competitions before.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1858273": "I noticed that CodeBert's returns 2 outputs. The first one of dimension [1, len(input), 768] and the second [1, 768]. What does each one represent?", "1863330": "(I guess that..)\nThe first output is the encoded vector of the input sequence and the second output is the vector representation of the [CLS] token. The second vector is often used for classification tasks.", "1870727": "Probably a little late. I am assuming you are just using a base model with no specific head on top of the model from hugging face. The first output is called the last hidden state output and contains the hidden states of all your input ids as each input id will have a hidden state output associated with it.\n\nIn this case it is given with shape (batch_size, len(input), 768). The 768 representing the amount of hidden units (3rd dimension) in the model. If you have the [CLS] token in your input you will access it like so \"transformer(input_ids, attention_mask)[0][:,0,:]\". Where the first [0] indexes to the last hidden state output then the [:,0,:] collects all the [CLS] token embeddings from the batch and from each hidden unit. The idea being that the first [CLS] token and its hidden units basically hold the overall sentiment for the given input. Both BERT and Roberta were pre-trained like that so the model will know that the [CLS] token is used for classification and is used for the majoirty of the notebooks in the tutorials here as you can train your own head on it to collect your final result.\n\nThe second output is called the pooler output / layer. From huggingface's desciption it uses the same output as the [CLS] token, so with shape (batch_size, 768). However it is additionally processed through a linear layer with a tanh activation function. The weights set on it were trained from the next sentence prediction classification objective during pretraining. See [model outputs on huggingface](https://huggingface.co/docs/transformers/main_classes/output#model-outputs).\n\nHowever, these outputs are based on the way BERT and Roberta were pretrained and the way you aggregate your outputs can change your results. If you have enough training data you don't have to just use the [CLS] token and you can start training for / utilizing other representations to help your final classification. Some ways are to try enabling (output_hidden_states=True) to get all hidden states at each layer and aggregating those as well or using a mean/max pool on your final output with your [CLS] token.\n\nFor more information and ideas on how to use your model outputs please read [Utilizing Transformer Representations Efficiently](https://www.kaggle.com/code/rhtsingh/utilizing-transformer-representations-efficiently/notebook). Although how succesfully or good these options are, in my opinon they are more for pushing for gains and time could be spent elsewhere such as ensembling / hyops / data transforms, but might be worth trying as mean/max pooling have won competitions before." }, "source": "meta" }