{
  "id": 329454,
  "title": "loading model - AutoModel.from_pretrained",
  "url": "/competitions/AI4Code/discussion/329454",
  "author_name": "",
  "post_date": "2022-06-06T18:56:53.708070800Z",
  "votes": 1,
  "comment_count": 2,
  "views": 0,
  "content": "<p>I am did pretrain and now going to load the model from my pretrain.</p>\n<p>distill_bert = AutoModel.from_pretrained(\"./outputs/bert-small/checkpoint-3000/\")</p>\n<p>But I get this error, and when I try to train, it hangs. nothing else happens and I have to restart the jupyter notebook server.</p>\n<p>Any suggestions, thanks</p>\n<hr>\n<p>Some weights of the model checkpoint at ./outputs/bert-medium/ were not used when initializing BertModel: ['cls.predictions.transform.dense.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.bias', 'cls.predictions.decoder.weight', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.LayerNorm.bias', 'cls.predictions.decoder.bias']</p>\n<ul>\n<li>This IS expected if you are initializing BertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).</li>\n<li>This IS NOT expected if you are initializing BertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).<br>\nSome weights of BertModel were not initialized from the model checkpoint at ./outputs/bert-medium/ and are newly initialized: ['bert.pooler.dense.bias', 'bert.pooler.dense.weight']<br>\nYou should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.</li>\n</ul>",
  "messages": [
    {
      "id": "1813377",
      "postDate": "06/06/2022 18:56:53",
      "content": "<p>I am did pretrain and now going to load the model from my pretrain.</p>\n<p>distill_bert = AutoModel.from_pretrained(\"./outputs/bert-small/checkpoint-3000/\")</p>\n<p>But I get this error, and when I try to train, it hangs. nothing else happens and I have to restart the jupyter notebook server.</p>\n<p>Any suggestions, thanks</p>\n<hr>\n<p>Some weights of the model checkpoint at ./outputs/bert-medium/ were not used when initializing BertModel: ['cls.predictions.transform.dense.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.bias', 'cls.predictions.decoder.weight', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.LayerNorm.bias', 'cls.predictions.decoder.bias']</p>\n<ul>\n<li>This IS expected if you are initializing BertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).</li>\n<li>This IS NOT expected if you are initializing BertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).<br>\nSome weights of BertModel were not initialized from the model checkpoint at ./outputs/bert-medium/ and are newly initialized: ['bert.pooler.dense.bias', 'bert.pooler.dense.weight']<br>\nYou should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.</li>\n</ul>",
      "rawMarkdown": "I am did pretrain and now going to load the model from my pretrain.\n\ndistill_bert = AutoModel.from_pretrained(\"./outputs/bert-small/checkpoint-3000/\")\n\nBut I get this error, and when I try to train, it hangs. nothing else happens and I have to restart the jupyter notebook server.\n\nAny suggestions, thanks\n\n-------------------------------------------------------------------------------------------------------------------\nSome weights of the model checkpoint at ./outputs/bert-medium/ were not used when initializing BertModel: ['cls.predictions.transform.dense.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.bias', 'cls.predictions.decoder.weight', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.LayerNorm.bias', 'cls.predictions.decoder.bias']\n- This IS expected if you are initializing BertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n- This IS NOT expected if you are initializing BertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\nSome weights of BertModel were not initialized from the model checkpoint at ./outputs/bert-medium/ and are newly initialized: ['bert.pooler.dense.bias', 'bert.pooler.dense.weight']\nYou should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.",
      "votes": null
    },
    {
      "id": "1815407",
      "postDate": "06/09/2022 03:02:46",
      "content": "<p>Another method is to copy the model code from your (pre)training notebook to the next use, and then save/load just the model's weights.  That should keep it from getting confused.</p>",
      "rawMarkdown": "Another method is to copy the model code from your (pre)training notebook to the next use, and then save/load just the model's weights.  That should keep it from getting confused.",
      "votes": null
    },
    {
      "id": "1815536",
      "postDate": "06/09/2022 06:16:16",
      "content": "<p>This is not an error. It just said you need to keep in mind that the model class for fine-tuning is not for the task at pretraining so the pretraining-task-specific layer won't be used, and the fine-tuning-task-specific layer is newly added.</p>",
      "rawMarkdown": "This is not an error. It just said you need to keep in mind that the model class for fine-tuning is not for the task at pretraining so the pretraining-task-specific layer won't be used, and the fine-tuning-task-specific layer is newly added.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1815407,
      "author_name": "happycube",
      "author_url": "",
      "post_date": "06/09/2022 03:02:46",
      "content": "<p>Another method is to copy the model code from your (pre)training notebook to the next use, and then save/load just the model's weights.  That should keep it from getting confused.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1815536,
      "author_name": "yufuin",
      "author_url": "",
      "post_date": "06/09/2022 06:16:16",
      "content": "<p>This is not an error. It just said you need to keep in mind that the model class for fine-tuning is not for the task at pretraining so the pretraining-task-specific layer won't be used, and the fine-tuning-task-specific layer is newly added.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1813377": "I am did pretrain and now going to load the model from my pretrain.\n\ndistill_bert = AutoModel.from_pretrained(\"./outputs/bert-small/checkpoint-3000/\")\n\nBut I get this error, and when I try to train, it hangs. nothing else happens and I have to restart the jupyter notebook server.\n\nAny suggestions, thanks\n\n-------------------------------------------------------------------------------------------------------------------\nSome weights of the model checkpoint at ./outputs/bert-medium/ were not used when initializing BertModel: ['cls.predictions.transform.dense.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.bias', 'cls.predictions.decoder.weight', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.LayerNorm.bias', 'cls.predictions.decoder.bias']\n- This IS expected if you are initializing BertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n- This IS NOT expected if you are initializing BertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\nSome weights of BertModel were not initialized from the model checkpoint at ./outputs/bert-medium/ and are newly initialized: ['bert.pooler.dense.bias', 'bert.pooler.dense.weight']\nYou should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.",
    "1815407": "Another method is to copy the model code from your (pre)training notebook to the next use, and then save/load just the model's weights.  That should keep it from getting confused.",
    "1815536": "This is not an error. It just said you need to keep in mind that the model class for fine-tuning is not for the task at pretraining so the pretraining-task-specific layer won't be used, and the fine-tuning-task-specific layer is newly added."
  },
  "source": "meta"
}