{
  "id": 351864,
  "title": "Please explain idea behind Token challenge",
  "url": "/competitions/ai-village-ctf/discussion/351864",
  "author_name": "",
  "post_date": "2022-09-12T07:02:28.704375100Z",
  "votes": 3,
  "comment_count": 1,
  "views": 0,
  "content": "<p>We are given a dataset, which is used to fit tokenizer. But from the solution it seems that the dataset is not used to fit the model itself. Thus we are able to make the trick with frequency of tokens.<br>\nBut is it a real situation when a static model used together with a trained tokenizer? What is the point?</p>",
  "messages": [
    {
      "id": "1935519",
      "postDate": "09/12/2022 07:02:28",
      "content": "<p>We are given a dataset, which is used to fit tokenizer. But from the solution it seems that the dataset is not used to fit the model itself. Thus we are able to make the trick with frequency of tokens.<br>\nBut is it a real situation when a static model used together with a trained tokenizer? What is the point?</p>",
      "rawMarkdown": "We are given a dataset, which is used to fit tokenizer. But from the solution it seems that the dataset is not used to fit the model itself. Thus we are able to make the trick with frequency of tokens.\nBut is it a real situation when a static model used together with a trained tokenizer? What is the point?",
      "votes": null
    },
    {
      "id": "1936674",
      "postDate": "09/13/2022 01:16:46",
      "content": "<p>I think it demonstrates a \"man in the middle\" attack. Let's say a certain company trains a model to accomplish a task like grading essays at a college. Now let's say the college is about to download the model to their university's campus. Models are saved as model weights and tokenizer separately. The student can intercept the transfer, alter the tokenizer, and continue the transfer with the modified tokenizer. Then when the university uses the model to grade that students' papers, every paper receives a grade of 100% because the student can alter the tokenizer in a way to favor key words that he/she inserts into their essays.</p>\n<p>Or after the model is downloaded correctly, the student can hack into the college's computers and alter the tokenizer. Either way this competition demonstrates that altering a tokenizer can affect the outcome of a transformers predictions. Whereas it would be very difficult to modify a model's weights to have it do what you wanted.</p>",
      "rawMarkdown": "I think it demonstrates a \"man in the middle\" attack. Let's say a certain company trains a model to accomplish a task like grading essays at a college. Now let's say the college is about to download the model to their university's campus. Models are saved as model weights and tokenizer separately. The student can intercept the transfer, alter the tokenizer, and continue the transfer with the modified tokenizer. Then when the university uses the model to grade that students' papers, every paper receives a grade of 100% because the student can alter the tokenizer in a way to favor key words that he/she inserts into their essays.\n\nOr after the model is downloaded correctly, the student can hack into the college's computers and alter the tokenizer. Either way this competition demonstrates that altering a tokenizer can affect the outcome of a transformers predictions. Whereas it would be very difficult to modify a model's weights to have it do what you wanted.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1936674,
      "author_name": "cdeotte",
      "author_url": "",
      "post_date": "09/13/2022 01:16:46",
      "content": "<p>I think it demonstrates a \"man in the middle\" attack. Let's say a certain company trains a model to accomplish a task like grading essays at a college. Now let's say the college is about to download the model to their university's campus. Models are saved as model weights and tokenizer separately. The student can intercept the transfer, alter the tokenizer, and continue the transfer with the modified tokenizer. Then when the university uses the model to grade that students' papers, every paper receives a grade of 100% because the student can alter the tokenizer in a way to favor key words that he/she inserts into their essays.</p>\n<p>Or after the model is downloaded correctly, the student can hack into the college's computers and alter the tokenizer. Either way this competition demonstrates that altering a tokenizer can affect the outcome of a transformers predictions. Whereas it would be very difficult to modify a model's weights to have it do what you wanted.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1935519": "We are given a dataset, which is used to fit tokenizer. But from the solution it seems that the dataset is not used to fit the model itself. Thus we are able to make the trick with frequency of tokens.\nBut is it a real situation when a static model used together with a trained tokenizer? What is the point?",
    "1936674": "I think it demonstrates a \"man in the middle\" attack. Let's say a certain company trains a model to accomplish a task like grading essays at a college. Now let's say the college is about to download the model to their university's campus. Models are saved as model weights and tokenizer separately. The student can intercept the transfer, alter the tokenizer, and continue the transfer with the modified tokenizer. Then when the university uses the model to grade that students' papers, every paper receives a grade of 100% because the student can alter the tokenizer in a way to favor key words that he/she inserts into their essays.\n\nOr after the model is downloaded correctly, the student can hack into the college's computers and alter the tokenizer. Either way this competition demonstrates that altering a tokenizer can affect the outcome of a transformers predictions. Whereas it would be very difficult to modify a model's weights to have it do what you wanted."
  },
  "source": "meta"
}