Thanks for having this interesting competition. At the first place, I did not intended to dive such deep in this competition, but after solving WAF and inference thanks for moo's hint, I was in this huge rabbit hole. However I can learn a lot of new things in ML and security.
\nBelow is my findings crop2. As most of people noticed, the model's behavior seems completely random, because most of predictions are highly biased on certain classes (e.g. 1, 5). At the first time, I can't believe the model learns meaningful content. However, I later found out the model behaves quite accurate when input is in a certain constraint. So I believe at least distillation head of the model (head_dist) learns something meaningful.
Research on model's last layers embeddings (i.e. output of last layer of model.blocks) indicates the model maps embeddings of un-constrained input into the extremely narrow subspace. Besides, the center point of embedding (z0) and learned linear weight (w1, w2) are almost un-correlated (i.e. similarity is very low). So I guess the attack is somehow relating to losing diversity in embedding space if we input certain pixel values.
Notebook: https://www.kaggle.com/code/tatamikenn/defcon30-observations-on-crop2
", "messages": [ { "id": 1935286, "postDate": "2022-09-12T02:19:08.067Z", "content": "Thanks for having this interesting competition. At the first place, I did not intended to dive such deep in this competition, but after solving WAF and inference thanks for moo's hint, I was in this huge rabbit hole. However I can learn a lot of new things in ML and security.
\nBelow is my findings crop2. As most of people noticed, the model's behavior seems completely random, because most of predictions are highly biased on certain classes (e.g. 1, 5). At the first time, I can't believe the model learns meaningful content. However, I later found out the model behaves quite accurate when input is in a certain constraint. So I believe at least distillation head of the model (head_dist) learns something meaningful.
Research on model's last layers embeddings (i.e. output of last layer of model.blocks) indicates the model maps embeddings of un-constrained input into the extremely narrow subspace. Besides, the center point of embedding (z0) and learned linear weight (w1, w2) are almost un-correlated (i.e. similarity is very low). So I guess the attack is somehow relating to losing diversity in embedding space if we input certain pixel values.
Notebook: https://www.kaggle.com/code/tatamikenn/defcon30-observations-on-crop2
", "rawMarkdown": "Thanks for having this interesting competition. At the first place, I did not intended to dive such deep in this competition, but after solving WAF and inference thanks for moo's hint, I was in this huge rabbit hole. However I can learn a lot of new things in ML and security.\n\n# Findings\n\nBelow is my findings crop2. As most of people noticed, the model's behavior seems completely random, because most of predictions are highly biased on certain classes (e.g. 1, 5). At the first time, I can't believe the model learns meaningful content. However, I later found out the model behaves quite accurate when input is in a certain constraint. So I believe at least distillation head of the model (`head_dist`) learns something meaningful.\n\nResearch on model's last layers embeddings (i.e. output of last layer of `model.blocks`) indicates the model maps embeddings of un-constrained input into the extremely narrow subspace. Besides, the center point of embedding (z0) and learned linear weight (w1, w2) are almost un-correlated (i.e. similarity is very low). So I guess the attack is somehow relating to losing diversity in embedding space if we input certain pixel values.\n\n1. Embedding of the last layer of blocks is almost the same if we input random color circles/boxes (similarity of each class centers are >0.95)\n2. In certain range of input, the model can classify highly accurate output. E.g. For the input is constrained in pixel range 192-250, target color (255, 128, 128), the model‘s F1=0.998. So I guess the model causes something like mode corruption when input have color of certain (e.g. lower) pixel values.\n3. at least distillation head (head_dist) learns something meaningful, because when constrained input, the similarity of class center (zc) and linear weight (w2) have high correlations.\n\nNotebook: https://www.kaggle.com/code/tatamikenn/defcon30-observations-on-crop2", "votes": 4 }, { "id": 1947217, "postDate": "2022-09-20T10:48:33.227Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 1947217, "author_name": "", "author_url": "", "post_date": "2022-09-20T10:48:33.227000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1935286": "Thanks for having this interesting competition. At the first place, I did not intended to dive such deep in this competition, but after solving WAF and inference thanks for moo's hint, I was in this huge rabbit hole. However I can learn a lot of new things in ML and security.\n\n# Findings\n\nBelow is my findings crop2. As most of people noticed, the model's behavior seems completely random, because most of predictions are highly biased on certain classes (e.g. 1, 5). At the first time, I can't believe the model learns meaningful content. However, I later found out the model behaves quite accurate when input is in a certain constraint. So I believe at least distillation head of the model (`head_dist`) learns something meaningful.\n\nResearch on model's last layers embeddings (i.e. output of last layer of `model.blocks`) indicates the model maps embeddings of un-constrained input into the extremely narrow subspace. Besides, the center point of embedding (z0) and learned linear weight (w1, w2) are almost un-correlated (i.e. similarity is very low). So I guess the attack is somehow relating to losing diversity in embedding space if we input certain pixel values.\n\n1. Embedding of the last layer of blocks is almost the same if we input random color circles/boxes (similarity of each class centers are >0.95)\n2. In certain range of input, the model can classify highly accurate output. E.g. For the input is constrained in pixel range 192-250, target color (255, 128, 128), the model‘s F1=0.998. So I guess the model causes something like mode corruption when input have color of certain (e.g. lower) pixel values.\n3. at least distillation head (head_dist) learns something meaningful, because when constrained input, the similarity of class center (zc) and linear weight (w2) have high correlations.\n\nNotebook: https://www.kaggle.com/code/tatamikenn/defcon30-observations-on-crop2", "1947217": "" } }