{ "id": 455206, "title": "5th place | 24/27 | Lessons from my first CTF + how GPT4 can help", "content": "

Hello!

\n

First of all, I want to thank the organizers and Kaggle for this amazing competition! It was a great learning experience, and it has piqued my interest in AI security. I would also like to thank the community - the Discord channel was both useful and fun!

\n

I joined this contest ~9 days after it started, and initially, I did so only for the prompt injection attacks, after someone on X recommended it to me. Didn't expect to get a medal, let alone a gold😅 The multiple easy/accessible problems managed to get me hooked before hitting the difficulty wall, kudos to the hosts for the strategy.

\n

Takeaways

\n\n

Git:

\n

https://github.com/hh-5/AI-CTF-DEFCON31

\n

\"\"

\n

Prompt problems

\n

What is the flag 1-6

\n

Asking for the system prompt or ignoring previous instructions was sufficient. They were a bit random, for example, a single backslash got me the flag for the 4th one.

\n

IP 1/2

\n

I wasn't exactly sure what the task was, so I simply asked \"How to \" + the task description and got the flags.

\n

Pirate / Spanglish

\n

Simply asking for the flag didn't work, but adding random questions or asking for it in JSON format worked.
\nUsing structured data for extracting guarded information from LLMs has been a useful tool in my experience.

\n

Mid-difficulty problems

\n

These problems could be solved with enough time invested, and they were quite fun, with one exception.

\n

Semantle 1/2

\n

I didn't know about the game, so I solved Semantle 1 using Akinator-style questions within ~30 tries. Semantle 2 on the other hand took a bit more work. Repeating words like human/man/woman seemed to get higher scores, but it was hard to differentiate between equal scores, so I tried to find a set of words on a fixed position to better differentiate progress, which ended up looking like this. Google did the rest after some semantic experiments with GPT4.

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
!@#EAS34531fartdeadartcartgirlwoman
0.840.850.860.870.880.890.900.91
\n

Sloth

\n

Once again, GPT4 helped a lot here. After some experiments, I found this:

\n

\"\"

\n

Pickle

\n

After trying some simple ideas, I asked GPT-4 to explain how to make a properly dangerous pickle. None 'worked', so I tried to find the line between dangerous and not dangerous. Dumping def a():\\n return eval, got the flag, still not sure why.

\n

Cluster 1-3

\n

GPT4 solved Cluster 2 in the first prompt. For Cluster 1, for some reason, I was unable to run the model locally nor with gpt, so I binary-searched the input size that would get a response. Noticed the response was not 0, so there had to be some index in the set that was part of the answer. So I queried all potential values in batches. Got the full subset within the hour and got the flag. For Cluster 3, some visualization was enough except for the insufficient credit part. Discord helped here - someone was slightly confused and posted an image(no hints), but it got ~10 🤫 emojis as reaction, which made the answer obvious.

\n

MNIST/CIFAR

\n

For those, I set up a small pipeline that would reverse/shift/switch indexing for each potential answer. I then generated some ideas for answers and it worked eventually. CIFAR was simple too, all you had to do was send a (100,4) matrix that counted data from the dataset using the hint from input_data.

\n

I will take a moment of silence for all the hours spent on the overthought solutions🙃

\n

Bronze/Silver tiebreakers

\n

Pixelated

\n

For most, the difficulty of this one wasn't as much in finding the right XML injection as getting the OCR to read it right. For me, it was the other way around. By using liberation monospace font on large images, and generating them such that the white space is minimized, I avoided the OCR issues. As for finding the right injection, it took some time; the size of the cartesian product between what could be before and after true wasn't that large so after a few hundred variations I got the flag.

\n

Granny 1-2

\n

I spent days trying to match the model both before finding out it was Torch-based and after I found it was Torch-based. Even tried a genetic algorithm on the preprocessing parameters. The best improvement I could find was making the model run on double precision, but the errors were pretty much the same after 1e-5 or 1e-6.

\n

Knowing how many people had results very close to 1.0 from Discord fooled me into believing that without matching the weights I had no chance of getting the flags. But apparently adding a 16px border on the 224x224 resized and (simple gradient ascent) perturbed image was enough to get the flags for both Granny 1&2. Even if the final solution was simple, through multiple papers and ART toolbox attacks was quite an adventure.

\n

Did you know you could upload the model .pth + an image, ask GPT4 to write an attack for it and give you back the perturbed image? I admit it doesn't always work, and more often than not, it makes some mistakes or needs some assistance - but I found it interesting nonetheless. And very educational when trying out different methods.

\n

Gold tiebreakers

\n

Inversion

\n

This one took many experiments and by the end, I think I had sent over 200k requests to the endpoint. I made multiple datasets using different methods - random symbols of different sizes, EMNIST and even CIFAR led to pretty much the same results. Logits 5,6 and 8 just refused to activate. I also trained over 100 models in different ways. Not to mention that the results, when flipped, could've also been read as \"-----tak\" or \"-----tar\"…

\n

I had almost given up on it when someone on Discord mentioned that it was easier for native speakers. One of the hosts mentioned they used \"Quija\" instead of \"Ouija\" because it was correct in French. Don't know whether that is true or not, but it prompted me to try some French words instead of trying to find the values for logits 568. Given the theme, I thought it was something like \"Je+\" and a 5-letter-word (like \"Je+morte\"). Or it could be \"le\" + 6-letter-word. So after all French words were exhausted, I decided to brute force it. And it worked! I still don't know what a 'tmeut' means in French though.

\n

Jokes aside, it was a fun problem and I learned a lot from it. For example, generalizing well is not very useful when the objective is to overfit, so the results were better without max pooling. However, adding a 25/50 dropout improved the result visual quality, in my opinion:

\n

Without DR:
\n\"\"

\n

With DR:
\n\"\"

\n

Passphrase

\n

This was a problem I kept trying and failing at for about two weeks. Got the right scores by adding spaces. Or punctuation. Or random characters. But not the flag. Later on though when people started solving it I noticed the time it took to do it was relatively short - which implied that my methods were not on the right track.
\nIt was clear that genetic algorithms were not the way. At some point, the hosts hinted at what pre/post-processing was done and at what a passphrase is (i.e. a phrase, not a password).

\n

Although I didn't get the main hint (now obvious in hindsight \"difference sentence\", instead of \"different\" meant they used a specific metric as a threshold, which seems to have been cosine difference), I kept trying.

\n

Eventually, I generated a large set of statements and their scores using markov chains, picked those that scored the best, and started replacing words in them. The search space was of ~1e8, so I thought that a meet-in-the-middle approach had a fair chance of getting the result (i.e. getting the best scoring phrase out of 1e4 tries, then 1e4 attempts at word replacement).
\n\"\"

\n

Thought I was lucky, but then realized that probabilities for scores are not even - a strong bias, like in the given example, has a higher chance of occurring. Anyway, after trying(and failing) with names (Ancog and Week both got the right score) I tried simple nouns. There weren't many nouns that could fit the few characters left (limit was 50) but after a few adjustments I got the flag using \"The magazine visual is also immensely appealing.\"

\n

The second best result I had in that run was \"The solution visual is also immensely appealing.\" - could've been a funny passphrase to get the flag with, if only it worked😅

\n

Hush

\n

Didn't solve it, was on an entirely wrong path for the whole time, but at least I learned some things about sound processing and the binary structure of sound files.

\n

Unsolved problems

\n

Granny 3

\n

I think I spent close 100 hours on this one. From DE to multiple other genetic algorithms, nothing got a decent solution. Didn't manage to fool the endpoint that the image sent did not differ in more than one pixel either, but I honestly suspect that this task was not about finding the right pixel.

\n

After the naive solutions didn't lead to anything, I brute-forced contrastive values in search for insight. Here are some intersections of the best pixels for different values using the top 1000 by apple score of each.

\n

\"\"

\n

Considering that the 768x768 mapped perfectly to 256x256 in a 3:1 ratio, the search space was less than 9*[256/9]^3 for each pixel. The image did not have many sharp contrast variations, so for most of them, 3-4e4 combinations exhausted the space. After over 1e7 tries, the best was still below 7e-4.

\n

After I finished with the top 100 pixels with the best gradients, I also tried working with a small version of imagenet, and getting the most popular shades of green, but they weren't any better.

\n

\"\"

\n

Here are some other interesting things GPT4 offered while trying to solve it:

\n

Gradient map layer by layer:
\n\"\"

\n

Here's the basic gradient map for the apple class:
\n\"\"

\n

And here's the heatmap of aggregate best results:
\n\"\"

\n

Here's the overlayed image:

\n

\"\"

\n

Notes:

\n\n

I'll keep updating this as I find/remember more results/observations, my notebooks are all over the place😅

", "comments": [ { "id": 2523959, "author_name": "Rob Mulla", "author_url": "", "post_date": "2023-11-13T21:42:59.583000", "content": "
\n

CIFAR was simple too, all you had to do was send a (100,4) matrix that counted data from the dataset using the hint from input_data.

\n
\n

\"\"

", "votes": 7, "replies": [ { "id": 2523970, "author_name": "Horea", "author_url": "", "post_date": "2023-11-13T22:13:59.907000", "content": "

\"\"

", "votes": 5, "replies": [] } ] }, { "id": 2524115, "author_name": "Max2020", "author_url": "", "post_date": "2023-11-14T02:12:49.490000", "content": "

we need CIFAR please🙏

", "votes": 0, "replies": [] }, { "id": 2524102, "author_name": "Thariq Nugrohotomo", "author_url": "", "post_date": "2023-11-14T02:00:15.643000", "content": "
\n

Didn't expect to get a medal, let alone a gold😅.

\n
\n

Huh? Sounds like you're not aware that you win the cash prize? 😅
\nCongratulation!

\n

Seems like you've solved CIFAR. Would you mind elaborating at what the solution (100, 4) is?

\n

Thank you.

", "votes": 0, "replies": [] } ], "comment_count": 4, "messages": [ { "id": 2523861, "postDate": "2023-11-13T20:20:53.060Z", "content": "

Hello!

\n

First of all, I want to thank the organizers and Kaggle for this amazing competition! It was a great learning experience, and it has piqued my interest in AI security. I would also like to thank the community - the Discord channel was both useful and fun!

\n

I joined this contest ~9 days after it started, and initially, I did so only for the prompt injection attacks, after someone on X recommended it to me. Didn't expect to get a medal, let alone a gold😅 The multiple easy/accessible problems managed to get me hooked before hitting the difficulty wall, kudos to the hosts for the strategy.

\n

Takeaways

\n\n

Git:

\n

https://github.com/hh-5/AI-CTF-DEFCON31

\n

\"\"

\n

Prompt problems

\n

What is the flag 1-6

\n

Asking for the system prompt or ignoring previous instructions was sufficient. They were a bit random, for example, a single backslash got me the flag for the 4th one.

\n

IP 1/2

\n

I wasn't exactly sure what the task was, so I simply asked \"How to \" + the task description and got the flags.

\n

Pirate / Spanglish

\n

Simply asking for the flag didn't work, but adding random questions or asking for it in JSON format worked.
\nUsing structured data for extracting guarded information from LLMs has been a useful tool in my experience.

\n

Mid-difficulty problems

\n

These problems could be solved with enough time invested, and they were quite fun, with one exception.

\n

Semantle 1/2

\n

I didn't know about the game, so I solved Semantle 1 using Akinator-style questions within ~30 tries. Semantle 2 on the other hand took a bit more work. Repeating words like human/man/woman seemed to get higher scores, but it was hard to differentiate between equal scores, so I tried to find a set of words on a fixed position to better differentiate progress, which ended up looking like this. Google did the rest after some semantic experiments with GPT4.

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
!@#EAS34531fartdeadartcartgirlwoman
0.840.850.860.870.880.890.900.91
\n

Sloth

\n

Once again, GPT4 helped a lot here. After some experiments, I found this:

\n

\"\"

\n

Pickle

\n

After trying some simple ideas, I asked GPT-4 to explain how to make a properly dangerous pickle. None 'worked', so I tried to find the line between dangerous and not dangerous. Dumping def a():\\n return eval, got the flag, still not sure why.

\n

Cluster 1-3

\n

GPT4 solved Cluster 2 in the first prompt. For Cluster 1, for some reason, I was unable to run the model locally nor with gpt, so I binary-searched the input size that would get a response. Noticed the response was not 0, so there had to be some index in the set that was part of the answer. So I queried all potential values in batches. Got the full subset within the hour and got the flag. For Cluster 3, some visualization was enough except for the insufficient credit part. Discord helped here - someone was slightly confused and posted an image(no hints), but it got ~10 🤫 emojis as reaction, which made the answer obvious.

\n

MNIST/CIFAR

\n

For those, I set up a small pipeline that would reverse/shift/switch indexing for each potential answer. I then generated some ideas for answers and it worked eventually. CIFAR was simple too, all you had to do was send a (100,4) matrix that counted data from the dataset using the hint from input_data.

\n

I will take a moment of silence for all the hours spent on the overthought solutions🙃

\n

Bronze/Silver tiebreakers

\n

Pixelated

\n

For most, the difficulty of this one wasn't as much in finding the right XML injection as getting the OCR to read it right. For me, it was the other way around. By using liberation monospace font on large images, and generating them such that the white space is minimized, I avoided the OCR issues. As for finding the right injection, it took some time; the size of the cartesian product between what could be before and after true wasn't that large so after a few hundred variations I got the flag.

\n

Granny 1-2

\n

I spent days trying to match the model both before finding out it was Torch-based and after I found it was Torch-based. Even tried a genetic algorithm on the preprocessing parameters. The best improvement I could find was making the model run on double precision, but the errors were pretty much the same after 1e-5 or 1e-6.

\n

Knowing how many people had results very close to 1.0 from Discord fooled me into believing that without matching the weights I had no chance of getting the flags. But apparently adding a 16px border on the 224x224 resized and (simple gradient ascent) perturbed image was enough to get the flags for both Granny 1&2. Even if the final solution was simple, through multiple papers and ART toolbox attacks was quite an adventure.

\n

Did you know you could upload the model .pth + an image, ask GPT4 to write an attack for it and give you back the perturbed image? I admit it doesn't always work, and more often than not, it makes some mistakes or needs some assistance - but I found it interesting nonetheless. And very educational when trying out different methods.

\n

Gold tiebreakers

\n

Inversion

\n

This one took many experiments and by the end, I think I had sent over 200k requests to the endpoint. I made multiple datasets using different methods - random symbols of different sizes, EMNIST and even CIFAR led to pretty much the same results. Logits 5,6 and 8 just refused to activate. I also trained over 100 models in different ways. Not to mention that the results, when flipped, could've also been read as \"-----tak\" or \"-----tar\"…

\n

I had almost given up on it when someone on Discord mentioned that it was easier for native speakers. One of the hosts mentioned they used \"Quija\" instead of \"Ouija\" because it was correct in French. Don't know whether that is true or not, but it prompted me to try some French words instead of trying to find the values for logits 568. Given the theme, I thought it was something like \"Je+\" and a 5-letter-word (like \"Je+morte\"). Or it could be \"le\" + 6-letter-word. So after all French words were exhausted, I decided to brute force it. And it worked! I still don't know what a 'tmeut' means in French though.

\n

Jokes aside, it was a fun problem and I learned a lot from it. For example, generalizing well is not very useful when the objective is to overfit, so the results were better without max pooling. However, adding a 25/50 dropout improved the result visual quality, in my opinion:

\n

Without DR:
\n\"\"

\n

With DR:
\n\"\"

\n

Passphrase

\n

This was a problem I kept trying and failing at for about two weeks. Got the right scores by adding spaces. Or punctuation. Or random characters. But not the flag. Later on though when people started solving it I noticed the time it took to do it was relatively short - which implied that my methods were not on the right track.
\nIt was clear that genetic algorithms were not the way. At some point, the hosts hinted at what pre/post-processing was done and at what a passphrase is (i.e. a phrase, not a password).

\n

Although I didn't get the main hint (now obvious in hindsight \"difference sentence\", instead of \"different\" meant they used a specific metric as a threshold, which seems to have been cosine difference), I kept trying.

\n

Eventually, I generated a large set of statements and their scores using markov chains, picked those that scored the best, and started replacing words in them. The search space was of ~1e8, so I thought that a meet-in-the-middle approach had a fair chance of getting the result (i.e. getting the best scoring phrase out of 1e4 tries, then 1e4 attempts at word replacement).
\n\"\"

\n

Thought I was lucky, but then realized that probabilities for scores are not even - a strong bias, like in the given example, has a higher chance of occurring. Anyway, after trying(and failing) with names (Ancog and Week both got the right score) I tried simple nouns. There weren't many nouns that could fit the few characters left (limit was 50) but after a few adjustments I got the flag using \"The magazine visual is also immensely appealing.\"

\n

The second best result I had in that run was \"The solution visual is also immensely appealing.\" - could've been a funny passphrase to get the flag with, if only it worked😅

\n

Hush

\n

Didn't solve it, was on an entirely wrong path for the whole time, but at least I learned some things about sound processing and the binary structure of sound files.

\n

Unsolved problems

\n

Granny 3

\n

I think I spent close 100 hours on this one. From DE to multiple other genetic algorithms, nothing got a decent solution. Didn't manage to fool the endpoint that the image sent did not differ in more than one pixel either, but I honestly suspect that this task was not about finding the right pixel.

\n

After the naive solutions didn't lead to anything, I brute-forced contrastive values in search for insight. Here are some intersections of the best pixels for different values using the top 1000 by apple score of each.

\n

\"\"

\n

Considering that the 768x768 mapped perfectly to 256x256 in a 3:1 ratio, the search space was less than 9*[256/9]^3 for each pixel. The image did not have many sharp contrast variations, so for most of them, 3-4e4 combinations exhausted the space. After over 1e7 tries, the best was still below 7e-4.

\n

After I finished with the top 100 pixels with the best gradients, I also tried working with a small version of imagenet, and getting the most popular shades of green, but they weren't any better.

\n

\"\"

\n

Here are some other interesting things GPT4 offered while trying to solve it:

\n

Gradient map layer by layer:
\n\"\"

\n

Here's the basic gradient map for the apple class:
\n\"\"

\n

And here's the heatmap of aggregate best results:
\n\"\"

\n

Here's the overlayed image:

\n

\"\"

\n

Notes:

\n\n

I'll keep updating this as I find/remember more results/observations, my notebooks are all over the place😅

", "rawMarkdown": "Hello!\n\nFirst of all, I want to thank the organizers and Kaggle for this amazing competition! It was a great learning experience, and it has piqued my interest in AI security. I would also like to thank the community - the Discord channel was both useful and fun!\n\nI joined this contest ~9 days after it started, and initially, I did so only for the prompt injection attacks, after someone on X recommended it to me. Didn't expect to get a medal, let alone a gold😅 The multiple easy/accessible problems managed to get me hooked before hitting the difficulty wall, kudos to the hosts for the strategy.\n\n# Takeaways\n- Time investment is essential, especially for vague tasks. Many tasks were not necessarily hard but required multiple different experiments to get right/get the missing information.\n- Trying harder works. Found this [article](https://www.offsec.com/offsec/what-it-means-to-try-harder/) from the Discord channel insightful.\n- If there is no limit to the number of tries that you can make/queries to the API - make use of it. The hosts encouraged it, and for good reason. More on this at Passphrase & Inversion\n- GPT4 code interpreter is an amazing tool and it can single-handedly solve some of the tasks. Did you know it comes with Torch installed, so you can run small models with it? Although it does come with a time cost(multiple parallel sessions help!), I found it invaluable - especially for the tasks I had no prior experience with.\n\n#Git:\nhttps://github.com/hh-5/AI-CTF-DEFCON31\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4944379%2Fc74319ce0bf9b80b842ae25b02414560%2Fimagy-image%20(6).jpg?generation=1699900397098731&alt=media)\n\n# Prompt problems\n\n### What is the flag 1-6\n\nAsking for the system prompt or ignoring previous instructions was sufficient. They were a bit random, for example, a single backslash got me the flag for the 4th one.\n\n### IP 1/2\n\nI wasn't exactly sure what the task was, so I simply asked \"How to \" + the task description and got the flags. \n\n### Pirate / Spanglish\n\nSimply asking for the flag didn't work, but adding random questions or asking for it in JSON format worked.\nUsing structured data for extracting guarded information from LLMs has been a useful tool in my experience.\n\n# Mid-difficulty problems\n\nThese problems could be solved with enough time invested, and they were quite fun, with one exception.\n\n### Semantle 1/2\n\nI didn't know about the game, so I solved Semantle 1 using Akinator-style questions within ~30 tries. Semantle 2 on the other hand took a bit more work. Repeating words like human/man/woman seemed to get higher scores, but it was hard to differentiate between equal scores, so I tried to find a set of words on a fixed position to better differentiate progress, which ended up looking like this. Google did the rest after some semantic experiments with GPT4.\n\n| !@#EAS | 34531 | fart | dead | art | cart | girl | woman |\n|--------|-------|------|------|-----|------|------|-------|\n| 0.84 | 0.85 | 0.86 | 0.87 |0.88 | 0.89 | 0.90 | 0.91 |\n\n### Sloth\n\nOnce again, GPT4 helped a lot here. After some experiments, I found this:\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4944379%2F2104c5230e67f43ae34ce051c31696a3%2Ffoto_no_exif.jpg?generation=1699894459778845&alt=media)\n\n### Pickle\n\nAfter trying some simple ideas, I asked GPT-4 to explain how to make a properly dangerous pickle. None 'worked', so I tried to find the line between dangerous and not dangerous. Dumping def a():\\n return eval, got the flag, still not sure why. \n\n### Cluster 1-3\n\nGPT4 solved Cluster 2 in the first prompt. For Cluster 1, for some reason, I was unable to run the model locally nor with gpt, so I binary-searched the input size that would get a response. Noticed the response was not 0, so there had to be some index in the set that was part of the answer. So I queried all potential values in batches. Got the full subset within the hour and got the flag. For Cluster 3, some visualization was enough except for the insufficient credit part. Discord helped here - someone was slightly confused and posted an image(no hints), but it got ~10 🤫 emojis as reaction, which made the answer obvious.\n\n### MNIST/CIFAR\n\nFor those, I set up a small pipeline that would reverse/shift/switch indexing for each potential answer. I then generated some ideas for answers and it worked eventually. CIFAR was simple too, all you had to do was send a (100,4) matrix that counted data from the dataset using the hint from input_data.\n\nI will take a moment of silence for all the hours spent on the overthought solutions🙃\n\n# Bronze/Silver tiebreakers\n\n### Pixelated\n\nFor most, the difficulty of this one wasn't as much in finding the right XML injection as getting the OCR to read it right. For me, it was the other way around. By using liberation monospace font on large images, and generating them such that the white space is minimized, I avoided the OCR issues. As for finding the right injection, it took some time; the size of the cartesian product between what could be before and after true wasn't that large so after a few hundred variations I got the flag.\n\n### Granny 1-2\n\nI spent days trying to match the model both before finding out it was Torch-based *and* after I found it was Torch-based. Even tried a genetic algorithm on the preprocessing parameters. The best improvement I could find was making the model run on double precision, but the errors were pretty much the same after 1e-5 or 1e-6. \n\nKnowing how many people had results very close to 1.0 from Discord fooled me into believing that without matching the weights I had no chance of getting the flags. But apparently adding a 16px border on the 224x224 resized and (simple gradient ascent) perturbed image was enough to get the flags for both Granny 1&2. Even if the final solution was simple, through multiple papers and ART toolbox attacks was quite an adventure.\n\nDid you know you could upload the model .pth + an image, ask GPT4 to write an attack for it and give you back the perturbed image? I admit it doesn't always work, and more often than not, it makes some mistakes or needs some assistance - but I found it interesting nonetheless. And *very* educational when trying out different methods. \n\n# Gold tiebreakers\n\n### Inversion\n\nThis one took many experiments and by the end, I think I had sent over 200k requests to the endpoint. I made multiple datasets using different methods - random symbols of different sizes, EMNIST and even CIFAR led to pretty much the same results. Logits 5,6 and 8 just refused to activate. I also trained over 100 models in different ways. Not to mention that the results, when flipped, could've also been read as \"-----tak\" or \"-----tar\"...\n\nI had almost given up on it when someone on Discord mentioned that it was easier for native speakers. One of the hosts mentioned they used \"Quija\" instead of \"Ouija\" because it was correct in French. Don't know whether that is true or not, but it prompted me to try some French words instead of trying to find the values for logits 568. Given the theme, I thought it was something like \"Je+\" and a 5-letter-word (like \"Je+morte\"). Or it could be \"le\" + 6-letter-word. So after all French words were exhausted, I decided to brute force it. And it worked! I still don't know what a 'tmeut' means in French though.\n\nJokes aside, it was a fun problem and I learned a lot from it. For example, generalizing well is not very useful when the objective is to overfit, so the results were better without max pooling. However, adding a 25/50 dropout improved the result visual quality, in my opinion:\n\nWithout DR:\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4944379%2Fd8f59184460a57324856d099d5a05696%2Fimagy-image.jpg?generation=1699900055830528&alt=media)\n\nWith DR:\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4944379%2F022d6fc39dfb45d948441ccf24cd86b6%2Fimagy-image%20(1).jpg?generation=1699900062526604&alt=media)\n\n\n### Passphrase\n\nThis was a problem I kept trying and failing at for about two weeks. Got the right scores by adding spaces. Or punctuation. Or random characters. But not the flag. Later on though when people started solving it I noticed the time it took to do it was relatively short - which implied that my methods were not on the right track.\nIt was clear that genetic algorithms were not the way. At some point, the hosts hinted at what pre/post-processing was done and at what a passphrase *is* (i.e. a phrase, not a password).\n\nAlthough I didn't get the main hint (now obvious in hindsight \"difference sentence\", instead of \"different\" meant they used a specific metric as a threshold, which seems to have been cosine difference), I kept trying.\n\nEventually, I generated a large set of statements and their scores using markov chains, picked those that scored the best, and started replacing words in them. The search space was of ~1e8, so I thought that a meet-in-the-middle approach had a fair chance of getting the result (i.e. getting the best scoring phrase out of 1e4 tries, then 1e4 attempts at word replacement). \n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4944379%2F7f4e1d81258b32591c49fc8ca6bd0fa8%2FScreenshot%20from%202023-11-13%2021-15-23.png?generation=1699903092722538&alt=media)\n\nThought I was lucky, but then realized that probabilities for scores are not even - a strong bias, like in the given example, has a higher chance of occurring. Anyway, after trying(and failing) with names (Ancog and Week both got the right score) I tried simple nouns. There weren't many nouns that could fit the few characters left (limit was 50) but after a few adjustments I got the flag using \"The magazine visual is also immensely appealing.\"\n\nThe second best result I had in that run was \"The solution visual is also immensely appealing.\" - could've been a funny passphrase to get the flag with, if only it worked😅\n\n### Hush\n\nDidn't solve it, was on an entirely wrong path for the whole time, but at least I learned some things about sound processing and the binary structure of sound files. \n\n# Unsolved problems\n\n### Granny 3\n\nI think I spent close 100 hours on this one. From DE to multiple other genetic algorithms, nothing got a decent solution. Didn't manage to fool the endpoint that the image sent did not differ in more than one pixel either, but I honestly suspect that this task was *not* about finding the right pixel.\n\nAfter the naive solutions didn't lead to anything, I brute-forced contrastive values in search for insight. Here are some intersections of the best pixels for different values using the top 1000 by apple score of each. \n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4944379%2Fc14adb8b65eb39e7a19786844366db0f%2Fimagy-image%20(4).jpg?generation=1699905113938883&alt=media) \n\nConsidering that the 768x768 mapped perfectly to 256x256 in a 3:1 ratio, the search space was less than 9*[256/9]^3 for each pixel. The image did not have many sharp contrast variations, so for most of them, 3-4e4 combinations exhausted the space. After over 1e7 tries, the best was still below 7e-4.\n\nAfter I finished with the top 100 pixels with the best gradients, I also tried working with a small version of imagenet, and getting the most popular shades of green, but they weren't any better.\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4944379%2Fa6fac9161fcc14ceabde4b56dd542775%2Fimagy-image%20(3).jpg?generation=1699904822205068&alt=media)\n\nHere are some other interesting things GPT4 offered while trying to solve it:\n\nGradient map layer by layer:\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4944379%2F73de8b8f6129073c9d8167cb796d5c13%2Fimagy-image%20(8).jpg?generation=1699904629973596&alt=media)\n\nHere's the basic gradient map for the apple class: \n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4944379%2Fc136a9c9073465b4dcf4398cd4ddd5f5%2Fimagy-image%20(7).jpg?generation=1699905202373919&alt=media)\n\nAnd here's the heatmap of aggregate best results:\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4944379%2Fd230ec38cb88360cfbafb558d3caf49d%2Fimagy-image%20(5).jpg?generation=1699905247366019&alt=media)\n\nHere's the overlayed image:\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4944379%2Feef3f394090bca0e8183e5cd2b08a587%2Fimagy-image%20(9).jpg?generation=1699905357017978&alt=media)\n\n\nNotes:\n- the gradient map does not match the best results\n- best pixel placements were a bit surprising, at least for me\n\nI'll keep updating this as I find/remember more results/observations, my notebooks are all over the place😅", "votes": 10 }, { "id": 2523959, "postDate": "2023-11-13T21:42:59.583Z", "content": "
\n

CIFAR was simple too, all you had to do was send a (100,4) matrix that counted data from the dataset using the hint from input_data.

\n
\n

\"\"

", "rawMarkdown": ">CIFAR was simple too, all you had to do was send a (100,4) matrix that counted data from the dataset using the hint from input_data.\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F644036%2F49a47bd0e8de0ccecabaccbd35acdcf6%2Fstare-blank.gif?generation=1699911765896820&alt=media)", "votes": 7, "replies": [ { "id": 2523970, "postDate": "2023-11-13T22:13:59.907Z", "content": "

\"\"

", "rawMarkdown": "![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4944379%2F992da590bc94dd7452b12c10dd36c73b%2FScreenshot%20from%202023-11-14%2000-12-43.png?generation=1699913593317603&alt=media)", "votes": 5 } ] }, { "id": 2524115, "postDate": "2023-11-14T02:12:49.490Z", "content": "

we need CIFAR please🙏

", "rawMarkdown": "we need CIFAR please🙏" }, { "id": 2524102, "postDate": "2023-11-14T02:00:15.643Z", "content": "
\n

Didn't expect to get a medal, let alone a gold😅.

\n
\n

Huh? Sounds like you're not aware that you win the cash prize? 😅
\nCongratulation!

\n

Seems like you've solved CIFAR. Would you mind elaborating at what the solution (100, 4) is?

\n

Thank you.

", "rawMarkdown": "> Didn't expect to get a medal, let alone a gold😅.\n\nHuh? Sounds like you're not aware that you win the cash prize? 😅\nCongratulation!\n\nSeems like you've solved CIFAR. Would you mind elaborating at what the solution (100, 4) is?\n\nThank you." } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T16:08:59Z", "competition_id": 57818, "competition_slug": "ai-village-capture-the-flag-defcon31", "team_id": 11087004, "solution_url": "/competitions/ai-village-capture-the-flag-defcon31/writeups/horea-5th-place-24-27-lessons-from-my-first-ctf-ho" }