This is our first year running something on this scale. We'd love your feedback for future iterations. What did you love? What did you hate?
\nNothing that spoils challenges for others before September 12th. I'm just starting this thread now in case DEFCON attendees aren't sticking around for the full month.
\nAlso, if you want to participate in the future (either as a volunteer or as a competitor), send me your contact info and I'll keep you in the loop: ctf@aivillage.org. Following @aivillage_dc, @moo_hax, and @josephtlucas on Twitter is another good way to stay in touch.
I'll start: The token server falling over was a real bummer. What else?
This is our first year running something on this scale. We'd love your feedback for future iterations. What did you love? What did you hate?
\nNothing that spoils challenges for others before September 12th. I'm just starting this thread now in case DEFCON attendees aren't sticking around for the full month.
\nAlso, if you want to participate in the future (either as a volunteer or as a competitor), send me your contact info and I'll keep you in the loop: ctf@aivillage.org. Following @aivillage_dc, @moo_hax, and @josephtlucas on Twitter is another good way to stay in touch.
I'll start: The token server falling over was a real bummer. What else?
Next time, give more notice of the competition starting. It's a time sensitive challenge and not everyone can start immediately. Give 2 or 4 weeks notice.
", "rawMarkdown": "Next time, give more notice of the competition starting. It's a time sensitive challenge and not everyone can start immediately. Give 2 or 4 weeks notice.", "votes": 6, "replies": [ { "id": 1908171, "postDate": "2022-08-21T12:16:36.427Z", "content": "Seconded.
\nYou could additionally release the competitions in intervals. E.g. if the competition runs for 4 weeks, release 1/3 immediately, another 3rd after a week, the rest after 2 weeks, leaving 2 more weeks. That allows people to catch up.
Oh that's a good one, thanks.
", "rawMarkdown": "Oh that's a good one, thanks.", "votes": 1 } ] }, { "id": 1940715, "postDate": "2022-09-15T13:58:37.820Z", "content": "I want to give a big thanks to the hosts for putting on an amazing competition. This was a masterclass in how a competition should be run IMO. The competition was challenging, fun, and I learned a ton about an area of ML that I didn't know about before. The only thing I wish would've been different is I wish I had more free time to work on it!
\nThis was my first time doing a CTF and the way you guys set it up so that we could still submit to the leaderboard was great. It was also nice to have a discord channel to scroll through and see what was being discussed.
\nI also was a big fan of having it be a solo competition, and not allowing any sharing at all. That wouldn't be best for other kaggle competitions but in this situation it worked great and made us feel like we were really going \"head to head\" with other competitors.
\nI didn't make a formal writeup about my solutions, mainly because it's been a while and I've forgotten a few things I did, I'm lazy, my code is a mess because I was just trying to fly through them as fast as possible, and also others have posted much better solutions at this point. But I did discuss it on one of my latest coding streams you can find on twitch/youtube.
\nDo it again next year please!
\nAlso sloth was frustrating- but that's not a bad thing. And I guess crop2 doesn't have a solution? I spent a long time on crop2 before just giving up. Just give us the solution! :D
", "rawMarkdown": "I want to give a big thanks to the hosts for putting on an amazing competition. This was a masterclass in how a competition should be run IMO. The competition was challenging, fun, and I learned a ton about an area of ML that I didn't know about before. The only thing I wish would've been different is I wish I had more free time to work on it!\n\nThis was my first time doing a CTF and the way you guys set it up so that we could still submit to the leaderboard was great. It was also nice to have a discord channel to scroll through and see what was being discussed.\n\nI also was a big fan of having it be a solo competition, and not allowing any sharing at all. That wouldn't be best for other kaggle competitions but in this situation it worked great and made us feel like we were really going \"head to head\" with other competitors.\n\nI didn't make a formal writeup about my solutions, mainly because it's been a while and I've forgotten a few things I did, I'm lazy, my code is a mess because I was just trying to fly through them as fast as possible, and also others have posted much better solutions at this point. But I did discuss it on one of my latest coding streams you can find on twitch/youtube.\n\nDo it again next year please!\n\nAlso sloth was frustrating- but that's not a bad thing. And I guess crop2 doesn't have a solution? I spent a long time on crop2 before just giving up. Just give us the solution! :D", "votes": 4, "replies": [ { "id": 1947637, "postDate": "2022-09-20T15:11:14.470Z", "content": "For crop2, although not an official solution, the following solution has been published by @gladwell.
\nhttps://www.kaggle.com/code/gladwell/one-crop2-solution
", "rawMarkdown": "For crop2, although not an official solution, the following solution has been published by @gladwell.\n\nhttps://www.kaggle.com/code/gladwell/one-crop2-solution", "votes": 1 } ] }, { "id": 1935337, "postDate": "2022-09-12T03:39:00.297Z", "content": "This competition was really enjoyable. I have done a few CTF style questions on other websites and always been vaguely curious, but the type of problems were usually in domains I was just not that adept in, so I could never properly get into the competitions. I had no clue it was possible that there could be just so much overlap between data science/ML and CTF style questions until this competition!
\nHuge credit to the question authors - for every single question I learned so much that I didn't know either about data science in general or about what you could do from a hacking perspective in ML. It was really fun and rewarding to participate, and every flag I got was so satisfying.
\nI think the only things to note would be that token being brute forcible and deepfake having an easy out of using any video of the same length to pass the test robbed people of the joy of having to properly understand how to solve what the question authors were getting at. But it was still satisfying to hack the questions so maybe that's part of the fun.
\nOutside of the questions, here would be my main points:
\nReally glad you kept the teamsize to 1 - this made it as fair as possible and meant everyone got to really enjoy and understand every single question. Imagine if you never had to even try and solve half the problems because your teammates solved them before you read them? Sure you could read them in your own time, but you'd read them and attempt them already knowing how to solve it because your teammate would have told you in their explanation.
I am perhaps bias here, but it would have been nice to win a medal and gain competition ranking points XD . I would also add that I wonder if even more people would have participated if these rewards were on the line, perhaps some people saw the comp, thought it was interesting, but without medals on the line opted not to properly sink time into it.
In terms of how the competition is structured, the only real thing I can say is giving people more notice of when the competition was starting would have made it fairer, and I think you'd have got a lot more people participating. I stumbled upon this competition by chance 3 days after it's start point, but I appreciate for people joining much later they may have been demotivated to participate given how far other people were ahead.
So many thanks to AI Village and Kaggle for such a great comp, will definitely see you next year if you do another one!
", "rawMarkdown": "This competition was really enjoyable. I have done a few CTF style questions on other websites and always been vaguely curious, but the type of problems were usually in domains I was just not that adept in, so I could never properly get into the competitions. I had no clue it was possible that there could be just so much overlap between data science/ML and CTF style questions until this competition! \n\nHuge credit to the question authors - for every single question I learned so much that I didn't know either about data science in general or about what you could do from a hacking perspective in ML. It was really fun and rewarding to participate, and every flag I got was so satisfying.\n\nI think the only things to note would be that token being brute forcible and deepfake having an easy out of using any video of the same length to pass the test robbed people of the joy of having to properly understand how to solve what the question authors were getting at. But it was still satisfying to hack the questions so maybe that's part of the fun.\n\nOutside of the questions, here would be my main points:\n\n- Really glad you kept the teamsize to 1 - this made it as fair as possible and meant everyone got to really enjoy and understand every single question. Imagine if you never had to even try and solve half the problems because your teammates solved them before you read them? Sure you could read them in your own time, but you'd read them and attempt them already knowing how to solve it because your teammate would have told you in their explanation.\n\n- I am perhaps bias here, but it would have been nice to win a medal and gain competition ranking points XD . I would also add that I wonder if even more people would have participated if these rewards were on the line, perhaps some people saw the comp, thought it was interesting, but without medals on the line opted not to properly sink time into it. \n\n- In terms of how the competition is structured, the only real thing I can say is giving people more notice of when the competition was starting would have made it fairer, and I think you'd have got a lot more people participating. I stumbled upon this competition by chance 3 days after it's start point, but I appreciate for people joining much later they may have been demotivated to participate given how far other people were ahead.\n\nSo many thanks to AI Village and Kaggle for such a great comp, will definitely see you next year if you do another one!\n", "votes": 4, "replies": [ { "id": 1936745, "postDate": "2022-09-13T03:11:21.027Z", "content": "Good review and insights. I think the best way to award medals and ranking points would be to organize the competition as multiple rounds. Every few weeks, the hosts reveal new challenges. In the first few rounds there are not medals or ranking points. But then in the final rounds, there are medals and ranking points.
\nThis would help advertise the competition. The first few rounds would attract participants and then everyone would start on the same day to fight for medals and ranking points.
", "rawMarkdown": "Good review and insights. I think the best way to award medals and ranking points would be to organize the competition as multiple rounds. Every few weeks, the hosts reveal new challenges. In the first few rounds there are not medals or ranking points. But then in the final rounds, there are medals and ranking points.\n\nThis would help advertise the competition. The first few rounds would attract participants and then everyone would start on the same day to fight for medals and ranking points.", "votes": 1 } ] }, { "id": 1909143, "postDate": "2022-08-22T11:22:18.297Z", "content": "Great idea! I'm off to go secure some swag…
", "rawMarkdown": "Great idea! I'm off to go secure some swag...", "votes": 4 }, { "id": 1909745, "postDate": "2022-08-22T22:05:16.730Z", "content": "This changes.. everything. 👀
", "rawMarkdown": "This changes.. everything. 👀" }, { "id": 1918731, "postDate": "2022-08-29T19:34:34.227Z", "content": "totally agree. would have posted my solution for crop2 if it made me win a hoodie ^^
\n(more seriously, I think some t shirts for some top ranks after 5 first ones could be a good idea. Would be proud to show what website I dedicated one month to)
I just ordered some goodies. Any proposals on categories?
\nI'm thinking:
\nEnjoyed the competition. 😄
\nLearned new things and was able practice some things that I've been studying.
\nStill learning a lot from the shared notebooks after the competition. Even for the few problems I was able to answer, it is really interesting to see different kinds of approach.
Nothing to say other that I love the concept of this challenge. It’s so different to the standard nlp and cv challenges we normally see. Super bummed that I’m overseas for the vast majority of it and can’t really compete. Looking forward to seeing write ups of how everyone got it done.
", "rawMarkdown": "Nothing to say other that I love the concept of this challenge. It’s so different to the standard nlp and cv challenges we normally see. Super bummed that I’m overseas for the vast majority of it and can’t really compete. Looking forward to seeing write ups of how everyone got it done.", "votes": 1, "replies": [ { "id": 1930123, "postDate": "2022-09-07T15:20:55.107Z", "content": "Let us know if there's any way to make this more accessible to overseas competitors next years. I think we had a few, but we're happy to help facilitate that next year.
", "rawMarkdown": "Let us know if there's any way to make this more accessible to overseas competitors next years. I think we had a few, but we're happy to help facilitate that next year." }, { "id": 1930507, "postDate": "2022-09-08T01:11:17.290Z", "content": "Oh it’s not because I’m in a different location, it’s just that all I have here is my iPhone (no PC, GPUs etc). Plus family would probably be a bit upset if I bailed on them to do CTF 😉
", "rawMarkdown": "Oh it’s not because I’m in a different location, it’s just that all I have here is my iPhone (no PC, GPUs etc). Plus family would probably be a bit upset if I bailed on them to do CTF 😉 ", "votes": 3 }, { "id": 1930745, "postDate": "2022-09-08T06:52:48.590Z", "content": "True 21st century hacker uses a smartphone!
\nSwiping instead of typing!
Maybe so, but I’m getting old. I still remember the Commodore 64 and 56kbps internet being blistering fast 😅
", "rawMarkdown": "Maybe so, but I’m getting old. I still remember the Commodore 64 and 56kbps internet being blistering fast 😅", "votes": 3 } ] }, { "id": 1914543, "postDate": "2022-08-26T07:20:31.713Z", "content": "This challenge may have may sparked Kagglers interest in the world of CTFs (and CTFers in the world of Kaggle).
\nWhere would one go to get a little more into CTFs? Especially ones more data science -esque. I can see many historical writeups. But is there some reliable source of future challenges?
", "rawMarkdown": "This challenge may have may sparked Kagglers interest in the world of CTFs (and CTFers in the world of Kaggle).\n\nWhere would one go to get a little more into CTFs? Especially ones more data science -esque. I can see many historical writeups. But is there some reliable source of future challenges?", "votes": 2, "replies": [ { "id": 1914683, "postDate": "2022-08-26T09:57:20.893Z", "content": "I'm glad to hear that! For cybersecurity CTF's, this is probably the best source of upcoming events: https://ctftime.org/event/list/. For those just getting started, I think PicoGym is great: https://play.picoctf.org/login?redirect=/practice.
\nAs for DS-like CTFs, I don't think there have been any on this scale. Sometimes there are specific challenges (often associated with conferences like NIPS or CAMLIS). Our current plan is to run a new CTF like this each summer for DEFCON, but it takes a lot of time/energy, so I'm glad to hear it's going in the right direction.
\nIn terms of a reliable source for future challenges, I think the AI Village (either on Discord or Twitter) is probably the best way to stay \"in tune\" with these sorts of events (and similar calls-for-papers, news articles, etc). https://twitter.com/aivillage_dc
", "rawMarkdown": "I'm glad to hear that! For cybersecurity CTF's, this is probably the best source of upcoming events: https://ctftime.org/event/list/. For those just getting started, I think PicoGym is great: https://play.picoctf.org/login?redirect=/practice.\n\nAs for DS-like CTFs, I don't think there have been any on this scale. Sometimes there are specific challenges (often associated with conferences like NIPS or CAMLIS). Our current plan is to run a new CTF like this each summer for DEFCON, but it takes a lot of time/energy, so I'm glad to hear it's going in the right direction.\n\nIn terms of a reliable source for future challenges, I think the AI Village (either on Discord or Twitter) is probably the best way to stay \"in tune\" with these sorts of events (and similar calls-for-papers, news articles, etc). https://twitter.com/aivillage_dc", "votes": 3 } ] }, { "id": 1920489, "postDate": "2022-08-31T07:25:03.917Z", "content": "Hi, maybe it would be more interesting to have the puzzles more integrated,
\nlike chain the puzzles together, only solvers of the first puzzle can see the second one. like a wargame
I wouldn't be interested in competing under those circumstances, it would penalise gaps in people's knowledge or experience too much.
", "rawMarkdown": "I wouldn't be interested in competing under those circumstances, it would penalise gaps in people's knowledge or experience too much.", "votes": 5 } ] }, { "id": 1952695, "postDate": "2022-09-23T20:36:14.713Z", "content": "Thanks for the competition, Joe! For the next one, I would like to see more ML challenges and less WAF/sloth.
", "rawMarkdown": "Thanks for the competition, Joe! For the next one, I would like to see more ML challenges and less WAF/sloth." }, { "id": 1938093, "postDate": "2022-09-13T21:59:16.147Z", "content": "I think crop2 can be used as a tie-breaker = rather than judging whether it's correct or not, we can score the participant answer based on how close it to the real answer. The score will be in range from [0..800]. I believe Kaggle has that kind of feature, seeing from how other competitions are being scored.
That way the competition is not only about being \"the fastest is the winner\" anymore. And we can see more dynamics in the top ranked leaderboard during the competition.
\nThough I believe everyone will just exhaust the submission quota to brute-force the answer :p
", "rawMarkdown": "I think `crop2` can be used as a tie-breaker = rather than judging whether it's correct or not, we can score the participant answer based on how close it to the real answer. The score will be in range from [0..800]. I believe Kaggle has that kind of feature, seeing from how other competitions are being scored.\n\nThat way the competition is not only about being \"the fastest is the winner\" anymore. And we can see more dynamics in the top ranked leaderboard during the competition.\n\nThough I believe everyone will just exhaust the submission quota to brute-force the answer :p", "replies": [ { "id": 1938983, "postDate": "2022-09-14T13:12:55.097Z", "content": "This proposal assumes that the organizers will know beforehand that this leaderboard setup will occur, i.e. that there will be a lot of people with the same score and one unsolved task.
\nI do not think the organizers could have predicted that. We have the benefit of hindsight now.
\nAnd changing the rules in the middle of the competition is unfair as well.
\nMaybe it would be possible to make one of the tasks a tiebreaker, but then it should be a well-defined \"unsolvable\" optimization task. But then again, this is what most of the Kaggle competitions are already, this one was supposed to be different. 😛
I love that last idea, thanks!
\nAs to walkthroughs, we're hoping that published solutions from this year can help provide foundational material for people to solve similar challenges in the future…. so make sure to publish your notebook after the competition ends!
", "rawMarkdown": "I love that last idea, thanks! \n\nAs to walkthroughs, we're hoping that published solutions from this year can help provide foundational material for people to solve similar challenges in the future.... so make sure to publish your notebook after the competition ends!", "votes": 5 } ] } ], "comments": [ { "id": 1907938, "author_name": "robga", "author_url": "", "post_date": "2022-08-21T07:54:13.717000", "content": "Next time, give more notice of the competition starting. It's a time sensitive challenge and not everyone can start immediately. Give 2 or 4 weeks notice.
", "votes": 6, "replies": [ { "id": 1908171, "author_name": "Gabriel Lindenmaier", "author_url": "", "post_date": "2022-08-21T12:16:36.427000", "content": "Seconded.
\nYou could additionally release the competitions in intervals. E.g. if the competition runs for 4 weeks, release 1/3 immediately, another 3rd after a week, the rest after 2 weeks, leaving 2 more weeks. That allows people to catch up.
Oh that's a good one, thanks.
", "votes": 1, "replies": [] } ] }, { "id": 1940715, "author_name": "Rob Mulla", "author_url": "", "post_date": "2022-09-15T13:58:37.820000", "content": "I want to give a big thanks to the hosts for putting on an amazing competition. This was a masterclass in how a competition should be run IMO. The competition was challenging, fun, and I learned a ton about an area of ML that I didn't know about before. The only thing I wish would've been different is I wish I had more free time to work on it!
\nThis was my first time doing a CTF and the way you guys set it up so that we could still submit to the leaderboard was great. It was also nice to have a discord channel to scroll through and see what was being discussed.
\nI also was a big fan of having it be a solo competition, and not allowing any sharing at all. That wouldn't be best for other kaggle competitions but in this situation it worked great and made us feel like we were really going \"head to head\" with other competitors.
\nI didn't make a formal writeup about my solutions, mainly because it's been a while and I've forgotten a few things I did, I'm lazy, my code is a mess because I was just trying to fly through them as fast as possible, and also others have posted much better solutions at this point. But I did discuss it on one of my latest coding streams you can find on twitch/youtube.
\nDo it again next year please!
\nAlso sloth was frustrating- but that's not a bad thing. And I guess crop2 doesn't have a solution? I spent a long time on crop2 before just giving up. Just give us the solution! :D
", "votes": 4, "replies": [ { "id": 1947637, "author_name": "Bilzard", "author_url": "", "post_date": "2022-09-20T15:11:14.470000", "content": "For crop2, although not an official solution, the following solution has been published by @gladwell.
\nhttps://www.kaggle.com/code/gladwell/one-crop2-solution
", "votes": 1, "replies": [] } ] }, { "id": 1935337, "author_name": "John Macgillivray", "author_url": "", "post_date": "2022-09-12T03:39:00.297000", "content": "This competition was really enjoyable. I have done a few CTF style questions on other websites and always been vaguely curious, but the type of problems were usually in domains I was just not that adept in, so I could never properly get into the competitions. I had no clue it was possible that there could be just so much overlap between data science/ML and CTF style questions until this competition!
\nHuge credit to the question authors - for every single question I learned so much that I didn't know either about data science in general or about what you could do from a hacking perspective in ML. It was really fun and rewarding to participate, and every flag I got was so satisfying.
\nI think the only things to note would be that token being brute forcible and deepfake having an easy out of using any video of the same length to pass the test robbed people of the joy of having to properly understand how to solve what the question authors were getting at. But it was still satisfying to hack the questions so maybe that's part of the fun.
\nOutside of the questions, here would be my main points:
\nReally glad you kept the teamsize to 1 - this made it as fair as possible and meant everyone got to really enjoy and understand every single question. Imagine if you never had to even try and solve half the problems because your teammates solved them before you read them? Sure you could read them in your own time, but you'd read them and attempt them already knowing how to solve it because your teammate would have told you in their explanation.
I am perhaps bias here, but it would have been nice to win a medal and gain competition ranking points XD . I would also add that I wonder if even more people would have participated if these rewards were on the line, perhaps some people saw the comp, thought it was interesting, but without medals on the line opted not to properly sink time into it.
In terms of how the competition is structured, the only real thing I can say is giving people more notice of when the competition was starting would have made it fairer, and I think you'd have got a lot more people participating. I stumbled upon this competition by chance 3 days after it's start point, but I appreciate for people joining much later they may have been demotivated to participate given how far other people were ahead.
So many thanks to AI Village and Kaggle for such a great comp, will definitely see you next year if you do another one!
", "votes": 4, "replies": [ { "id": 1936745, "author_name": "Chris Deotte", "author_url": "", "post_date": "2022-09-13T03:11:21.027000", "content": "Good review and insights. I think the best way to award medals and ranking points would be to organize the competition as multiple rounds. Every few weeks, the hosts reveal new challenges. In the first few rounds there are not medals or ranking points. But then in the final rounds, there are medals and ranking points.
\nThis would help advertise the competition. The first few rounds would attract participants and then everyone would start on the same day to fight for medals and ranking points.
", "votes": 1, "replies": [] } ] }, { "id": 1909143, "author_name": "robga", "author_url": "", "post_date": "2022-08-22T11:22:18.297000", "content": "Great idea! I'm off to go secure some swag…
", "votes": 4, "replies": [] }, { "id": 1909745, "author_name": "arthur", "author_url": "", "post_date": "2022-08-22T22:05:16.730000", "content": "This changes.. everything. 👀
", "votes": 0, "replies": [] }, { "id": 1918731, "author_name": "HiddenCell", "author_url": "", "post_date": "2022-08-29T19:34:34.227000", "content": "totally agree. would have posted my solution for crop2 if it made me win a hoodie ^^
\n(more seriously, I think some t shirts for some top ranks after 5 first ones could be a good idea. Would be proud to show what website I dedicated one month to)
I just ordered some goodies. Any proposals on categories?
\nI'm thinking:
\nEnjoyed the competition. 😄
\nLearned new things and was able practice some things that I've been studying.
\nStill learning a lot from the shared notebooks after the competition. Even for the few problems I was able to answer, it is really interesting to see different kinds of approach.
Nothing to say other that I love the concept of this challenge. It’s so different to the standard nlp and cv challenges we normally see. Super bummed that I’m overseas for the vast majority of it and can’t really compete. Looking forward to seeing write ups of how everyone got it done.
", "votes": 1, "replies": [ { "id": 1930123, "author_name": "Joe Lucas", "author_url": "", "post_date": "2022-09-07T15:20:55.107000", "content": "Let us know if there's any way to make this more accessible to overseas competitors next years. I think we had a few, but we're happy to help facilitate that next year.
", "votes": 0, "replies": [] }, { "id": 1930507, "author_name": "Michael", "author_url": "", "post_date": "2022-09-08T01:11:17.290000", "content": "Oh it’s not because I’m in a different location, it’s just that all I have here is my iPhone (no PC, GPUs etc). Plus family would probably be a bit upset if I bailed on them to do CTF 😉
", "votes": 3, "replies": [] }, { "id": 1930745, "author_name": "fauxsmart", "author_url": "", "post_date": "2022-09-08T06:52:48.590000", "content": "True 21st century hacker uses a smartphone!
\nSwiping instead of typing!
Maybe so, but I’m getting old. I still remember the Commodore 64 and 56kbps internet being blistering fast 😅
", "votes": 3, "replies": [] } ] }, { "id": 1914543, "author_name": "robga", "author_url": "", "post_date": "2022-08-26T07:20:31.713000", "content": "This challenge may have may sparked Kagglers interest in the world of CTFs (and CTFers in the world of Kaggle).
\nWhere would one go to get a little more into CTFs? Especially ones more data science -esque. I can see many historical writeups. But is there some reliable source of future challenges?
", "votes": 2, "replies": [ { "id": 1914683, "author_name": "Joe Lucas", "author_url": "", "post_date": "2022-08-26T09:57:20.893000", "content": "I'm glad to hear that! For cybersecurity CTF's, this is probably the best source of upcoming events: https://ctftime.org/event/list/. For those just getting started, I think PicoGym is great: https://play.picoctf.org/login?redirect=/practice.
\nAs for DS-like CTFs, I don't think there have been any on this scale. Sometimes there are specific challenges (often associated with conferences like NIPS or CAMLIS). Our current plan is to run a new CTF like this each summer for DEFCON, but it takes a lot of time/energy, so I'm glad to hear it's going in the right direction.
\nIn terms of a reliable source for future challenges, I think the AI Village (either on Discord or Twitter) is probably the best way to stay \"in tune\" with these sorts of events (and similar calls-for-papers, news articles, etc). https://twitter.com/aivillage_dc
", "votes": 3, "replies": [] } ] }, { "id": 1920489, "author_name": "guanw", "author_url": "", "post_date": "2022-08-31T07:25:03.917000", "content": "Hi, maybe it would be more interesting to have the puzzles more integrated,
\nlike chain the puzzles together, only solvers of the first puzzle can see the second one. like a wargame
I wouldn't be interested in competing under those circumstances, it would penalise gaps in people's knowledge or experience too much.
", "votes": 5, "replies": [] } ] }, { "id": 1952695, "author_name": "Felipe Loque", "author_url": "", "post_date": "2022-09-23T20:36:14.713000", "content": "Thanks for the competition, Joe! For the next one, I would like to see more ML challenges and less WAF/sloth.
", "votes": 0, "replies": [] }, { "id": 1938093, "author_name": "Thariq Nugrohotomo", "author_url": "", "post_date": "2022-09-13T21:59:16.147000", "content": "I think crop2 can be used as a tie-breaker = rather than judging whether it's correct or not, we can score the participant answer based on how close it to the real answer. The score will be in range from [0..800]. I believe Kaggle has that kind of feature, seeing from how other competitions are being scored.
That way the competition is not only about being \"the fastest is the winner\" anymore. And we can see more dynamics in the top ranked leaderboard during the competition.
\nThough I believe everyone will just exhaust the submission quota to brute-force the answer :p
", "votes": 0, "replies": [ { "id": 1938983, "author_name": "fauxsmart", "author_url": "", "post_date": "2022-09-14T13:12:55.097000", "content": "This proposal assumes that the organizers will know beforehand that this leaderboard setup will occur, i.e. that there will be a lot of people with the same score and one unsolved task.
\nI do not think the organizers could have predicted that. We have the benefit of hindsight now.
\nAnd changing the rules in the middle of the competition is unfair as well.
\nMaybe it would be possible to make one of the tasks a tiebreaker, but then it should be a well-defined \"unsolvable\" optimization task. But then again, this is what most of the Kaggle competitions are already, this one was supposed to be different. 😛
I love that last idea, thanks!
\nAs to walkthroughs, we're hoping that published solutions from this year can help provide foundational material for people to solve similar challenges in the future…. so make sure to publish your notebook after the competition ends!
", "votes": 5, "replies": [] } ] } ], "raw_markdown_by_id": { "1906036": "This is our first year running something on this scale. We'd love your feedback for future iterations. What did you love? What did you hate? \n\n**Nothing that spoils challenges for others before September 12th. I'm just starting this thread now in case DEFCON attendees aren't sticking around for the full month.**\n\nAlso, if you want to participate in the future (either as a volunteer or as a competitor), send me your contact info and I'll keep you in the loop: ctf@aivillage.org. Following `@aivillage_dc`, `@moo_hax`, and `@josephtlucas` on Twitter is another good way to stay in touch.\n\nI'll start: The `token` server falling over was a real bummer. What else?", "1907938": "Next time, give more notice of the competition starting. It's a time sensitive challenge and not everyone can start immediately. Give 2 or 4 weeks notice.", "1940715": "I want to give a big thanks to the hosts for putting on an amazing competition. This was a masterclass in how a competition should be run IMO. The competition was challenging, fun, and I learned a ton about an area of ML that I didn't know about before. The only thing I wish would've been different is I wish I had more free time to work on it!\n\nThis was my first time doing a CTF and the way you guys set it up so that we could still submit to the leaderboard was great. It was also nice to have a discord channel to scroll through and see what was being discussed.\n\nI also was a big fan of having it be a solo competition, and not allowing any sharing at all. That wouldn't be best for other kaggle competitions but in this situation it worked great and made us feel like we were really going \"head to head\" with other competitors.\n\nI didn't make a formal writeup about my solutions, mainly because it's been a while and I've forgotten a few things I did, I'm lazy, my code is a mess because I was just trying to fly through them as fast as possible, and also others have posted much better solutions at this point. But I did discuss it on one of my latest coding streams you can find on twitch/youtube.\n\nDo it again next year please!\n\nAlso sloth was frustrating- but that's not a bad thing. And I guess crop2 doesn't have a solution? I spent a long time on crop2 before just giving up. Just give us the solution! :D", "1935337": "This competition was really enjoyable. I have done a few CTF style questions on other websites and always been vaguely curious, but the type of problems were usually in domains I was just not that adept in, so I could never properly get into the competitions. I had no clue it was possible that there could be just so much overlap between data science/ML and CTF style questions until this competition! \n\nHuge credit to the question authors - for every single question I learned so much that I didn't know either about data science in general or about what you could do from a hacking perspective in ML. It was really fun and rewarding to participate, and every flag I got was so satisfying.\n\nI think the only things to note would be that token being brute forcible and deepfake having an easy out of using any video of the same length to pass the test robbed people of the joy of having to properly understand how to solve what the question authors were getting at. But it was still satisfying to hack the questions so maybe that's part of the fun.\n\nOutside of the questions, here would be my main points:\n\n- Really glad you kept the teamsize to 1 - this made it as fair as possible and meant everyone got to really enjoy and understand every single question. Imagine if you never had to even try and solve half the problems because your teammates solved them before you read them? Sure you could read them in your own time, but you'd read them and attempt them already knowing how to solve it because your teammate would have told you in their explanation.\n\n- I am perhaps bias here, but it would have been nice to win a medal and gain competition ranking points XD . I would also add that I wonder if even more people would have participated if these rewards were on the line, perhaps some people saw the comp, thought it was interesting, but without medals on the line opted not to properly sink time into it. \n\n- In terms of how the competition is structured, the only real thing I can say is giving people more notice of when the competition was starting would have made it fairer, and I think you'd have got a lot more people participating. I stumbled upon this competition by chance 3 days after it's start point, but I appreciate for people joining much later they may have been demotivated to participate given how far other people were ahead.\n\nSo many thanks to AI Village and Kaggle for such a great comp, will definitely see you next year if you do another one!\n", "1909143": "- Many kaggle challenges have a \"best writeup\" or \"best notebook\" award. Post competition, you'd get more writeups, and better ones, if there was an incentive prize. Even swag - it's amazing the amount of work a kaggler will do for a chance at a mug and hoodie.\n- Kaggle has spurts of openness. You could make it mandatory for prizewinners to publish their repeatable solutions. This'd be a nice check vs the often secretive world of security hacks.\n", "1935772": "Enjoyed the competition. 😄\nLearned new things and was able practice some things that I've been studying.\nStill learning a lot from the shared notebooks after the competition. Even for the few problems I was able to answer, it is really interesting to see different kinds of approach.", "1929379": "Nothing to say other that I love the concept of this challenge. It’s so different to the standard nlp and cv challenges we normally see. Super bummed that I’m overseas for the vast majority of it and can’t really compete. Looking forward to seeing write ups of how everyone got it done.", "1914543": "This challenge may have may sparked Kagglers interest in the world of CTFs (and CTFers in the world of Kaggle).\n\nWhere would one go to get a little more into CTFs? Especially ones more data science -esque. I can see many historical writeups. But is there some reliable source of future challenges?", "1920489": "Hi, maybe it would be more interesting to have the puzzles more integrated, \nlike chain the puzzles together, only solvers of the first puzzle can see the second one. like a wargame", "1952695": "Thanks for the competition, Joe! For the next one, I would like to see more ML challenges and less WAF/sloth.", "1938093": "I think `crop2` can be used as a tie-breaker = rather than judging whether it's correct or not, we can score the participant answer based on how close it to the real answer. The score will be in range from [0..800]. I believe Kaggle has that kind of feature, seeing from how other competitions are being scored.\n\nThat way the competition is not only about being \"the fastest is the winner\" anymore. And we can see more dynamics in the top ranked leaderboard during the competition.\n\nThough I believe everyone will just exhaust the submission quota to brute-force the answer :p", "1906050": "" } }