If you attended the closing ceremony, skip straight to the Feedback Survey section — the first section repeats the same findings.
As announced in my previous blogpost, NorthSec allowed AI agents without restriction during the CTF this year. We asked the community to help us out by reporting flags found with AI agents. We also asked participants for their opinion on AI and how we should adapt. Here’s my analysis of that data.
Flag data
Caveat on this data: several teams (including top teams) didn’t rigorously declare their use of agents to find flags. These numbers underrepresent the real amount of flags found by AI agents.
Data: 84% (76/90) of CTF teams declared a flag found with an AI agent. Takeaway: The vast majority of teams use these tools, and they’re part of the workflow for a large share of participants.
Data: 29% of the CTF’s valid points were declared as found by AI agent. Takeaway: Since this includes teams that used agents little or not at all, we can assume the majority of points found by teams that do use agents came through those agents.
Data: 89% (143/160) of the CTF’s distinct flags had at least one solve by an agent. I manually sanitized this data to remove a few false positives. Takeaway: The overwhelming majority of the CTF could be solved by an AI agent. Since we had a bit under 20 physical flags, we can assume that nearly every non-physical challenge is sloppable or near-sloppable.
Data: Teams in the second quintile submitted the smallest proportion of points with agents, and the last quintile submitted the most (~38%). Takeaway: The use of AI agents (or effectiveness at using them) isn’t uniform. In every case, they contribute a share of points that can’t be ignored.
Data: Proportion of flags submitted by humans vs. agents by flag value. Takeaway: AI agents seem very useful for solving the hardest challenges, not just the easy ones.
Data: The proportion of flags submitted by humans vs. agents over time. Takeaway: The proportion stayed stable for the entire duration of the CTF, which seems to indicate that the “out of tokens” effect didn’t play a significant role.
Data: Comparison of the number of valid flags submitted since the start of the CTF for 2025 (161 flags, 90 teams) vs. 2026 (165 flags, 94 teams). Takeaway: 2026’s AI increased the number of flags submitted per team by about 90% on Friday night. That figure drops to 60% by the end of the CTF.
Data: Same as the previous chart, but for the top 10 teams of each CTF. Takeaway: The gap is bigger. On Friday night, 2026’s AI contributed to a 136% increase in flags submitted. That figure drops to 63% by the end of the CTF.
Post-CTF feedback survey
Caveat on this data: this sample is non-probabilistic, since respondents weren’t randomly selected. That said, we made several efforts to reach as many people as possible (announcement at the closing ceremony, on Discord, and by email), and more than 40% of participants filled out the survey.
Some data to understand the respondents’ profile: Number of responses: 284 Number of first-timers at NorthSec CTF: 34% Number of NorthSec CTF participations (1–12): average of 4.63, median of 4 Why come to the CTF: 22% to be competitive, 73% to learn, 5% for the social aspect
Takeaways:
- 17% of participants consider that agents improved their experience. Three times as many participants consider, on the contrary, that AI agents hurt their experience, and 30% don’t seem to have a strong opinion on the topic.
- Three out of four participants consider that learning deep domain knowledge is more important during the CTF than learning how to use AI, while 12% consider that learning industry tools is more important.
How much more/less likely are you to keep coming to NorthSec if we implement this change?
Transition to a no-AI-agent CTF (honors-based)
Net: +39.6%
Allow AI agents only on Sunday (honors-based)
Net: -9.7%
Remove the scoreboard and prize money for top teams
Net: -13%
Having two scoreboards where one of them is no-AI-agents
Net: +49.1%
Lean more into the community aspect of the event (more things like hacker jeopardy and social aspects, less time for CTF)
Net: -19.4%
Have much more physical challenges and much fewer web/software challenges
Net: +45.3%
Be much more adversarial to LLMs (this will make the human experience much worse)
Net: -31.8%
Force labelling of AI agent submissions (with penalty if not respected)
Net: +52.7%
Have dynamic scoring (flags solved a lot will be worth zero points)
Net: -30.6%
Takeaways:
- Participants seem strongly open to restricting certain AI uses.
- Similarly, a strong majority seems open to very clearly separating how challenges are solved (either via a separate scoreboard, or via a requirement to identify flags found by an agent).
- People love the physical tracks and are willing to have more of them at the expense of other challenge types.
- People don’t want us to make the non-AI experience worse in order to address the problems brought on by AI agents.
How do you think NorthSec should adapt to the Agentic CTF era?
This question was qualitative, and nearly 220 of you answered it! My partner and I read every response and manually analyzed and sorted them into categories. Here’s the wordcloud Claude then generated:
Takeaways:
The opinions expressed show a significant rift in the community. A majority considers that the status quo is harmful to the CTF and that we should take restrictive measures against AI use. Conversely, a non-negligible minority believes that, if adaptation is needed, it can’t come through restricting AI. It’s also worth noting that many weren’t particularly affected by agents during the CTF, even though they didn’t use them.
People in favor of restrictions consider that agents hurt their CTF experience (through a decrease in mutual help and the quality of exchanges between participants, through a decrease in the perceived value of the scoreboard, and a feeling of “pay to win”). They’re also worried about the event’s long-term sustainability, for example through an exodus of challenge designers.
On the other side, opinions advocating for unrestricted AI use mention a desire for the CTF to stay aligned with the tools used by real attackers, and a motivation to learn how to properly use this emerging technology. They also fear that NorthSec would reduce its own relevance by avoiding a technology “that’s here to stay.”
Splitting the CTF into two scoreboards is the most popular proposal. Some go as far as suggesting differentiated scoring (where, for example, points for flags on the AI scoreboard decrease over time) or having a separate series of “AI-first” challenges (challenges measuring the ability to use AI agents). Some propose requiring the use of an internal AI in order to level the playing field for everyone and remove the “pay to win” aspect of AI.
Among those who want a complete ban on agents, two trends emerge. Many think a simple honor-based commitment and respect for the event is enough. Others want more systematic mechanisms to enforce a ban, for example: requiring an explanation of how challenges were solved, blocking certain agent domains at the network level, or actively hindering their use through adversarial techniques.
Despite this rift, we can identify a few near-consensus points. Participants love the physical tracks and want more of them. Respondents also seem to agree that “autosolver”-type agents, if allowed, should be in a separate category. Finally, almost everyone agrees that some uses of AI are beneficial, whether for learning the basics of certain challenge types, looking up information/documentation, or, to a lesser extent, writing code.
What’s next
The CTF data will serve as a benchmark on AI’s capabilities and its impact on the competition. The feedback survey data will serve as a pulse check on the community’s stance on agentic use in the CTF.
Next steps: the CTF team is consulting internally. We’ll debate. We’ll make decisions. Finally, we’ll announce what’s changing for NorthSec CTF 2027.