AI is already changing penetration testing. Modern tools can enumerate attack surfaces in minutes, analyze APIs at scale, map cloud permissions, and continuously scan environments in ways that were impossible just a few years ago. Claims that “AI pentesting replaces humans” sound convincing on the surface.

In practice, that isn’t what’s happening.

AI is making penetration testers more effective, not obsolete.

The reason is simple: penetration testing is not just about finding vulnerabilities, it’s about understanding systems, assumptions, and human decisions.

AI Is Getting Very Good — The Hack the Box CTF Proved That

In a recent experiment by Hack The Box – https://www.hackthebox.com/blog/ai-vs-human-ctf-hack-the-box-results autonomous AI agents competed directly against hundreds of human teams in a live Capture the Flag. The results were impressive. Several AI teams solved nearly all challenges and performed at a level comparable to strong human players.

This confirmed something many practitioners already suspected:
AI is extremely good at well-scoped, well-defined problems — cryptography challenges, reverse engineering tasks, and logic-heavy puzzles with clear success conditions.

But CTFs are still controlled environments.

Real penetration tests are not.

Practitioner-Level Reality: Where AI Starts to Struggle

To understand how close AI really is to replacing human testers, we ran practical experiments using public, non–pentest-trained LLMs, rather than purpose-built security agents.

One example was a practitioner-level PortSwigger SQL injection lab, involving:

• input filtering,
• SQL injection via XML,
• and bypassing filters through XML encoding.

We tested this using Haiku 4.5, accessed via Claude, integrated through an MCP server with Burp Suite.

The outcome was telling.

 

The model eventually solved the lab, but not cleanly, and not confidently. It required repeated nudging, backtracking, and trial-and-error suggestions. It frequently proposed payloads that looked plausible but were syntactically incorrect or ineffective against the specific filter logic.

Importantly, this wasn’t a failure of the lab, and it wasn’t a “bad” result for a general-purpose model. It demonstrated that the model could reason, but not reliably operate like a penetration tester.

A human tester, familiar with PortSwigger-style filters and XML parsing quirks, typically identifies the bypass pattern far earlier by recognizing intent rather than brute-forcing syntax.

We saw the same pattern on medium-difficulty Hack The Box machines.

In one case, the model failed to solve a box that most intermediate human testers would consider straightforward. It identified the correct attack surface but then repeatedly guessed exploitation paths that were technically invalid or irrelevant to the environment.

This is a critical point: AI sometimes fills gaps by guessing.

Humans do too, but humans validate assumptions continuously. AI models, especially general-purpose ones, can sound confident while being wrong. In penetration testing, confident but incorrect guidance can waste hours or lead testing down dead ends.

This inconsistency is one of the biggest reasons AI cannot yet replace human pentesters.

Here’s another example: Practical Insecure Deserialization in the Real World

In real penetration tests, insecure deserialization is rarely obvious. There’s no loud error, no crash, and no obvious exploit path at first glance. Instead, it often hides inside requests that look completely legitimate.

A request might look like this:

 

To an automated tool, or a general-purpose AI model, this often appears as nothing more than base64-encoded data. If the application responds normally, the signal is easy to miss.

A human pentester sees something different.

The rO0AB prefix is immediately recognizable as Java serialized object data. That alone raises critical questions: why is serialized data coming from the client, and is the server blindly deserializing it?

To test those assumptions, the tester modifies the payload slightly:
 

The response still returns a 200 OK, but the behavior changes. Response timing shifts. The application log volume increases. Maybe a downstream service reacts differently.

Those signals matter.

At this point, exploitation isn’t about dropping a gadget chain. It’s about understanding how objects are processed, what validation exists, and whether unsafe classes are reachable. That requires experience and environmental awareness, not just payload generation.

This is where AI often struggles. A model may suggest generic deserialization payloads or conclude the input is “likely safe” when exploitation isn’t immediate. A human tester keeps probing and correlating behavior across requests and environments.

That difference, between assuming safety and validating assumptions, is why insecure deserialization vulnerabilities are still far more likely to be found by humans than by automated systems.

This is why vulnerabilities like insecure deserialization still depend heavily on human testers who can recognize subtle signals and validate assumptions, rather than automated systems that look only for obvious failures.

Interpretation Is Still a Human Skill

AI excels at speed and coverage. It can crawl thousands of endpoints, parse schemas, and correlate large volumes of data far faster than any human. That makes it excellent for reconnaissance, enumeration, and baseline testing.

Where it falls short is interpretation.

A human pentester doesn’t stop at “this endpoint returns data”. They ask why it exists, what business process it supports, and what happens when it’s combined with other behaviors that look harmless in isolation. Many real-world compromises don’t come from a single critical bug, but from chaining smaller weaknesses together, something that still relies heavily on intuition and experience.

Exploitation is also rarely clean or predictable. Authentication flows break in unexpected ways. APIs behave differently across environments. Rate limiting, logging, and defensive controls change dynamically during testing. Human testers adapt tactics in real time, slow down, pivot tools, and treat defensive responses as signal rather than failure.

AI generally follows a reasoning path. Humans improvise.

Business Logic, Cloud, and Identity: Still Human Territory

Business logic flaws remain one of the hardest classes of issues for AI. These vulnerabilities live in workflows, role assumptions, and edge cases, not in obvious code smells. Identifying them requires understanding how a system should behave and deliberately breaking those expectations.

The same applies to cloud and identity testing. Many modern breaches stem from:

• over-privileged service accounts,
• forgotten OAuth applications,
• stale trust relationships.

AI can enumerate permissions. Humans question why those permissions exist, who uses them, and what happens if they leak. That questioning mindset is still not something public LLMs consistently demonstrate.

Even after exploitation, the human role doesn’t end.

Penetration testing is also about communication: translating technical findings into real business impact, explaining realistic attack paths, and proposing remediation that actually fits the organization. AI can generate reports, but judgment — what matters, what doesn’t, and how to explain it, is still a human responsibility.

The Reality

AI is close. Closer than many expected.

The Hack The Box CTF proved that AI can compete with humans in structured offensive challenges. Our own experiments showed that even non–pentest-trained public models can eventually solve practitioner-level labs.

But “eventually” and “reliably” are very different things.

AI today:

• accelerates reconnaissance,
• helps explore ideas,
• and removes repetitive work.

Humans still:

• recognize patterns faster,
• avoid false paths,
• adapt under uncertainty,
• and understand intent.

AI will not replace penetration testers anytime soon.

However, penetration testers who know how to use AI effectively will absolutely outperform those who don’t.

The future of penetration testing isn’t AI versus humans.

It’s AI making human testers faster, sharper, and more effective, while humans remain in control of judgment and strategy.

 

BACK TO BLOGS