Imagine an AI agent reviewing the source code of a hypothetical customer portal. It reports that an authenticated customer could read someone else’s documents. That would be serious. Before fixing anything, the team needs to establish whether the path is reachable and which protective checks apply.
My whitepaper “Security Testing with AI Agents” develops that process from the first suspicion through a separate counter-check to remediation and retest. If you are responsible for development, architecture or application security, it offers a way to incorporate agents into your existing review work.
What the agent contributes to the investigation
A skill supplies instructions for a coding agent. The actual investigation still needs source code, suitable context and executable analysis tools. Trail of Bits provides structured workflows; CodeQL and Semgrep perform repeatable source-code analyses. A separate counter-check examines whether a reported flaw stands up to the available evidence. Active tools such as Strix are candidates for a specifically authorized trial.
My recommendation is to start with one repository and a clearly bounded security question. Record the revision, confirm the operating context and follow a suitable case through correction and retest. This makes it possible to assess whether additional confirmed insights emerge and whether the team can address them at a reasonable cost.
The instructions follow this investigation step by step. They explain which code and context to provide, how to run the tools, which result files to expect and how to interpret them. The agent’s own permissions and data paths are part of the review.
What the exercises actually show
The exercises use artificial code with known flaws. They show, among other things, that existing tests can stay green after authorization checks are removed, and that a skill scanner’s overall recommendation does not replace its individual findings.
Not all 18 workflows were fully tested. Twelve were executed within their stated scope, three partially; two remained blocked and one started run failed technically. Strix and Shannon were not actively run. Complete native installation and execution of all plugins is not established either. These distinctions appear alongside the relevant methods. The exercise does not establish detection performance for unknown production flaws.
Read the whitepaper and supporting material
Read the full whitepaper online · Download PDF · Subscribe to the newsletter
The separate full practice guide (German) and test report (German) provide the commands, artifacts and execution records. The supporting artifact ZIP contains the recorded test sources, reports and saved outputs, with documentation in German. It is not a configured test environment.
Documentation checked: October 28, 2026. Recorded tool runs: October 29, 2026. Illustration credits and font licenses.