Security Testing
with AI Agents

From suspected unauthorized access to a review brief

A customer opens a document in a web portal. The application checks the login, accepts a document number and returns the file. An AI agent examines the relevant source code for security flaws and reports that a logged-in customer could use a different document number to access another customer's files.

That would be a serious vulnerability. At first, it is a claim with several prerequisites. Can the customer choose the number freely? Does the request reach the function in question? Does an upstream service already check who owns the document? And might the database prevent access somewhere outside the first code excerpt?

This whitepaper uses that fictional example to develop an application security review process. If you are responsible for development, architecture or security, it should help you assess what agents can contribute and prepare an initial deployment. The investigation must move from suspicion through sound evidence to an accountable decision and follow-up. That requires repeatable analyses, a separate challenge of the findings and, where necessary, controlled tests.

The portal already presents two distinct questions. Authentication establishes who a user is. Authorization determines what that user may do with a particular object. Login can work correctly while the check on an individual document is missing. These flaws arise from interactions within the application and often cannot be assessed from one code excerpt.

The proposal below therefore connects tools, context and human responsibility. It concerns agent skills for coding agents, not penetration-testing certifications. Domain and technical details appear alongside the questions they explain. Observations from an exercise using artificial test code accompany the relevant investigation steps. They show which result files are produced and where execution reaches its limits. The selection and adoption plan are my recommendations; a pilot—a bounded trial—must first establish their effectiveness in your target system. Operational monitoring and response to real security incidents must remain connected to the teams responsible for them.

How a coding agent works and which tools support it

A coding agent combines a language model with access to files and tools. The model interprets the assignment and proposes next steps. The agent software executes permitted tool calls, processes their results and continues the work. It can read a function, find its callers and then launch an analysis. Codex, Claude Code and Cursor are possible working environments.

The working copy is the directory of source code and configuration provided for the investigation. The findings register is an access-controlled record of cases shared by the review team, for example in a ticketing system. It links claims, evidence, status, owners and treatment.

A skill provides task instructions, sometimes with scripts and other resources. A plugin bundles extensions for a particular agent environment. Trail of Bits Security Skills is a published collection of such security workflows. Figure 1 shows how these instructions, the model and executable tools work together. Trail of Bits

The language model processes the approved context at the selected model provider or in a self-hosted deployment. CodeQL and Semgrep are separate analysis programs in the review environment. They need source files and configuration and save analysis results.

The agent therefore uses a skill's instructions and the outputs of executable tools. Installing a skill does not automatically set up those programs; their installation must be checked separately.

Practical context

Prioritizing tools for an initial deployment

The following ranking orders the initial selection. It is a professional recommendation, not a standardized performance comparison. Project descriptions establish documented functionality, not measured detection performance in your system.

For a first deployment, I recommend the static core of static-analysis and fp-check, supplemented by context building, variant analysis and change review. Strix follows for specifically authorized active tests. Before this selection can do useful work, the team needs a defined subject and a bounded assignment.

Before the agent searches, it needs a brief

A repository is the shared store of source code and its change history. A revision, usually identified by a Git commit, fixes the code version under investigation. The relevant operational configuration is documented separately. Without this baseline, developers may already have changed the flaw while the agent is still explaining an older version.

For the portal, the team also needs to know which services accept requests, where customer identities are established and who enforces document permissions. An application programming interface, or API, is a defined entry point through which another program calls a function. Its reachability changes the possible attack paths.

A threat model organizes this information: what needs protection, who might attack it, through which entry points and with what capabilities? In the example, confidential customer documents are the asset to protect. A trust boundary exists where data or actions are accepted from a context with different permissions. At the customer-to-portal boundary, the application must enforce which documents belong to the customer. A supplied document number does not establish ownership.

OpenAI's security-threat-model skill supports this assessment from the source code. The application team confirms the operational context because the repository cannot reliably show all actual network exposure, identity services and protective measures. Systems that build and test code are also attack surfaces with their own permissions. Threat-modeling skill

Your team names the responsible owners before the run. The application owner confirms purpose, operation and review scope. The application security function, often called AppSec, assesses cases and maintains the findings register. Platform owners configure access and execution boundaries. In small teams, one person may perform several roles; their decisions remain traceable.

Practical context

What was actually tested in the portal example

To explain the review process through concrete outputs, a small test fixture was examined. It contains two artificial customers, each with one document. The ownership check was deliberately removed from two retrieval functions; a third retains it. For a document assistant, a tool description and its executing function, the handler, extend the example. The baseline revision, the faulty change and a corrected copy are recorded separately.

For the conditional risk assessment, the requester also confirmed the operating assumption of an internet-facing portal containing confidential customer documents. No such portal was built or attacked. The actual function calls used artificial data only; a complete server for the document assistant was also outside the fixture. References below to the exercise have this limited scope. The fictional operating scenario illustrates broader decisions that the local exercise cannot substantiate.

The flaws and expected responses were known. The exercise covered the 18 workflows and tools specifically named in this whitepaper, not everything in their linked collections. It shows observed results, adaptations and missing prerequisites. It does not measure detection performance against unknown vulnerabilities.

Not all 18 entries were fully tested. After the follow-up runs, twelve had been executed within their stated scope and three partially; two remained blocked and one run failed technically. Executed steps include agent methods and adapted workflows. This does not establish native installation and complete execution of all Codex or Claude plugins.

These statuses describe execution: “executed” means the named review step took place and produced an artifact; “partial” identifies a missing stage; “blocked” identifies an unmet prerequisite; and “failed” describes a run that started but ended in a technical failure. None of these statuses, on its own, is a judgment about the application's security.

A result artifact is a durably saved file that makes an investigation step traceable. Artifacts include agent reports, native scanner files written by the tool itself, and execution logs. Their purposes differ: a findings report explains a suspicion; an execution log shows whether the investigation completed. The following chapters explain these files alongside the methods that produce them. Relative file paths refer to the linked test archive. Its test report (German) contains the complete mapping, including source revisions, commands and original outputs; custom report names are conventions of this exercise.

The first such result is runs/threat-model/fixture-threat-model.md. Created using security-threat-model, it connects components, a data-flow diagram, assets and assumptions to three threats, their conditional priorities and mitigations. It also records which assessment applies only under the confirmed assumption of internet exposure. This makes the basis for the subsequent review brief visible.

Instructions for your own work appear in blue boxes labeled “Practice guide,” directly alongside the relevant method. The additional label distinguishes an agent assignment, a tool command and setup. Observed files, results and limitations are explicitly described in the surrounding text. An expected artifact in a box is not yet a produced test result. The practice guide for all 18 entries (German) additionally brings together prerequisites, assignments, commands, success criteria and remaining limitations. Unless a section specifies another environment, the shell examples assume Bash/WSL after the relevant tool has been installed, with the current directory set to the root of a disposable copy of the test archive. SkillSpector explicitly uses the tested Windows Python entry point. eigener-lauf/—“own run”—is a new output directory, preserving the recorded originals.

Practical context

Defining the first run precisely

This assignment initially investigates document retrieval. Other review areas require explicit assignments and appropriate methods. Reports and analysis data may be created outside the protected application working copy. Builds—the preparation of source code and dependencies for execution—require separately bounded permission to execute. “Read-only” describes a permission that must be technically enforced. The analysis begins at this boundary.

Connecting source code, data flow and scanner results

The review team follows the document number from the request to file delivery and looks for the effective authorization check. This path through variables, functions and services is called data flow. The sequence of function calls forms the call chain. Both views are needed: a checked variable is of little use if the application later uses a different value.

Scanners automate checks on source code. This static application security testing, or SAST, does not require an attack on the running target application. CodeQL prepares code in an analysis database and examines it using queries. Semgrep uses rules to search for suspicious patterns and can also analyze data flow, depending on the rule, language and edition. Taint analysis traces whether an attacker-influenced input reaches a security-sensitive use and which checks lie between them. Rules and data-flow models largely determine what becomes visible. CodeQL, Semgrep

A rule might flag an unchecked document number. Whether another component protects access remains a context question. The agent reads related functions and their callers and develops testable hypotheses. It can investigate relationships that existing rules do not express. It can also misunderstand protective functions or miss paths. The scanner baseline and agent investigation complement each other; both need visible coverage.

Technical detail

The seven Trail of Bits plugins

A SKILL.md contains task instructions; some plugins use other entry points. In the examined revision, for example, insecure-defaults consists of a command file and a workflow. For an initial deployment, I use the priority ratings “essential,” “very high” and “high.” These prioritize general enterprise codebases; the specific task determines their use.

Sources: static analysis, context building, variants, rule development, insecure defaults.

For an initial full audit, the proposed sequence is audit-context-building → semgrep → codeql → fp-check → variant-analysis. Context comes first; variants follow from understood starting cases. This is a practical choice, not a prescribed sequence for every repository.

The exercise shows what this context looks like in practice: audit-context-building left a DOSSIER.md, six individual function descriptions under functions/, and execution.json in runs/context/audit-context/. Six separate Codex function agents were used; this workflow replaced the native Claude workflow API. The dossier records assumptions and data flows without yet issuing a security judgment. The Trail of Bits workflows used are pinned to specific revisions.

Technical detail

Configuring scanners and reading their outputs

For CodeQL, record the language, database extraction, build mode and query suite. For compiled languages, the build can determine which files enter the database. Project-specific inputs, sensitive operations and protective functions may require additional models. Extraction and build diagnostics, together with excluded areas, belong in the execution report. Successful database creation or zero findings does not establish a complete investigation. Build modes

For Semgrep, document the engine, edition, language and rule packs. Community and Pro capabilities differ particularly across function and file boundaries; support depends on rule type and language. Failed, skipped, partial or ineffective runs for the relevant code must be identified. --metrics=off does not replace a review of other network and platform features. Product boundaries, Semgrep skill

The exercise made this distinction visible. Semgrep created six native JSON/SARIF pairs under raw/ in runs/semgrep/; rulesets.json describes the rule selection, scans.json the execution status, and results/results.sarif the merged results. The baseline rules reported no matches. However, two of the six rule-pack runs were incomplete because of rule problems, and Pro was unavailable. The files therefore document analyses that took place and their limits. The known authorization gap went unreported in this baseline.

CodeQL was missing during the initial run. CodeQL 2.27.3 was subsequently installed within the project and demonstrated on a separate public MIT-licensed regression example. The outputs under runs/followup/codeql/ include codeql.db/, build.log, quality diagnostics, the explicit suite raw/demo.qls and raw/results.sarif. The database check recorded one project file with 44 baseline lines of code and zero extraction errors. The single query reported 13 locations marked in the example; upstream explicitly labels one of them SPURIOUS, meaning a false alarm.

This is a native tool demonstration, not a complete CodeQL skill test. The broader suite-selection and modeling sequence remained untested; CodeQL did not analyze the portal code. The public example file was analyzed only, never executed. The successful run therefore also provides a concrete reason for the subsequent challenge of findings.

The CodeQL CLI has its own terms of use; open queries do not imply unrestricted use of the engine. Appropriate entitlement must be established for private commercial repositories. Semgrep's engine, rules and products have different terms. Check the selection against existing agreements before the pilot. CodeQL CLI, Semgrep rules terms

Codex Security is an additional investigation route for semantic questions such as authorization, trust boundaries and control flow. The local plugin workflow documents scans without changes to application source code, finding and coverage artifacts, and source-code validation even without a reproducer. Model processing and its data path still require separate review; a local launch does not guarantee exclusively local processing. Cloud scans of connected GitHub repositories are a separate operating route. Plugin quickstart, cloud overview

The plugin's Deep scan combines several complete standard runs performed by parallel review workers and aggregates completed outputs. It does not support diff targets; a separate change review exists for those. Exclusions, limitations and successful finalization must be checked here too. More workers do not establish complete coverage. Deep scans

The Codex Security standard run produced a third kind of outcome: input validation, authentication and the dry run succeeded; the real scan then failed technically during preflight. The logs name a native-process failure and timeouts in a technical helper used to prepare requests. runs/codex-security/ contains version data, execution metadata and error JSON; the intended final output directory lacks report.md, findings.json and coverage.json. Diff and Deep were not started afterward. A successful dry run checks the startup path but does not replace a completed scan. The CLI documentation distinguishes those steps accordingly.

All these routes need to describe a case so that another reviewer can examine it. For the portal, the core claim is: “The document number comes from the customer; retrieval uses it without a visible comparison to the customer's identifier. Whether an upstream service prevents access remains unresolved.” Such a case is often called a finding. Its register entry connects code locations, context and scanner runs. Multiple reports of the same cause are consolidated into one case; their distinct code locations and evidence are retained. The next step is to challenge that case.

A good counter-check tries to disprove the suspicion

The next task is to seek the best evidenced explanation for why the flaw might not exist. In the portal, a central check before every document access or a database rule might filter out other customers' documents. A function name such as checkAccess is not evidence of that protection.

fp-check examines existing candidates, including the attacker model, callers and relevant data flow. The skill requires a reasoned verdict of True Positive for a substantiated finding or False Positive for a refuted report. If a required prerequisite cannot be established, the overarching process keeps the case open as unresolved. This is not an additional final status of the skill. fp-check

The counter-check receives its own assignment, reconstructs the decisive steps and documents its attempts to disprove the claim. Two roles using the same model and the same incorrect assumptions can repeat the same mistake. For consequential or disputed cases, a knowledgeable person supplements the review. Separate role names do not create a technical security boundary either.

In the exercise, fp-check received three predefined claims about the retrieval functions. Two were substantiated within the examined function contract; the claim against the safe control was refuted. runs/fp-check/verdicts.md connects data flow, protective conditions, counterarguments and observed calls for each verdict. This also makes a rejected report traceable. It still does not establish reachability through a real login or network-facing portal.

Complete code and context evidence can confirm a vulnerability without executing an attack. A reproducer, a small test that recreates the flaw under recorded conditions, adds observations of actual behavior. A technically failed test, by contrast, does not settle whether the vulnerability exists. Figure 2 shows which decision follows from the counter-check and when more information is needed.

Practical context

Record findings, test execution and treatment separately

The register entry identifies repository and revision, file and code location, tool and rule versions, skill revision and model configuration. It also records the asset, attack path, prerequisites, possible impact, counterevidence and open assumptions. Tests link to test accounts and sanitized logs; treatment records an owner and deadline. Access keys and real customer documents do not belong in a widely distributed report.

“Confirmed; reproducer not executed” can therefore be a correct outcome. If it remains unclear whether an upstream service prevents cross-customer access in the portal, the next test addresses exactly that assumption.

Active runtime tests for unresolved security questions

A penetration test uses authorized attack actions to investigate whether protective measures can be overcome. A proof of concept, or PoC, demonstrates a specific attack under particular conditions. Its conclusion applies to the tested state.

For the portal, one test customer receives a document; a second tries to retrieve it. The expected behavior is defined beforehand: access to one's own document succeeds, access to another customer's document is rejected. A failed login for the second customer would be a test-setup problem; a correctly authenticated customer receiving another customer's document would be the observed protection failure. The environment must represent the mechanism in question sufficiently well, or the test will not answer the open operational question.

An active test may modify files, create data, trigger messages or overload services. It therefore starts in an explicitly approved staging environment, a separate test deployment sufficiently representative of the intended operation. The application owner and AppSec approve targets, test accounts and actions. The platform establishes limits and a technical shutdown mechanism; a person who can supervise and stop the run is named. Afterward, test data is removed and the agreed initial state restored.

Technical detail

Active tests, tools and execution limits

Strix is my preferred candidate for this addition. Its self-hosted CLI requires, among other things, Docker and access to a supported model. Docker provides containers for test tools; a container isolates the processes running inside it. Local target directories are mounted with write access, so use a disposable working copy. Cloud code reviews may transfer the source tree. Operating mode, reachable targets and data paths are therefore part of the selection. Strix, Strix skill

For automated Strix runs, read exit codes—the program's return values—alongside the report: 0 means no validated finding was reported at the configured threshold, 1 means an execution error, and 2 means findings at that threshold. Document the cost limit --max-budget, agent-step limit --max-turns and reporting threshold --fail-on. Even completed can describe a shortened run after a budget warning. Costs, warnings and the areas actually examined determine what the result supports.

Shannon also connects source-code analysis and active validation. In exploit mode, the executed attack provides the evidence; in analysis mode, exploit: "false", severity and evidentiary confidence are assessed. If the additional agentic static analysis fails, the penetration test can continue without its results and finish partially. Unreported or unreviewed candidates remain in the register. The freely available open-source edition (OSS), under the AGPL license, and commercial offerings differ in terms and functionality. Shannon, configuration and partial failures

For both tools, the exercise stopped at a technical prerequisite: the Docker daemon was unavailable. Strix and Shannon were therefore not executed; no active attack evidence was produced. Strix Cloud would have been a separate operating route requiring configured access and an engagement; it was not used. The files below are documented expectations from pinned sources, not existing execution artifacts.

Sources for expected outputs: Strix, Shannon. Direct Python calls to the artificial portal do not replace either product's runs.

Fuzzing suits other questions: a fuzzer generates or mutates many inputs for a bounded function and observes crashes, timeouts or violated properties. A harness makes the function executable for testing. A crash is initially a reason to investigate, not a finished exploit demonstration.

The Trail of Bits testing-handbook-skills family supports tools such as libFuzzer, AFL++ and harness development. Language and runtime determine the selection. For native C/C++ code, compiler instrumentation and memory checks add measurement and checking mechanisms. Seed inputs, time and memory limits, and the code areas under examination belong in the campaign. Fuzzing is dynamic; for the unresolved document permission, the targeted two-customer test is closer to the question. Testing Handbook skills, libFuzzer

The exercise produced runs/fuzzing/harness.cc, which limits the artificial number parser to 32 bytes. After the initially missing compiler was added, it was actually compiled using Clang 14 and for AFL++. Build logs, inputs and binaries are stored under runs/followup/fuzzing/.

The actual run processed 20,000 inputs. Its log reports 13 coverage counters and 44 features; these instrumentation values are not a percentage of total application coverage. No parser crash was observed. A separate, deliberately crashing control function ended as expected with exit code 77. It checks the setup's failure observation and is not a flaw discovered in the parser. Corpus, build logs and execution logs are stored under runs/followup/fuzzing/libfuzzer/.

For aflpp, build the harness using an instrumenting compiler and a suitable program entry point. In the exercise, the first Clang wrapper failed on a compiler option. afl-g++ then worked with the saved driver.cc, which accepts inputs from standard input.

This example command also starts with one seed. The saved AFL campaign used two seeds, numeric and malformed. It recorded 6,444 executions and ten corpus entries in 15 seconds; a repeat need not reproduce these time-dependent values. runs/followup/fuzzing/afl/ contains the build log, run.log, out/default/queue/ and fuzzer_stats; no crashes or hangs were saved.

The AFL crash precheck was bypassed using an environment option without changing kernel settings. Reliable capture of AFL crashes was not positively controlled; the separate crash control belongs to libFuzzer. This limitation is part of the result. For your own repeat, the practice guide provides compiler commands, seeds, resource limits and the remaining precheck. Both short campaigns examine an artificial parser; they establish neither the security of the document portal nor complete fuzzing coverage.

The reviewing agent is part of the attack surface too

The execution environment, often called a runner, receives only the access it needs. Network rules limit reachable targets; time, cost and resource limits bound the run. These controls are enforced outside the model's instructions. A container helps separate processes but does not automatically protect mounted files or reachable networks. The working copy therefore contains no valuable unsaved changes or unrelated credentials.

Figure 3 shows how the approved review scope becomes technical rules for a run. Platform owners implement them in the relevant environment.

A source-code assignment also has data paths: a local scanner may give results to a coding agent that sends them to a model provider. Analysis databases, build logs, reports and reproducers may contain confidential details. Transmission, access and retention of these artifacts are agreed before use. “Started locally” does not answer where they end up.

Third-party skills add another attack surface. Prompt injection attempts to change an agent's operating instructions through the content being examined. Exfiltration means the unauthorized removal of data. Before installation, therefore, review origin, specific revision, executable components, dependencies, permissions and data paths. This includes installers.

NVIDIA SkillSpector supports investigation of suspicious skill patterns through static and optional model analysis. Also examine the providers, endpoints and possible network requests of those analyses. Its output provides signals, not a certificate of safety. The environment remains responsible for limiting permissions. SkillSpector

Static testing of two custom test skills showed how much a report summary can conceal. The benign instructions produced no finding. The second file contained an artificial prompt injection requesting a concealed credential upload. This instruction was only read and scanned, never executed. SkillSpector reported one HIGH individual finding, yet also an overall score of 20, overall severity LOW and recommendation SAFE.

Native JSON, Markdown and SARIF reports for both texts were produced under runs/skillspector/. On Windows, this required capturing standard output, the program's terminal output channel; direct file output was rejected by the program. The working agent subsequently reviewed the skill texts and reports separately and chose REJECT for the harmful text. The verdict is in inspector-verdict.md; the product's LLM analysis remained disabled. The Skill Inspector specifies both review steps. This separate challenge shows why an installation decision should not follow from the overall score alone.

If the application being examined contains an agent itself, that agent's tool access becomes a review subject in its own right. To illustrate this, imagine extending the portal with a document assistant. The Model Context Protocol, or MCP, can connect it to tools and data sources. Tool poisoning uses manipulated tool descriptions or results to steer its behavior. An innocuous description says little about the action actually executed.

The mcp-server-security-review skill helps examine tool handlers, identity checks, object and action permissions, inputs and dangerous side effects in the actual server. Document retrieval must enforce authorization itself; a customer identity supplied by the agent is insufficient. The skill is a review guide; code and protocol requirements remain authoritative. MCP review

The synthetic tool handler chose exactly the wrong source of authority: it compared document ownership with a customer identifier supplied by the model and did not use the independent identity parameter. runs/mcp-review/report.md documents the tool inventory, data flow, conditional finding, proposed fix and review gaps. The review was executed within the scope of this handler. A complete MCP server, its transport, OAuth and runtime input-schema validation remained unassessed; no active server attack took place. The protocol details below therefore describe requirements for a real deployment that the handler test could not examine.

Technical detail

MCP authorization for HTTP and stdio

HTTP-based MCP services that implement authorization are subject to the corresponding OAuth requirements. OAuth governs delegated access. An access token is the credential presented with a request; the server must verify that the token is intended for it. Passing through a third party's token without validating it is prohibited. An OAuth scope limits the permissions available to the client, but it does not replace checking permission for the specific document and action.

Local stdio connections communicate through a process's standard input and output; the same HTTP authorization flow does not apply to them by default. Here, controls include limiting process permissions and credential provisioning. The action must be checked for permission in both cases. Authorization, security best practices

From a confirmed finding to a verified correction

To continue the teaching example, assume the application team has confirmed from code and operational documentation that the upstream service checks only login and that no further effective ownership check exists on the reachable document-retrieval path. This assumption leads to the remediation recommendation in the example. The exercise did not investigate a real service chain. In a real investigation, the corresponding evidence must be in the register before the team assesses the case in this way.

AppSec justifies severity using possible harm, reachability, required attacker capabilities and existing protective measures. Severity and evidentiary confidence remain separate fields. A responsible risk owner decides on any temporary acceptance, with a rationale, compensating controls where appropriate and a review date. An unresolved case instead receives an assignment to establish the missing facts.

The authorization check is added so that all relevant retrieval paths actually pass through it. An agent can propose the change; the development team reviews it through the normal process. A retest checks the same access path against the corrected revision. A regression test records that access to one's own document still works and access to another customer's document is rejected.

This bounded path can actually be traced for the test code. Eighteen direct function calls across the faulty and corrected copies checked own, cross-customer and anonymous access. They showed two cross-customer access violations in the faulty version and rejection after correction. Observations are stored in runs/unit/cross-customer-and-retest.json. This confirms the behavior of the examined functions; it did not test integration with real authentication or an upstream service.

Practical context

Finding status and closure in the portal example

In fictional register case DEMO-01, the status before remediation reads: “Confirmed on the basis of code and confirmed operational context; runtime test not performed.” After correction, the fix revision, renewed code review and, if appropriate, the separately approved two-customer test are linked. The case closes with the agreed completion evidence and its remaining limitations. These entries are a teaching example, not real test results.

Technical detail

Finding variants and preventing regressions

With the cause known, assign variant-analysis to other retrieval paths. In the exercise, a gradually expanded search of the program's syntactic structure, its syntax tree, found another affected export function; a protected retrieval variant was rejected with a rationale. runs/variants/search.json, variant-report.md and the reproducible script variant_search.py record search steps and controls. This is a search for variants of a known cause, not discovery of unknown flaws without a starting case.

The variant search leads to the next step: capture the known cause in a regression test.

In the exercise, semgrep-rule-creator produced a targeted regression rule. runs/rules/portal-owner-check/ contains the rule and test file, alongside the syntax tree, validation, test log and native scanner report. Three positive examples and three safe controls passed. The rule then flagged both intended return statements that the baseline rules had missed. This does not imply general superiority: it recognizes a local form of the missing comparison and does not model arbitrary upstream protective services.

The SARIF format introduced with the analysis tools connects results to rules and code locations. Operating assumptions, counter-checks and treatment remain linked to the complete register case. SARIF can carry or link some of this information but does not replace case management. SARIF 2.1.0

The meaning of these outputs must also survive further processing. In the exercise, sarif-parsing merged two reports with different code locations into one. Both carried the placeholder requires login as their fingerprint, the identifier used to recognize a report. The unmodified SARIF helper treated that placeholder as a unique identity. The original outputs were retained; a documented custom post-processing step ignores only that placeholder and uses the existing fallback identifier based on rule, path, line and message. Both locations were then retained.

runs/sarif/ contains combined.sarif, summary.json and the derived findings.json and findings.csv. These make the loss and limited correction traceable. When merging reports, therefore, also check whether code locations and evidence survive processing. A shared case may contain several affected locations.

One retrieval path has now been examined and treated. A second overview shows which questions remain open for the rest of the application.

Reviewing other functions, dependencies and configurations

For each accessible function, the team records which protective condition should apply and which method examines it. In the portal, this includes uploads, modifications and deletion of documents, exports and administrative access, alongside reading. Variants of a known cause remain distinguishable from functions that have not yet been considered.

Further reviews cover included libraries, exposed credentials and deployment. Secret scanning looks for items such as keys and passwords in the source tree and its history. Dependency review examines dependencies and their known vulnerabilities. A code audit of document retrieval does not automatically cover these tasks. Credentials, dependencies

The owner revokes or rotates a discovered credential; deleting it from a file is insufficient. A library finding is assessed against the deployed version and usage context. Container and infrastructure configurations require suitable review methods, data and assessments of their own.

Technical detail

Trace defaults through to their security impact

insecure-defaults requires an evidenced path from a suspicious default to its security impact. The test code contained a fallback key and a debug value enabled by default, neither used by any examined function. These did not become confirmed product flaws. A suspicious value alone does not establish whether an attacker can exploit it.

Technical testing of this plugin remained partial: its offline tests using predefined agent responses passed 102 checks; all 36 deliberately mutated workflow versions were detected as faulty. A separate search-pattern comparison against 18 documented examples found one example without a matching hit on the Windows host. runs/insecure-defaults/ contains test logs and report.md with adaptations and limitations. The native agent workflow did not run. These files substantiate the examined workflow sequencing and error handling, not the detection performance of a complete analysis agent.

The coverage overview connects method, owner, result and limitations for each area. It marks areas as included, partially investigated, not investigated or outside scope. “Not investigated” does not mean “no issues.” Cases enter the same register. New entry points or a different actual permission distribution also update the threat model and subsequent review assignments. This baseline leads to ongoing change review.

Integrating agent reviews into development

A diff describes the change between code versions; a pull request presents it for shared review. differential-review examines its security implications. The proposed sequence is differential-review with Semgrep/CodeQL, followed by fp-check for new candidates. It accompanies the software development lifecycle, or SDLC: the development and maintenance of the application. Context from the initial audit is checked for currency. Change review

In the exercise, differential-review connected the four removed protective lines in two functions to real Git history and unchanged existing tests. The safe baseline revision was exported without switching the working copy under investigation. runs/diff/review.md refers to diff, history and blame outputs—the mapping of code lines to changes. The three original unit tests passed despite both ownership checks having been removed. This exposes a concrete gap: the existing green tests were insufficient to preserve the protective condition across the change.

Existing continuous integration, or CI, automatically runs defined checks when code changes. During the pilot, AppSec and the application team agree whether every change or a clearly defined selection triggers agent review. For a selection, they name the functions or components and ownership of the selection rule. Reports and register cases reach the responsible team through pull requests and linked tasks. Active tests against shared staging are scheduled separately, so multiple changes do not alter the same test state without coordination. Code scanning

The effect on release approval also needs agreement. In this proposed pilot, new agent reports do not initially block a change on the strength of their text alone; existing mandatory checks continue to apply. For confirmed findings, the team defines which severity requires remediation or an accountable risk decision before approval. Open cases and technically failed mandatory checks receive explicit treatment. A missing report is not a pass.

To make these rules workable, a team must first trial the process within a bounded scope. The pilot examines both the additional insights and the route by which they reach the people responsible.

Setting up a pilot and assessing its value

The application team names a pilot owner to bring together scope, capacity and decision criteria. Domain decisions remain with the owners described above. Start with a selected repository, confirmed operational context and the bounded assignment from the brief chapter. First set up the five core plugins and scanners and check their actual permissions.

The exercise provides concrete setup questions. The fuzzers installed later enabled bounded native campaigns; CodeQL was demonstrated on a public example. The complete CodeQL skill, two Semgrep rule-pack runs and the native insecure-defaults workflow remain partially or incompletely examined. Strix and Shannon still require a working Docker service; Codex Security did not reach a final report. Decisions about your pilot rest on these individual pieces of evidence and gaps, not a general success rate.

Technical detail

Installing plugins and checking host support

Codex and Claude Code have their own plugin mechanisms. Cursor supports agent skills in locations such as .cursor/skills/<name>/SKILL.md. A portable skill file does not guarantee identical execution of all additional actions, subagents and command files on every host. On Windows, Bash-based components need a suitable environment; this does not imply that every scanner requires Windows Subsystem for Linux, or WSL. Cursor skills, Claude Code plugins

The installation record captures host and version, origin and specific package revision, plugin versions, available programs, licenses, data paths and the trial run. Linked main pages can change. The test archive therefore documents the source and tool versions actually used. It does not replace the setup record for your own agent environment or a review of the plugins loaded there.

For this exercise, instructions and tool versions were recorded separately on October 29, 2026. Semgrep 1.182.1 ran in a separate Ubuntu/WSL environment, SkillSpector 2.12.0 in its own Python environment; Codex Security CLI 0.8.1 included plugin 0.1.95. The setup also used Clang 14, AFL++ 4.00c and CodeQL 2.27.3. Compilers and CodeQL were added exclusively to the project-local test environment; origins and hashes are recorded. These versions and the adaptations described are part of the results. A different installation may have different prerequisites and outputs.

Technical detail

Organizing the brief, runs and findings

The authoritative register can remain in the ticketing system and link to these artifacts. The platform configures access; confidential evidence retains its restrictions when linked. AppSec keeps review status current, and the application team takes responsibility for remediation tasks.

Practical context

Using pilot data to decide on expansion

The comparison starts with a scanner baseline on the same revision. Agent investigation is then added. The decision uses the report, findings register, coverage overview and effort record. Track additional confirmed cases, refuted reports, open candidates and unexamined areas, as well as human review and remediation time, scanner licenses, model resources and execution resources. Agree limits and available capacity before starting; only setup and baseline measurement provide a basis for a credible schedule.

The proportion of confirmed cases among reports with a final substantive assessment describes usefulness in the pilot. It does not measure all missed flaws. Known vulnerabilities or explicitly introduced test cases help make misses visible; success on them cannot be extrapolated to unknown production flaws. Without a known, complete set of flaws, an overall detection rate cannot be determined credibly.

Where possible, at least one sufficiently substantiated case should also pass through remediation and retest. If no suitable finding exists, this stage remains unassessed or is trialed using an explicitly identified test case. Expansion makes sense when additional insights are confirmed, the counter-check remains manageable and treatment works. Otherwise, close process gaps first or narrow the purpose. Unanswered operational questions, repeatedly aborted runs and missing responsibilities receive tasks and owners.

Conclusion

At the document portal, value turns on a specific question: can the team substantiate whether a customer can retrieve someone else's documents and verify any necessary correction? An agent can reconstruct the path through code, use analysis tools and prepare a counter-check. A sound assessment rests on the brief, context and evidence; a separate retest establishes whether a correction works. Named people remain responsible for access rights, approvals and residual risks.

The exercises show why these connections matter. Existing unit tests stayed green despite ownership checks being removed. When a harmful skill instruction was examined, a high-severity individual finding appeared alongside an overall recommendation of SAFE. Anyone taking only the green test result or aggregate score loses the decisive context. Result files must therefore be read together with execution status, individual evidence and open questions.

These observations come from bounded exercises with known flaws and artificial data. They show concrete results and technical gaps; they do not establish detection performance for unknown production flaws. Not all 18 workflows were fully tested. In particular, active Strix and Shannon runs and a completed Codex Security scan remain outstanding. The documented instructions and expected artifacts help with preparation but do not replace a successfully completed run of your own.

If you adopt this approach, start with one repository, a confirmed threat model and a clearly bounded security question. My recommendation remains the static core of Trail of Bits, suitable CodeQL or Semgrep analyses and a separate counter-check using fp-check. Follow at least one suitable case through remediation and retest. If an unresolved question requires active evidence, you can trial Strix in a separately approved pilot. The test environment and its technical boundaries must be in place first.

Then decide on expansion according to whether additional confirmed insights emerge and whether the team can address them at a reasonable cost. The measure is a review process in which a plausible claim becomes a traceable decision and a verified improvement. The next extension of the security stack should be judged by that standard too.

Source dates and limits of the claims

The underlying documentation was reviewed as of October 28, 2026; test sources and tool versions were recorded separately on October 29. The portal case is synthetic. Tools actually executed include Semgrep and SkillSpector; the Codex Security run started but failed technically. Execution limitations and result files are described alongside each method. No external applications were attacked or production vulnerabilities confirmed. A pilot must test effectiveness in your own target system.

Style reference: “Enterprise Architecture für agentische KI.” Image provenance, prompts and diagram sources: CREDITS.md, image-prompts.json, diagrams/. The generated scenes provide no evidence of security effectiveness.