- AWS describes a gate that runs a scoped penetration test on each deploy and can block promotion to production above a chosen severity.
- AWS's own guidance shows the gate can pass code unchecked: report-only runs advance the baseline, and infrastructure errors fail open by default.
- Before relying on it, confirm the failure settings, who can override pipeline variables, and which commits earlier runs already cleared.
A penetration test used to be a date on a calendar. A specialist team arrived, probed the application, and left a report. Code shipped in between, and nobody had confirmed whether it could be exploited.
AWS now wants that test to be a step in the pipeline. The idea is sound. The harder question is what happens when the step is set up loosely. The lesson here is that a security gate is only as strict as its settings, and the settings are where the risk now sits.
What AWS describes
In a post on its Security Blog, AWS describes AWS Continuum for penetration testing, formerly called AWS Security Agent. AWS says the service uses AI agents to find and confirm exploitable vulnerabilities in running applications.
The post shows how to make it a post-deployment gate. The pipeline deploys to staging and runs the test. It then promotes to production only if the result is clean. AWS says the pattern works on GitHub Actions, GitLab CI/CD, Bitbucket Pipelines and Azure DevOps Pipelines.
How the gate works
Common pipeline tools such as static analysis and dynamic scanners flag patterns. AWS notes that a flagged pattern shows where to look, while a penetration test confirms whether something is exploitable. The gate aims to supply that confirmation on every deploy.
To keep each run fast, the service tests only the change. The head is the commit being gated. The base is the previous commit the gate already cleared. AWS says this is what makes a per-deploy test fast and cheap enough for a pipeline. That is AWS's claim, and the post gives no timings or prices.
Each run ends in one of three outcomes: in scope, scoped out, or scope conflict. A scope conflict means the change fell outside the configured scope and was not tested. The customer decides whether that blocks.
The pipeline signs in through OpenID Connect. A short-lived token from the CI/CD provider is exchanged for temporary AWS credentials, so no long-lived access keys sit in pipeline variables. The testing runs inside the customer's AWS account, not on the CI/CD runner.
Where the gate can loosen
The post is candid about the traps. Three independent settings decide whether the step fails: one for findings, one for untested changes, and one for jobs that could not run. Relaxing one does not change the others.
First, the staged rollout. AWS suggests starting in report-only mode, then tightening the threshold to HIGH. But a passing run, including a report-only one, advances the baseline. Those commits are recorded as cleared and are not re-tested later.
So a month of report-only runs can leave a month of code outside the stricter gate. To reassess a range, teams must override the base and head commit SHAs.
Second, the infrastructure default. The setting fail-on-infrastructure-error ships as false, which AWS calls fail-open. If the job cannot run, the pipeline can still proceed. AWS advises setting it to true wherever promotion needs security sign-off.
Third, GitLab. A pipeline variable overrides what a job sets in its own file. Anyone who can run a pipeline could lower the threshold, switch on dry-run or relax error handling. The gate would follow the variable, not the committed file.
Credential timing adds a fourth trap. If the temporary credentials expire mid-test, a healthy run can fail as though the service were at fault. The cleanup step may then be unable to stop the abandoned job.
Limits of the source
This is AWS describing its own product. The post does not report how many real flaws the agents find or miss. It does not say how often scoping to a change misses a flaw that sits in older, unchanged code. Leaders should treat the design as sound but unproven by this source.
Questions for your team
Is fail-on-infrastructure-error set to true on any pipeline that gates a release? If not, who accepted that a broken test run lets code through?
If we started in report-only mode, which commits did those runs mark as cleared? Do we need to override the base and head commits to re-test them?
Who can override pipeline variables in our projects? Is the gate role's trust policy pinned to the narrowest branch or environment we actually deploy from?
Is the integration pinned to a fixed version, so an upstream change is reviewed before it runs?
A pentest on every deploy replaces a calendar problem with a configuration problem. The control is only as strong as the settings someone chose on day one.
Produced by the WebPulse Newsroom with AI assistance from the original reporting credited below, and checked against that source by our editorial review. How we use AI.
Original reporting: AWS Security.
