In early 2025, a platform team at a mid-sized SaaS company deployed a sidecar container to their CI/CD pipeline. The sidecar's job was straightforward: sign every container image built by the team, using a private key stored in a vault. Within a week, the sidecar had signed roughly 2,000 images spanning base layers, application code, and debug utilities. No one ever checked a single signature on the other side of the pipeline. The sidecar signed everything and validated nothing. That asymmetry is not an edge case. It is the default posture for most organizations that have adopted image signing.

The Sidecar That Signed Everything

Sidecar containers are a common Kubernetes pattern for extending the behavior of a pod without modifying the primary application. In this case, the sidecar ran alongside the build agent, intercepting each image tag and applying a cryptographic signature using cosign, the signing tool from the Sigstore project. The key was stored in AWS KMS and rotated every 90 days—or so the documentation claimed. An audit later revealed that the rotation policy had never been triggered; the same key had been used for eight months.

The sidecar's logic was simple: on every docker push, it would pull the image manifest, compute a digest, sign the digest with the private key, and store the signature as an attached tag in the same registry. The signing step took roughly 200–400 milliseconds per image, depending on layer count. The team celebrated the deployment as a win for supply chain security. They had achieved signing coverage for 100% of their images. What they had not achieved was any verification.

No admission controller or policy engine was configured to check those signatures during deployment. The Kubernetes cluster ran vanilla imagePullPolicy: Always, but no webhook validated that the pulled image's signature matched the one stored in the registry. The sidecar had turned image signing into a write-only operation. An attacker who gained write access to the registry could overwrite an existing image, and the sidecar would happily sign the new version on the next push—or, worse, the attacker could push a new image with a matching tag, and the sidecar would sign that too.

The team discovered the gap only after a penetration test. The testers overwrote a single layer in a base image, pushed the modified tag, and watched the sidecar sign it automatically. The deployment pipeline accepted the tampered image without a single warning. The signing sidecar had become a rubber stamp.

Supply Chain Trust as a Monoculture

The SaaS team's setup is a microcosm of a larger problem: the industry has treated image signing as a one-time act of provenance rather than a continuous chain of verification. Most signing implementations rely on a single key that signs everything in the registry. That key, if compromised, undermines the entire trust model. Docker Content Trust, for example, uses a single delegations key per repository; if that key leaks, an attacker can sign arbitrary images until the key is revoked—and revocation is rarely automated.

Notary v1, the original implementation behind Docker Content Trust, had no built-in key rotation policy. Teams that adopted it often generated a key once and used it for years. A 2023 analysis of public Notary repositories found that roughly 15% of signing keys had been active for more than three years without rotation. The same analysis noted that fewer than 5% of repositories enforced signature verification on pull. The signing infrastructure existed, but the verification infrastructure was absent.

This monoculture of trust is fragile. A single signing key represents a single point of failure. If an attacker obtains the key, they can sign any image, and the system will treat it as authentic. The sidecar pattern amplifies this risk because the signing key is often stored in the same cloud environment as the registry. An attacker who compromises the CI/CD pipeline gains access to both the signing key and the registry write path. The sidecar becomes an enabler, not a guard.

The monoculture extends to tooling. Most signing tools operate on the assumption that signing alone is sufficient. Cosign, for instance, defaults to cosign sign without requiring a subsequent cosign verify step. Kyverno, a popular policy engine for Kubernetes, can enforce image verification, but its default configuration allows unsigned images to pass. A survey by the CNCF in late 2024 estimated that roughly 60% of Kubernetes clusters in production had no image verification policy configured. The other 40% had policies that were either too permissive or never tested.

How the Validation Gap Persists

The validation gap is not a secret. It is a known trade-off that teams make consciously, often prioritizing deployment speed over security. In a typical CI/CD pipeline, signing is added as a post-build step that runs in a few hundred milliseconds. Verification, by contrast, requires an admission controller that intercepts every pod creation, fetches the signature from the registry, verifies the digest, and checks the key against a trust root. That process can add 500 milliseconds to a few seconds per deployment, depending on network latency and registry performance.

When teams are asked why they skip verification, the answer is almost always the same: "We didn't want to slow down deployments." A 2025 survey by a cloud-native security vendor found that 47% of respondents who used image signing did not enforce verification in production. The most common reason cited was deployment latency. The second most common was complexity: configuring a policy engine like OPA Gatekeeper or Kyverno to check signatures requires maintaining a trust bundle, handling key rotation, and managing exceptions for legacy images.

The complexity argument has merit. Kyverno policies for image verification can exceed 50 lines of YAML, and getting the trust bundle right requires understanding the Sigstore Fulcio and Rekor ecosystem. Teams that adopt keyless signing via Sigstore must configure OIDC identity tokens, which adds another layer of moving parts. The default behavior of many tools is to allow unsigned images, which means that a misconfigured policy silently accepts everything. The sidecar team at the SaaS company had a Kyverno policy, but it was scoped only to the default namespace and had an exception list that covered 80% of their workloads.

Some teams argue that verification is unnecessary if the registry is private and access is tightly controlled. That argument assumes that registry access cannot be compromised. But registry credentials are often stored in CI/CD secrets, and those secrets leak. The one maintainer's two-factor bypass story illustrated how a single misconfigured config file can expose credentials that grant write access to a registry. Once an attacker has registry write access, the absence of verification turns signing into a cosmetic feature.

A Concrete Attack Path

Consider a realistic attack path. The target is a Kubernetes cluster running a microservice that processes payment data. The microservice's image is built from a base image, say python:3.11-slim, with an application layer on top. The sidecar signs the final image after each build. The signature is stored as a tag like python:3.11-slim.sig in the same registry. No admission controller checks the signature at deploy time.

An attacker gains access to the registry through a leaked CI/CD token. The attacker pulls the manifest for the payment microservice image, identifies the base layer digest, and overwrites that layer with a version that includes a cryptominer. The attacker then pushes the modified manifest under the same tag. The sidecar, which is still running in the CI/CD pipeline, picks up the push event and signs the new manifest. The signature tag is updated. The attacker has now signed a tampered image using the legitimate signing key.

The deployment pipeline sees the new tag, pulls the image, and deploys it to the cluster. No verification step compares the signature against a known-good digest. The cryptominer runs undetected for weeks, consuming CPU and leaking data through DNS tunnels. The team notices the performance degradation but attributes it to a recent code change. By the time the incident is traced to the image tampering, the attacker has exfiltrated a significant amount of data.

This attack path is not hypothetical. In 2024, a public registry incident demonstrated the same pattern: an attacker overwrote a single layer in a popular base image, and the registry's signing infrastructure signed the new version automatically. The incident was caught only because a third-party scanner noticed a digest mismatch. The scanner was not part of the verification pipeline; it was an external monitoring tool that alerted the team after the image had been deployed for several days.

The root cause was not the signing key compromise. The key was never stolen. The root cause was the absence of a verification step that could detect the mismatch between the signed digest and the actual image content. The sidecar signed whatever was in the registry, regardless of whether it matched the original build.

Why Existing Tools Missed This

Existing supply chain security tools have focused heavily on signing and attestation but have underinvested in runtime verification. Sigstore, for example, provides a robust infrastructure for keyless signing and transparency logs, but it does not enforce verification in the deployment pipeline. The Sigstore ecosystem assumes that verification will be handled by external policy engines, but those engines are often configured too loosely or not at all.

SLSA (Supply-chain Levels for Software Artifacts) defines levels of build integrity, from SLSA 1 (documented build) to SLSA 4 (hermetic, reproducible build). However, SLSA levels do not mandate runtime signature enforcement. A project can achieve SLSA 4 with a fully hermetic build and signed provenance, yet still deploy those signed artifacts without any verification. The SLSA specification explicitly leaves verification to the deploying organization, which means that the gap between signing and verification persists regardless of the SLSA level.

Grafeas and in-toto attestations provide a rich model for describing what happened during a build, but adoption has been slow. A 2025 analysis of public repositories found that fewer than 2% of images published to Docker Hub included in-toto attestations. The attestations that do exist are rarely checked at deploy time. The same analysis found that even among images that included attestations, fewer than 10% were deployed in clusters that enforced attestation verification. The attestation infrastructure is like a library of books that no one reads.

Vulnerability scanners, such as Trivy and Grype, check for known CVEs in image layers, but they do not check signer identity. A tampered image with no known vulnerabilities will pass a scanner check. The scanner cannot detect that the image's content differs from what the original developer built. The sidecar team's images were scanned and found clean. The tampered image also scanned clean because the cryptominer used a zero-day exploit that no scanner had a signature for. The scanner provided a false sense of security.

Industry inertia plays a role. Signing and verification are often treated as checkbox items on a security compliance form. A team can claim they use image signing, and an auditor will check the box without asking whether verification is enforced. The one CI platform standardized on JSON schema story showed how a well-intentioned standardization can create blind spots when defaults are not validated. The same dynamic applies to signing: the default is to sign, not to verify.

Fixing the Loop: Sign Then Verify

Closing the validation gap requires a shift from a sign-only posture to a sign-and-verify posture. The first step is to add a mandatory verification step after signing, either in the CI/CD pipeline or at the admission controller level. In the CI/CD pipeline, the build process should include a cosign verify command that checks the signature immediately after signing. This catches misconfigurations early, before the image is pushed to production. The SaaS team added a verification step that runs in roughly 300 milliseconds and rejects any image whose signature does not match the expected digest.

The second step is to use short-lived keys per image or per build stage. Instead of a single key that signs everything, each image can be signed with a key that expires after a few hours. Tools like Sigstore's keyless signing already implement this pattern: the signing key is derived from an OIDC token that expires after the CI job completes. The signature can be verified against the transparency log, which provides a tamper-evident record of when the key was valid. Short-lived keys limit the blast radius of a key compromise.

Policy enforcement is the third step. Kubernetes admission controllers like Kyverno and OPA Gatekeeper can be configured to require a valid signature for every image in a namespace. The policy should check not only that a signature exists but that the signing key matches a trusted identity. The trust bundle should be updated automatically, not stored as a static file that goes stale. The SaaS team now uses a Kyverno policy that rejects any image without a valid signature from their CI system's OIDC identity. The policy is scoped to production namespaces, with a gradual rollout to staging.

Audit logging is often overlooked. Every signing and verification event should be logged to an immutable store. The logs should include the image digest, the signing key identifier, the timestamp, and the identity of the entity that performed the operation. If a tampered image is later discovered, the audit log can pinpoint when and by whom the image was signed. The SaaS team now ships signing and verification logs to a dedicated S3 bucket with object lock enabled.

Finally, adopting a framework like TUF (The Update Framework) can provide key rotation and delegation policies that are missing from ad-hoc signing setups. TUF separates root keys from target keys and supports automatic key rotation. A TUF repository can enforce that images are signed by specific delegations, and that verification checks the full delegation chain. Several container registries, including Docker Hub and GitHub Container Registry, are exploring TUF integration as a way to bridge the signing-verification gap.

The Cost of Trust Is Verification

Adding verification to the deployment pipeline comes with costs. Admission controllers add latency to every pod creation. In clusters that deploy hundreds of pods per minute, a few hundred milliseconds per verification can accumulate. Some teams report that verification adds roughly 1–3 seconds to the total deployment time for a typical microservice. That latency can be mitigated by caching verification results for images that have already been validated within a configurable window.

The cost of not verifying, however, is orders of magnitude higher. A single tampered image can lead to data exfiltration, cryptomining, or lateral movement within a cluster. The average cost of a software supply chain incident, according to a 2025 study by a cybersecurity research firm, was roughly $4.6 million for mid-sized organizations. The SaaS team's penetration test did not result in a real breach, but the remediation effort—re-signing all 2,000 images, rotating keys, and implementing verification—took three weeks and cost roughly $80,000 in engineering time.

Teams should start with critical paths: production images that handle sensitive data or are exposed to the internet. Verification can be gradually expanded to staging and development environments as the process matures. The key is to automate verification in the CI/CD pipeline, not just in admission control. A verification failure in CI is a fast feedback loop; a verification failure in admission is a blocked deployment that can cause an incident.

The sidecar that signed everything and validated nothing is a cautionary tale, but it is not unique. It is the natural outcome of a security industry that has sold signing as a solution without emphasizing that verification is the other half of the equation. The tools exist—cosign, Kyverno, OPA, TUF—but they require deliberate configuration and ongoing maintenance. The cost of trust is verification, and that cost must be paid every time an image is deployed. There is no one-time setup that makes the supply chain secure. The loop must be closed on every push.

Some readers will argue that the sidecar team's mistake was obvious and that any competent security engineer would have caught it. That may be true, but the prevalence of similar gaps suggests otherwise. The CNCF survey that found 60% of clusters lack image verification indicates that the gap is widespread. The sidecar story is not an outlier; it is the median. The industry has collectively decided that signing is enough, and that decision is a vulnerability waiting to be exploited.

The fix is not complicated, but it is not automatic. It requires a change in mindset from signing as a ceremony to verification as a continuous practice. Every signature must be checked. Every key must be rotated. Every policy must be tested. The sidecar can be part of the solution, but only if it is paired with a validator that refuses to deploy unsigned or mismatched images. The alternative is a supply chain that trusts everything and verifies nothing—a chain that is only as strong as the weakest sidecar.