Kubernetes container image security: vulnerability scanning with Trivy

Table of contents

• Goal
• Prerequisites
• Security advisory: Trivy supply chain compromise (March 2026)
• What Trivy scans in a container image
• Step 1: install Trivy and run your first scan
• Step 2: filter results and suppress accepted risks
• Step 3: gate your CI pipeline with GitHub Actions
• Step 4: generate and attest an SBOM
• Step 5: sign images with Cosign
• Step 6: enforce image signatures with Kyverno
• Step 7: continuous in-cluster scanning with the Trivy Operator
• Step 8: monitor vulnerability reports with Prometheus and Grafana
• Reducing your attack surface: image minimisation
• Three misconceptions about container scanning
• Verify the final result
• Common troubleshooting
• When to escalate

Goal

At the end of this guide you will have a layered container image security pipeline: Trivy scanning in CI that blocks builds on critical findings, signed images verified at admission time by Kyverno, and continuous in-cluster monitoring through the Trivy Operator with Prometheus alerting.

Prerequisites

• A Kubernetes cluster running version 1.26 or later (for CRD v1 and Helm OCI support)
• kubectl configured with cluster-admin access
• helm v3.8.0 or later (for OCI chart support)
• A container registry you push images to (Docker Hub, ECR, GCR, ACR, or a private registry)
• A GitHub Actions workflow (or equivalent CI system) for your container builds
• Kyverno installed on your cluster if you want admission enforcement (Step 6)
• Familiarity with Kubernetes RBAC for creating ServiceAccounts and ClusterRoles

Security advisory: Trivy supply chain compromise (March 2026)

Between March 19 and 23, 2026, a threat actor compromised Aqua Security's Trivy distribution infrastructure. The attack, documented in GHSA-69fq-xp46-6x23, injected credential-stealing malware into Trivy binaries (v0.69.4 through v0.69.6), the trivy-action GitHub Action (76 of 77 version tags force-pushed to malicious commits), and Docker Hub images (v0.69.5, v0.69.6).

Safe versions: Trivy binary v0.69.3 or earlier, trivy-action v0.35.0, setup-trivy v0.2.6.

The root cause was mutable Git tags. A tag like @v0.28.0 can be reassigned to a different commit at any time. The fix is straightforward: pin GitHub Actions to full commit SHA hashes, not version tags. Every Actions reference in this article follows that practice.

If your pipeline ran a compromised version during March 19-23 UTC, rotate all secrets that were accessible to the runner (cloud credentials, SSH keys, API tokens, Docker registry passwords). Search workflow logs for repositories named tpcp-docs-* and outbound connections to scan.aquasecurtiy.org (note the misspelling).

What Trivy scans in a container image

Trivy analyzes every layer of a container image, not just the base OS. It detects:

• OS package vulnerabilities (Alpine, Debian, Ubuntu, RHEL, CentOS) from vendor security advisories
• Language dependency vulnerabilities (pip, npm, Go modules, Maven/Gradle, NuGet, Cargo, Gemfile) from the GitHub Advisory Database, PyPI Advisory DB, and others
• Embedded secrets (API keys, tokens, credentials found in image layers)
• Dockerfile misconfigurations (running as root, unnecessary exposed ports) when enabled with --scanners misconfig

The vulnerability database is rebuilt every six hours from NVD, GitHub Advisory Database, and OS vendor advisories, then distributed via GitHub Container Registry.

Step 1: install Trivy and run your first scan

Install the Trivy CLI on your workstation. Pick one method:

# Homebrew (macOS / Linux)
brew install aquasecurity/trivy/trivy

# APT (Debian/Ubuntu)
sudo apt-get install trivy

# Or run without installing, using Docker
docker run --rm aquasec/trivy:0.69.3 image nginx:1.27.4

Run a scan against a public image to see the output format:

trivy image nginx:1.27.4

Expected output (abbreviated):

nginx:1.27.4 (debian 12.9)
===========================
Total: 143 (UNKNOWN: 0, LOW: 82, MEDIUM: 43, HIGH: 14, CRITICAL: 4)

┌──────────────┬────────────────┬──────────┬────────────────────┬───────────────┬──────────────────────────────────────┐
│   Library    │ Vulnerability  │ Severity │ Installed Version  │ Fixed Version │             Title                    │
├──────────────┼────────────────┼──────────┼────────────────────┼───────────────┼──────────────────────────────────────┤
│ libssl3t64   │ CVE-2024-9143  │ CRITICAL │ 3.0.14-1~deb12u2   │ 3.0.15-1      │ OpenSSL: Low-level ...              │
└──────────────┴────────────────┴──────────┴────────────────────┴───────────────┴──────────────────────────────────────┘

Each row maps a package to a CVE, its severity, the installed version, and the version that fixes it. An empty "Fixed Version" means no patch is available yet.

Step 2: filter results and suppress accepted risks

A raw scan often reports hundreds of findings, many without available fixes. Filter the noise:

# Show only HIGH and CRITICAL, skip unfixed CVEs
trivy image --severity HIGH,CRITICAL --ignore-unfixed nginx:1.27.4

For CVEs you have reviewed and accepted, create a .trivyignore file in your repository root:

# Reviewed 2026-04-01 — no exploitable path in our deployment
CVE-2024-9143

# Accept temporarily, re-evaluate in 90 days
CVE-2023-3817 exp:2026-07-01

The exp: syntax auto-reactivates the finding after the date passes. This prevents forgotten suppressed CVEs from accumulating silently.

For finer control, use the YAML format that scopes suppression to specific packages or paths:

# .trivyignore.yaml
vulnerabilities:
  - id: CVE-2022-40897
    paths:
      - "usr/local/lib/python3.12/site-packages/setuptools"
    expired_at: 2026-09-01

Load it explicitly: trivy image --ignorefile .trivyignore.yaml nginx:1.27.4

Step 3: gate your CI pipeline with GitHub Actions

Add Trivy as a hard gate that fails your build when it finds CRITICAL or HIGH vulnerabilities with available fixes. This workflow uploads results to GitHub's Security tab in SARIF format:

# .github/workflows/image-scan.yml
name: Container image scan

on:
  push:
    branches: [main]
  pull_request:

jobs:
  scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Build image
        run: docker build -t myorg/myapp:$ .

      # Pin to commit SHA, not version tag — see security advisory above
      - name: Scan image (hard gate)
        uses: aquasecurity/trivy-action@18f2510ee396bbf400402947e795cb3a0d0f5cc1 # v0.35.0
        with:
          image-ref: "myorg/myapp:$"
          format: "table"
          exit-code: "1"                    # fail the build on findings
          ignore-unfixed: true              # skip CVEs without a fix
          severity: "CRITICAL,HIGH"
          vuln-type: "os,library"
          trivyignores: ".trivyignore"

      # Second step: upload SARIF for the GitHub Security tab
      - name: Scan image (SARIF report)
        uses: aquasecurity/trivy-action@18f2510ee396bbf400402947e795cb3a0d0f5cc1 # v0.35.0
        if: always()
        with:
          image-ref: "myorg/myapp:$"
          format: "sarif"
          output: "trivy-results.sarif"
          ignore-unfixed: true
          severity: "CRITICAL,HIGH"
          vuln-type: "os,library"

      - name: Upload SARIF to GitHub Security
        uses: github/codeql-action/upload-sarif@v3
        if: always()
        with:
          sarif_file: "trivy-results.sarif"

The SHA 18f2510ee396bbf400402947e795cb3a0d0f5cc1 corresponds to v0.35.0 at the time of writing. Use Dependabot or Renovate to track SHA updates automatically.

Step 4: generate and attest an SBOM

A Software Bill of Materials records every package in your image. Generate it in CycloneDX format alongside your build:

trivy image --format cyclonedx --output sbom.cdx.json myorg/myapp:v1.2.0

Attest the SBOM to the image with Cosign (requires Cosign from the next step):

cosign attest --key cosign.key \
  --type cyclonedx \
  --predicate sbom.cdx.json \
  myorg/myapp:v1.2.0

Later, anyone can verify the attestation and scan the SBOM without pulling the full image:

cosign verify-attestation --type cyclonedx --key cosign.pub myorg/myapp:v1.2.0
trivy sbom ./sbom.cdx.json

This matters because it lets you rescan old images against the latest vulnerability database without rebuilding them.

Step 5: sign images with Cosign

Cosign signs container images so that downstream consumers (including Kyverno) can verify that an image came from your trusted build pipeline.

Keyless signing (recommended for CI). Uses ephemeral keys tied to your CI provider's OIDC identity. No key management required:

# In GitHub Actions — OIDC token is available automatically
cosign sign myorg/myapp:v1.2.0

The signing event is recorded in Rekor, Sigstore's transparency log.

Key-based signing (for environments without OIDC):

# Generate a key pair (once)
cosign generate-key-pair

# Sign with the private key
cosign sign --key cosign.key myorg/myapp:v1.2.0

# Verify with the public key
cosign verify --key cosign.pub myorg/myapp:v1.2.0

For production, store keys in a KMS rather than on disk:

# AWS KMS
cosign sign --key awskms://alias/image-signing myorg/myapp:v1.2.0

# GCP KMS
cosign sign --key gcpkms://projects/myproject/locations/global/keyRings/signing/cryptoKeys/cosign/cryptoKeyVersions/1 myorg/myapp:v1.2.0

# HashiCorp Vault
cosign sign --key hashivault://image-signing myorg/myapp:v1.2.0

Step 6: enforce image signatures with Kyverno

With images signed, block unsigned images from running on your cluster. Kyverno natively integrates with Cosign for this. If you have not installed Kyverno yet, see the admission controllers comparison for installation options.

Create a ClusterPolicy that verifies signatures for your registry:

# verify-image-signature.yaml
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: verify-image-signature
spec:
  validationFailureAction: Enforce    # Audit first, then switch to Enforce
  rules:
    - name: check-signature
      match:
        any:
          - resources:
              kinds:
                - Pod
      verifyImages:
        - imageReferences:
            - "myorg/*"               # match all images from your registry
          attestors:
            - entries:
                - keys:
                    publicKeys: |-
                      -----BEGIN PUBLIC KEY-----
                      MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE...
                      -----END PUBLIC KEY-----

Apply it:

kubectl apply -f verify-image-signature.yaml

Test by deploying an unsigned image:

kubectl run test-unsigned --image=myorg/unsigned-app:latest

Expected output:

Error from server: admission webhook "mutate.kyverno.svc-fail" denied the request:
resource Pod/default/test-unsigned was blocked due to the following policies:
verify-image-signature: check-signature: image verification failed

Start with validationFailureAction: Audit to log violations without blocking. Switch to Enforce after you have confirmed all production workloads use signed images.

Step 7: continuous in-cluster scanning with the Trivy Operator

CI scanning catches problems at build time, but new CVEs appear daily against images already running in your cluster. The Trivy Operator runs as a Kubernetes Operator that automatically scans workloads when pods are created or updated, and stores results as Custom Resources.

Install via Helm:

helm repo add aqua https://aquasecurity.github.io/helm-charts/
helm repo update

helm install trivy-operator aqua/trivy-operator \
  --namespace trivy-system \
  --create-namespace \
  --set trivy.ignoreUnfixed=true

Check the Helm chart repository for the latest version. The Operator is marked as "incubating"; pin your Helm chart version in production.

After installation, the Operator starts scanning existing workloads. Query the results:

# List all vulnerability reports
kubectl get vulnerabilityreports -A

# Check a specific namespace
kubectl get vulnerabilityreports -n production -o wide

# Inspect a specific report
kubectl describe vulnerabilityreport deployment-myapp-myapp -n production

Reports follow the naming convention <workload-kind>-<workload-name>-<container-name>. Each report contains a summary with counts per severity level and a full list of CVEs with package names, installed versions, and fix versions.

Private registries. The Operator discovers credentials automatically from imagePullSecrets in Pod specs and ServiceAccounts. For cloud registries (ECR, GCR, ACR), the Operator uses the node's IAM identity or workload identity through the standard cloud SDK credential chain.

Step 8: monitor vulnerability reports with Prometheus and Grafana

The Trivy Operator exposes Prometheus metrics via a ServiceMonitor. If you run kube-prometheus-stack, Prometheus discovers these metrics automatically.

Key metrics:

Useful PromQL queries for Grafana dashboards:

# Critical vulnerabilities in the production namespace
sum(trivy_image_vulnerabilities{namespace="production", severity="CRITICAL"})

# Top 5 most vulnerable namespaces
topk(5, sum by (namespace) (trivy_image_vulnerabilities))

Set up an alert for new critical findings:

# trivy-alerts.yaml (PrometheusRule)
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: trivy-critical-vulns
  namespace: trivy-system
spec:
  groups:
    - name: trivy
      rules:
        - alert: CriticalVulnerabilityDetected
          expr: trivy_image_vulnerabilities{severity="CRITICAL"} > 0
          for: 5m
          labels:
            severity: critical
          annotations:
            summary: "Critical vulnerability in /"

Reducing your attack surface: image minimisation

Scanning is reactive. Minimising what goes into the image is proactive and reduces the number of CVEs you need to manage.

Use distroless or minimal base images. Chainguard Images and Google Distroless strip shells, package managers, and system utilities. Sysdig's scanning data shows roughly 24 CVEs in an Ubuntu-based nginx image versus roughly 2 in a distroless equivalent.

Pin image references to digests. Mutable tags like :latest or :1.27 can be silently updated in a registry. Use immutable SHA digests in production manifests:

# Instead of this
image: nginx:1.27.4
# Use this
image: nginx@sha256:4a2923e3e2e26e4c0c77b2f7b8f10b3d7a2b0f1e...

Multi-stage builds. Keep build tools out of the final image. A Go application does not need the Go compiler at runtime; a Node.js app does not need npm or yarn:

FROM golang:1.22 AS build
WORKDIR /app
COPY . .
RUN CGO_ENABLED=0 go build -o /myapp

FROM gcr.io/distroless/static-debian12:nonroot
COPY --from=build /myapp /myapp
ENTRYPOINT ["/myapp"]

Three misconceptions about container scanning

"Scanning the base image is sufficient." Container images consist of multiple layers: the base OS, runtime dependencies, application code, and bundled packages. A clean Ubuntu base image tells you nothing about the vulnerable setuptools pinned in your Python requirements.txt. Trivy scans every layer and every package manager manifest it finds. Always scan the final built image, not just the base.

"Zero CVEs means my image is secure." Zero CVEs means "no known vulnerabilities in this scanner's database at this moment." New CVEs are published daily. Alpine's advisory database only lists CVEs that have been patched, so a scanner may report zero findings simply because no fix has been issued yet. And CVE scanning misses hardcoded secrets, Dockerfile misconfigurations, weak RBAC, and license violations entirely.

"You only need to scan at build time." An image built six months ago without findings may have dozens of known CVEs today. Admission controllers check images at deployment time but cannot detect new CVEs that emerge in already-running containers. That is why this guide sets up four layers: build-time scanning (Step 3), SBOM generation for later rescanning (Step 4), admission enforcement (Step 6), and continuous in-cluster monitoring (Step 7).

Verify the final result

After completing all steps, verify each layer is working:

# 1. CI gate: push a Dockerfile with an old base image and confirm the pipeline fails
# 2. Image signing: verify your latest image
cosign verify --key cosign.pub myorg/myapp:v1.2.0

# 3. Kyverno enforcement: try deploying an unsigned image
kubectl run test --image=docker.io/library/nginx:latest --dry-run=server

# 4. Trivy Operator: check reports exist for running workloads
kubectl get vulnerabilityreports -A | head -20

# 5. Prometheus: query for the metric
curl -s http://prometheus:9090/api/v1/query?query=trivy_image_vulnerabilities | jq .

If step 3 returns a Kyverno denial and step 4 shows VulnerabilityReports, your layered scanning pipeline is operational.

Common troubleshooting

Trivy Operator reports are missing for some workloads. The Operator scans workloads managed by a controller (Deployment, StatefulSet, DaemonSet, ReplicaSet, Job, CronJob). Bare pods created without a controller are not scanned by default. Check that the Operator pods are running: kubectl get pods -n trivy-system.

VulnerabilityReport exceeds etcd size limit. Images with many packages can produce reports that exceed etcd's 1.5 MiB per-object limit. Filter by severity in the Operator configuration (trivy.severity: "HIGH,CRITICAL") or limit report scope.

Private registry authentication fails. The Operator resolves credentials from imagePullSecrets in the Pod spec and ServiceAccount. Verify the secret exists in the workload's namespace and that the credentials are valid. For cloud registries, confirm the node's IAM role or workload identity has pull permissions.

Kyverno blocks legitimate images after enabling Enforce mode. Switch back to Audit mode, check PolicyReports for false positives, and verify the public key matches the key used during signing. Use cosign triangulate myorg/myapp:v1.2.0 to confirm the signature artifact exists in the registry.

Trivy database download fails in air-gapped environments. Download the database manually from Trivy's GitHub releases and host it on an internal OCI registry using ORAS CLI. Configure the Operator's trivy.dbRepository to point to your internal mirror.

When to escalate

Collect these details before asking for help:

• Trivy CLI version (trivy version)
• Trivy Operator Helm chart version and Operator pod logs (kubectl logs -n trivy-system deploy/trivy-operator)
• Kyverno version and admission webhook logs (kubectl logs -n kyverno deploy/kyverno-admission-controller)
• The specific image reference (with digest) that is failing
• Output of kubectl get vulnerabilityreports -A -o wide
• Your .trivyignore file contents
• Cosign version and the signing method used (keyless vs key-based vs KMS)