Skip to content
Hands-on IT IT Drills

Chapter 8: Automated scanning tools, finding triage and thresholds (SAST, SCA, image and IaC scans with Trivy, Checkov)

In this chapter you will learn:

  • which kinds of scanners exist (SAST, SCA, image, IaC and secret scans) and where they belong in the pipeline
  • how to prioritize findings with CVSS, EPSS and KEV instead of drowning in lists
  • how to set thresholds (“fail on CRITICAL with an available fix”) and implement the baseline principle “no new findings”
  • how to document exceptions with a justification and an expiry date (.trivyignore.yaml, Checkov skip)
  • how to run Trivy offline with your own database mirror and combine everything into a GitLab CI stage security

What is this about?

Picture a food safety inspection in a large commercial kitchen. One inspector checks whether the cooks work cleanly (how is the food cooked?). A second checks the delivered ingredients against a recall list (is this batch of flour known to be contaminated?). A third looks at the finished, packaged dish. A fourth checks the kitchen’s floor plan (is the cold room set correctly?). And one looks for keys that someone left stuck to the fridge.

None of these inspectors is enough alone. And if every inspector reports every little thing, the kitchen shuts down even though nothing dangerous has happened. That is why two things are needed: rules for when operations are stopped (thresholds) and someone who assesses the reports (triage).

That is exactly what this chapter is about. You unleash five kinds of scanners on the demo-app, learn to assess the findings, and build a pipeline stage from them that only turns red when it really matters.

  Commit / merge request                  Build                      Release
 +----------------------+       +----------------------+     +-------------------+
 | Secret scan (gitleaks)|      | Build image          |     | Gate: thresholds  |
 | SAST (Semgrep)        | ---> | Image scan (Trivy)   | --> | met?              |
 | SCA  (trivy fs)       |      | SBOM (Chapter 7)     |     | yes -> signature  |
 | IaC  (trivy config,   |      +----------------------+     | no -> triage      |
 |       Checkov)        |                                   +-------------------+
 +----------------------+                                            |
           |                                                         v
           +-------------> Reports (SARIF/JSON) -----------> Triage board / tickets

The concepts in plain terms

Five kinds of scanners at a glance

The abbreviations sound similar, but they check completely different things.

Kind What it finds Example tool When in the pipeline
SAST Insecure patterns in your own code Semgrep On every commit/MR
SCA Known vulnerabilities in dependencies trivy fs, Grype On every commit/MR
Image scan Vulnerabilities in OS packages and libs in the image trivy image After the build, before release
IaC scan Misconfiguration in manifests, Terraform, Dockerfile trivy config, Checkov On every commit/MR
Secret scan Passwords, tokens, keys in the code gitleaks, trivy Pre-commit and on every MR

SAST stands for Static Application Security Testing, that is, static analysis of the source code without running it. The scanner looks for patterns like “user input ends up unchecked in a shell command”.

SCA stands for Software Composition Analysis, that is, analysis of what the software is composed of. The scanner reads requirements.txt, package-lock.json or pom.xml and compares the versions with vulnerability databases. So it does not find bugs in your code, but known bugs in other people’s code that you ship along.

The image scan is an SCA scan of the finished container image. It additionally sees the operating system packages from the base image (for example openssl or libc from Debian).

The IaC scan (Infrastructure as Code) checks configuration files: does the container run as root? Are resource limits missing? Is an S3 bucket public?

The secret scan looks for credentials. You went deeper into this topic in Chapter 6; it belongs in the stage here for completeness.

Remember: SAST finds your bugs, SCA and image scans find other people’s known bugs, IaC scans find configuration errors. No tool replaces another.

How serious is a finding? CVSS, EPSS and KEV

A scanner quickly reports hundreds of CVEs. A CVE (Common Vulnerabilities and Exposures) is a unique number for a known vulnerability, for example CVE-2023-32681. To decide what comes first, there are three ratings.

Metric Question it answers Range Publisher
CVSS How bad would it be if it happened? 0.0 to 10.0 FIRST (standard), NVD/vendors
EPSS How likely is it to be exploited soon? 0 to 1 (percent) FIRST
KEV Is it demonstrably being exploited already? yes/no CISA (US agency)

CVSS (Common Vulnerability Scoring System) rates the theoretical severity. Scanners derive the levels from it: LOW (0.1–3.9), MEDIUM (4.0–6.9), HIGH (7.0–8.9), CRITICAL (9.0–10.0). Version 3.1 is widespread; version 4.0 is slowly gaining ground.

EPSS (Exploit Prediction Scoring System) estimates the probability that a vulnerability will be exploited in the next 30 days. A CVSS 9.8 with EPSS 0.001 is often less urgent than a CVSS 7.5 with EPSS 0.9.

KEV (Known Exploited Vulnerabilities) is a CISA catalog of vulnerabilities that are demonstrably being exploited in the wild. If a CVE is in the KEV catalog: act immediately.

Tip: A simple prioritization to start with: (1) KEV-listed, (2) CRITICAL/HIGH with high EPSS, (3) CRITICAL/HIGH with a fix, (4) the rest in the normal patch cycle.

Context also matters. A vulnerability in a library whose vulnerable function you never call is less urgent. A vulnerability in a service reachable from the internet is more urgent than one in a batch job behind three zone boundaries.

False positives and false negatives

A false positive is a finding that is not one in your context. Examples:

  • Semgrep reports shell=True, but the command is a fixed string without user input.
  • Trivy reports a CVE in a package that only existed in the build stage, but the image layer was attributed incorrectly.
  • Checkov demands an image digest, but the digest is only enforced at deployment via Kyverno (see Chapter 9).

A false negative is worse: a real vulnerability that is not reported. This happens, for example, when the vulnerability database is outdated, when dependencies are not pinned (the scanner sees no version), or when binaries are copied into the image without package metadata.

Warning: A green scan only means: “Nothing found that the scanner knows about.” It does not mean “secure”.

Thresholds: when does the pipeline turn red?

A threshold defines which findings abort the build. Typical levels:

Rule Effect Suitable for
Fail on CRITICAL only critical findings abort getting started, legacy projects
Fail on HIGH,CRITICAL stricter new projects, production
only with a fix (--ignore-unfixed) ignores findings without a patch almost always sensible
KEV always fails regardless of CVSS regulated environments

The argument for “only with a fix”: if there is no patch, the development team can do nothing but wait. A build that stays red for weeks gets ignored or bypassed. Findings without a fix belong in a risk register and are watched, but do not block.

Baseline and ratchet: “no new findings”

In a legacy project, the first scan often finds 200 findings. If you switched to “fail on HIGH” right away, every build would be red. The solution is called a baseline: you freeze today’s state and only require that nothing new is added.

The ratchet principle (like the tool: it turns in one direction only) complements this. Every time the team fixes a legacy finding, the baseline shrinks. It never grows again.

 Findings
  200 |#####
  150 |#####  ####
  100 |#####  ####  ###
   50 |#####  ####  ###  ##
    0 +-----------------------> time
        Q1    Q2    Q3   Q4     (baseline only drops, new findings = build red)

Tools support this in different ways:

  • Semgrep: --baseline-commit <ref> reports only findings that are new since this commit.
  • Checkov: --create-baseline writes a .checkov.baseline, --baseline <datei> compares against it.
  • Trivy: no built-in baseline; you use an ignore file with expiry dates.

SARIF: one format for all reports

SARIF (Static Analysis Results Interchange Format) is a JSON-based OASIS standard for scan results. Semgrep, Trivy and Checkov can write SARIF. The advantage: a triage tool, a code review system or a central dashboard only has to understand one format. For your own evaluations with jq, the respective native JSON is often more convenient. That is why you store both as artifacts in the pipeline.

Trivy offline: mirroring the vulnerability database

On startup, Trivy downloads its vulnerability database (trivy-db). This DB is distributed as an OCI artifact, that is, stored in a container registry like an image (default sources: mirror.gcr.io/aquasec/trivy-db:2 and ghcr.io/aquasecurity/trivy-db:2). For Java dependencies there is additionally trivy-java-db:1, and for IaC check rules the trivy-checks bundle.

In an isolated (air-gapped) environment, there is no internet. Then you have two options:

  1. Mirror the DB as an OCI artifact: On the internet side, you copy trivy-db:2 into your registry mirror (Harbor, Nexus, see Chapter 3) with oras cp. You point Trivy at this mirror with --db-repository.
  2. Transport the cache directory: You download the DB once with trivy image --download-db-only into a cache directory, transport it across the zone boundary (see Chapter 4) and start Trivy there with --skip-db-update.

Security: The DB is itself a supply chain artifact. Whoever tampers with it can make findings disappear. Treat it like any other artifact: verify its origin, adopt it by digest, log the transfer.

Pay attention to freshness: the DB is rebuilt several times a day. A DB that is a week old misses new CVEs. Trivy warns when the DB is too old.

Triage: from finding to decision

Triage comes from emergency medicine: who gets treated first? For every finding that breaks the threshold, there are exactly three outcomes.

                 +-------------------+
                 |  New finding      |
                 +---------+---------+
                           |
          +----------------+------------------+
          v                v                  v
  +---------------+ +----------------+ +-----------------------+
  | Confirmed     | | False positive | | Risk accepted         |
  | -> fix ticket | | -> ignore with | | -> ignore with expiry |
  |    + due date | |    reason      | |    + ticket + owner   |
  +---------------+ +----------------+ +-----------------------+

Every exception needs four pieces of information: what (ID, path), why (justification), who (responsible person or team), until when (expiry date). Without an expiry date, the exception becomes a permanent state. Chapter 10 goes deeper into how to negotiate such exceptions organizationally with development teams.

Hands-on Lab: Scanner chain with triage and thresholds

You work on a copy of the demo-app, so that the original version stays unchanged for other chapters.

Step 1: Prepare the working directory

mkdir -p ~/seclab/kap08 && cd ~/seclab/kap08
cp -r ~/seclab/demo-app ./demo-app
cd demo-app && git init -q && git add -A && git commit -qm "Ausgangsstand"
ls

Expected output (order may differ):

Dockerfile  app.py  k8s  requirements.txt

Check that the tools are there:

semgrep --version && trivy --version | head -1 && checkov --version

You should see three version numbers. If one is missing, go back to Chapter 0.

Step 2: Build in a vulnerability

Now you deliberately build in a command injection. In app.py, add the following above the if __name__ ... block (if present, otherwise at the end):

import subprocess
from flask import request


@app.route("/ping")
def ping():
    host = request.args.get("host", "127.0.0.1")
    # DELIBERATELY INSECURE - for the lab only!
    out = subprocess.run(
        f"ping -c 1 {host}", shell=True, capture_output=True, text=True
    )
    return {"output": out.stdout}

Why is this dangerous? With shell=True, a shell interprets the string. If someone calls /ping?host=127.0.0.1;cat /etc/passwd, the shell executes both commands.

Step 3: SAST with Semgrep

cd ~/seclab/kap08/demo-app
semgrep scan --config p/python --metrics=off .

p/python is a rule pack from the Semgrep registry; Semgrep needs internet access the first time for this. --metrics=off prevents usage data from being sent.

Expected output (shortened; rule IDs may differ depending on the rule version):

    app.py
   > python.lang.security.audit.subprocess-shell-true.subprocess-shell-true
          Found 'subprocess' function 'run' with 'shell=True'. This is dangerous
          because this call will spawn the command using a shell process. ...

 Ran ... rules on ... files: 1+ findings.

Depending on the rule version, Semgrep additionally reports a Flask-specific rule that detects that request.args flows into the command. This is called taint analysis: the scanner tracks whether an input (source) reaches a dangerous place (sink) unchecked.

Now as a report and with an exit code for the pipeline:

mkdir -p ../reports
semgrep scan --config p/python --metrics=off --error \
  --sarif --output ../reports/semgrep.sarif .
echo "Exit-Code: $?"

Expected: Exit-Code: 1. Without --error, Semgrep would exit with 0 even when there are findings.

In offline operation and in the pipeline (Step 12 and Chapter 11), there is no Semgrep registry. There you use your own rules, versioned as YAML in the repository. For this, create the .semgrep/ directory with two rules: one for exactly the finding from before, and one against disabled TLS verification in requests, which the demo-app ships with:

mkdir -p .semgrep
cat > .semgrep/python-sicherheit.yaml <<'EOF'
rules:
  - id: subprocess-shell-true
    languages: [python]
    severity: ERROR
    message: >-
      subprocess mit shell=True: Eingaben koennen weitere Befehle einschleusen.
      Argumente als Liste uebergeben, ohne shell=True.
    pattern: subprocess.$FUNC(..., shell=True, ...)
  - id: requests-ohne-tls-pruefung
    languages: [python]
    severity: ERROR
    message: >-
      verify=False schaltet die Zertifikatspruefung ab.
      Interne CA mit verify="/pfad/zur/ca.pem" angeben.
    pattern: requests.$FUNC(..., verify=False, ...)
EOF
semgrep scan --config .semgrep/ --metrics=off --error .
echo "Exit-Code: $?"

Expected: exactly one finding in app.py for the rule subprocess-shell-true (Semgrep prefixes the ID with the path of the rule file) and Exit-Code: 1. The rules are reviewed like code in the merge request. Chapter 11 uses exactly this directory.

Fix the vulnerability as a test by passing the arguments as a list:

    out = subprocess.run(
        ["ping", "-c", "1", host], capture_output=True, text=True
    )

Run both scans (p/python and .semgrep/) again. The shell=True rule disappears, and the scan with .semgrep/ exits with exit code 0. Leave this corrected version in place for the next steps.

Remember: Without a shell, there are no shell metacharacters such as ; or |. A real application would additionally validate host against a pattern for host names or IP addresses.

Step 4: SCA with trivy fs

cd ~/seclab/kap08/demo-app
trivy fs --scanners vuln .

Expected output (shortened; the number and exact CVEs depend on the DB state):

requirements.txt (pip)
Total: 3 (UNKNOWN: 0, LOW: 0, MEDIUM: 3, HIGH: 0, CRITICAL: 0)

| Library  | Vulnerability  | Severity | Status | Installed | Fixed Version |
|----------|----------------|----------|--------|-----------|---------------|
| requests | CVE-2023-32681 | MEDIUM   | fixed  | 2.25.0    | 2.31.0        |
| requests | CVE-2024-35195 | MEDIUM   | fixed  | 2.25.0    | 2.32.0        |
| requests | CVE-2024-47081 | MEDIUM   | fixed  | 2.25.0    | 2.32.4        |

The important columns are Status and Fixed Version. fixed means: there is a corrected version. That is exactly what --ignore-unfixed targets later.

Now with a threshold:

trivy fs --scanners vuln --severity HIGH,CRITICAL --exit-code 1 .
echo "Exit-Code: $?"
trivy fs --scanners vuln --severity MEDIUM,HIGH,CRITICAL --exit-code 1 .
echo "Exit-Code: $?"

Expected: Exit-Code: 0 for the first call (no HIGH/CRITICAL findings in requests), Exit-Code: 1 for the second. You see: the threshold decides whether the same finding stops the pipeline.

Step 5: Image scan with thresholds

Build the image (with the corrected app.py) and push it to reg-build:

docker build -t localhost:5001/demo-app:1.0.1 .
docker push localhost:5001/demo-app:1.0.1
crane digest localhost:5001/demo-app:1.0.1

Expected: output of the form sha256:…. Keep the digest in a variable:

DIGEST=$(crane digest localhost:5001/demo-app:1.0.1)
IMG="localhost:5001/demo-app@${DIGEST}"

First a full scan as a report, without aborting:

trivy image --format json --output ../reports/trivy-image.json "$IMG"
jq '[.Results[].Vulnerabilities // [] | .[]] | group_by(.Severity)
  | map({(.[0].Severity): length}) | add' ../reports/trivy-image.json

Expected output (numbers vary with the base image and DB state):

{
  "LOW": 50,
  "MEDIUM": 12,
  "HIGH": 3,
  "CRITICAL": 0
}

Most findings come from Debian packages of the base image python:3.12-slim. Now the gate:

trivy image --severity HIGH,CRITICAL --ignore-unfixed --exit-code 1 "$IMG"
echo "Exit-Code: $?"

With --ignore-unfixed, all findings without an available fix drop out. With a freshly pulled base image, the result is often Exit-Code: 0. For comparison, run the command without --ignore-unfixed and count how many findings would block only because Debian does not have a patch yet.

Finally, the SARIF report from the JSON you already generated, without scanning again:

trivy convert --format sarif --output ../reports/trivy-image.sarif \
  ../reports/trivy-image.json

Tip: trivy convert has existed since Trivy 0.49. With older versions, you scan a second time with --format sarif.

Step 6: IaC scan with trivy config

cd ~/seclab/kap08/demo-app
trivy config .

Trivy detects the Dockerfile and Kubernetes manifests automatically. Expected output (shortened, depending on the content of your manifests from Chapter 0):

Dockerfile (dockerfile)
Tests: 2x (SUCCESSES: 2x, FAILURES: 1)
LOW: Add HEALTHCHECK instruction in your Dockerfile
...
k8s/deployment.yaml (kubernetes)
Failures: ... (LOW: ..., MEDIUM: ..., HIGH: ...)
MEDIUM: Container 'demo-app' of Deployment 'demo-app' should set
        'securityContext.readOnlyRootFilesystem' to true
...

Every message has an ID, for example for the Dockerfile check “HEALTHCHECK missing” or the Kubernetes check “readOnlyRootFilesystem”. Depending on the Trivy version, the ID looks like AVD-KSV-0014 or KSV014. Note down the IDs exactly as your version prints them; you will need them shortly for exceptions.

The gate call with a threshold:

trivy config --severity HIGH,CRITICAL --exit-code 1 .
echo "Exit-Code: $?"

If your Deployment from Chapter 0 is already hardened (non-root, no privilege escalation, capabilities dropped), the exit code is 0. Otherwise, this is where you get your first real tasks.

Step 7: Checkov on Kubernetes and Terraform

Checkov checks the same manifests with its own rule set. Two scanners often see different things.

checkov -d k8s/ --compact --quiet

--quiet shows only failed checks, --compact leaves out code excerpts. Expected output (shortened):

kubernetes scan results:

Passed checks: 70, Failed checks: 12, Skipped checks: 0

Check: CKV_K8S_43: "Image should use digest"
        FAILED for resource: Deployment.demo.demo-app
        File: /deployment.yaml:1-40
Check: CKV_K8S_14: "Image Tag should be fixed - not latest or blank"
...

Now a small Terraform example. You need neither Terraform nor a cloud account for this; Checkov only reads the code.

mkdir -p ~/seclab/kap08/terraform
cat > ~/seclab/kap08/terraform/main.tf <<'EOF'
resource "aws_s3_bucket" "artefakte" {
  bucket = "seclab-artefakte"
}

resource "aws_s3_bucket_public_access_block" "artefakte" {
  bucket                  = aws_s3_bucket.artefakte.id
  block_public_acls       = false
  block_public_policy     = false
  ignore_public_acls      = false
  restrict_public_buckets = false
}
EOF
checkov -d ~/seclab/kap08/terraform --framework terraform --compact --quiet

Expected: several failed checks, including findings about public access (for example CKV_AWS_53 to CKV_AWS_56 for the four false values), missing access logging (CKV_AWS_18), missing versioning (CKV_AWS_21) and missing KMS encryption (CKV_AWS_145). The exact IDs depend on the Checkov version.

Set the four values to true and scan again. The public access findings disappear.

Reports for the pipeline:

cd ~/seclab/kap08/demo-app
checkov -d k8s/ --quiet -o sarif --output-file-path ../reports/
ls ../reports/

Expected: next to the previous reports, results_sarif.sarif is in the folder.

Step 8: Baseline for legacy findings

Imagine the 12 Checkov findings are legacy debt. You freeze them:

checkov -d k8s/ --create-baseline --quiet
ls -a k8s/

Expected: a new file k8s/.checkov.baseline. Now you check only against the baseline:

checkov -d k8s/ --baseline k8s/.checkov.baseline --quiet
echo "Exit-Code: $?"

Expected: Failed checks: 0 and Exit-Code: 0. Now, as a test, add privileged: true to the container’s securityContext and scan again. Now at least one new check fails (CKV_K8S_16, “Container should not be privileged”), and the exit code is 1. That is exactly the ratchet principle. Remove the line again.

Warning: The baseline belongs in the repository and is reviewed in the merge request. Whoever regenerates the baseline file can make any finding disappear with it. In GitLab, assign a code owner from the security team for this file.

Step 9: Exceptions with justification and expiry date

Suppose the triage for CVE-2024-47081 in requests concludes: the team will upgrade in the next sprint, and the risk is accepted until then. The simplest form is the classic .trivyignore:

cat > .trivyignore <<'EOF'
# Risk accepted: netrc credential leak, app does not use .netrc.
# Owner: team-demo, ticket: SEC-142, reviewed by security on 2026-10-08
CVE-2024-47081 exp:2026-12-31
EOF
trivy fs --scanners vuln .

Expected: CVE-2024-47081 no longer shows up in the table. After December 31, 2026, the entry automatically stops working and the finding comes back.

The YAML variant is more structured. It allows paths and a justification in the statement field:

cat > .trivyignore.yaml <<'EOF'
vulnerabilities:
  - id: CVE-2024-47081
    paths:
      - "requirements.txt"
    statement: "Risiko akzeptiert, keine .netrc im Container. Owner team-demo, SEC-142"
    expired_at: 2026-12-31
misconfigurations:
  - id: AVD-DS-0026
    statement: "Healthcheck erfolgt per Kubernetes-Probe statt Dockerfile. SEC-143"
    expired_at: 2027-03-31
EOF
trivy fs --scanners vuln,misconfig --ignorefile .trivyignore.yaml .

Warning: The YAML ignore file is marked as experimental in Trivy and must be passed explicitly with --ignorefile. Under misconfigurations, enter the ID the way your Trivy version prints it (see Step 6). A wrong ID is silently ignored.

For Checkov, you write the exception directly on the resource. In Kubernetes manifests this is done via an annotation:

metadata:
  name: demo-app
  namespace: demo
  annotations:
    checkov.io/skip1: CKV_K8S_43=Digest wird per Kyverno erzwungen (Kap. 9), SEC-144

In Terraform via a comment inside the block:

resource "aws_s3_bucket" "artefakte" {
  # checkov:skip=CKV_AWS_144:Keine Cross-Region-Replikation noetig, Lab. SEC-145
  bucket = "seclab-artefakte"
}

Check the result:

checkov -d k8s/ --compact | grep -A2 "SKIPPED"

Expected: CKV_K8S_43 appears as SKIPPED together with its justification (“digest is enforced by Kyverno”). Checkov skips have no built-in expiry date. That is why the ticket number belongs in the justification, and the ticket carries the due date.

So that local runs and the pipeline use the same settings, you put them into a configuration file .checkov.yaml in the root directory of the repository. The keys are named like the long command line options. It should contain the baseline from Step 8 and exceptions that apply to the whole repository:

cat > .checkov.yaml <<'EOF'
# Shared Checkov settings for demo-app (Chapter 8)
# Every exception with justification and ticket; the due date is in the ticket.
baseline: k8s/.checkov.baseline
compact: true
skip-check:
  # CKV_K8S_15 (imagePullPolicy Always): images are referenced
  # by digest (Chapter 11), Always brings no benefit. SEC-147
  - CKV_K8S_15
EOF
checkov -d k8s/ --config-file .checkov.yaml
echo "Exit-Code: $?"

Expected: Failed checks: 0 and Exit-Code: 0. CKV_K8S_15 no longer shows up at all: checks excluded via skip-check do not run, unlike the annotation above, which appears as SKIPPED with a justification. That is why global skips belong only in this one file, and, like the baseline, it gets a code owner from the security team.

Step 10: Triage of a finding with EPSS and KEV

Play through the triage for CVE-2023-32681 once. First the details from the Trivy report:

trivy fs --scanners vuln --format json . \
  | jq '.Results[].Vulnerabilities[]
        | select(.VulnerabilityID=="CVE-2023-32681")
        | {id: .VulnerabilityID, pkg: .PkgName, fix: .FixedVersion,
           sev: .Severity, title: .Title}'

Then the exploitation probability via FIRST’s public EPSS API:

curl -s "https://api.first.org/data/v1/epss?cve=CVE-2023-32681" \
  | jq '.data[0] | {cve, epss, percentile}'

Expected: a small epss value (well below 0.1). Finally, the KEV check:

KEV_URL="https://www.cisa.gov/sites/default/files/feeds"
curl -s "${KEV_URL}/known_exploited_vulnerabilities.json" -o kev.json
jq '[.vulnerabilities[] | select(.cveID=="CVE-2023-32681")] | length' kev.json

Expected: 0; the CVE is not in the KEV catalog. Your triage note could look like this:

Finding:    CVE-2023-32681, requests 2.25.0, MEDIUM
Context:    app does not use requests with a proxy, reachable internally only
EPSS/KEV:   low / no
Fix:        requests >= 2.31.0 available
Decision:   confirmed -> fix ticket SEC-146, due in 30 days (MEDIUM)

Actually fix the finding: set a current requests version in requirements.txt (for example requests==2.32.4 or newer) and scan again. All three requests findings disappear.

Tip: In the air-gapped environment, you mirror the EPSS CSV file and the KEV catalog just like the Trivy DB, for example daily via the zone transfer from Chapter 4.

Step 11: Mirror the Trivy DB as an OCI artifact

Now you simulate offline operation. reg-build takes on the role of the internal registry mirror.

oras cp --to-plain-http \
  ghcr.io/aquasecurity/trivy-db:2 localhost:5001/aquasecurity/trivy-db:2
oras manifest fetch --plain-http localhost:5001/aquasecurity/trivy-db:2 | jq .

Expected: an OCI manifest with one layer of media type application/vnd.aquasec.trivy.db.layer.v1.tar+gzip. Now Trivy uses the mirror with a fresh, empty cache:

trivy image --cache-dir ~/seclab/kap08/cache-offline \
  --db-repository localhost:5001/aquasecurity/trivy-db:2 \
  --download-db-only
ls ~/seclab/kap08/cache-offline/db

Expected: the files metadata.json and trivy.db. From now on you scan without any network access for the DB:

trivy fs --cache-dir ~/seclab/kap08/cache-offline \
  --skip-db-update --scanners vuln .
jq . ~/seclab/kap08/cache-offline/db/metadata.json

In metadata.json you see UpdatedAt and NextUpdate. That is your checkpoint for freshness.

Remember: For complete offline operation, you mirror trivy-java-db:1 (flag --java-db-repository) and the IaC rule bundle trivy-checks (flag --checks-bundle-repository) in the same way. With --skip-java-db-update and --skip-check-update you prevent download attempts. The --offline-scan flag additionally suppresses online queries for individual language ecosystems.

Step 12: Everything in a GitLab CI stage security

Now you assemble the parts into a stage. The images come from your mirror (see Chapter 3). The security stage extends the pipeline from Chapter 1: the test and build stages, the build job, the IMAGE_NAME variable and the image-digest.txt artifact come from there. Adjust the MIRROR variable to your setup.

stages: [test, build, security]

variables:
  MIRROR: "nexus:8082"                        # Docker proxy from Chapter 3
  IMAGE_NAME: "reg-build:5000/demo-app"       # as in Chapter 1
  TRIVY_NO_PROGRESS: "true"
  TRIVY_INSECURE: "true"                      # reg-build speaks HTTP only (lab)
  TRIVY_CACHE_DIR: ".trivycache"
  TRIVY_DB_REPOSITORY: "reg-build:5000/aquasecurity/trivy-db:2"

.reports: &reports
  artifacts:
    when: always
    expire_in: 30 days
    paths: [reports/]

sast-semgrep:
  stage: security
  image: ${MIRROR}/semgrep/semgrep:1.90.0    # example version, pin by digest
  <<: *reports
  script:
    - mkdir -p reports
    - semgrep scan --config .semgrep/ --metrics=off
        --json --output reports/semgrep.json .
    - semgrep scan --config .semgrep/ --metrics=off --error
        --severity ERROR --sarif --output reports/semgrep.sarif .

sca-trivy-fs:
  stage: security
  image: { name: "${MIRROR}/aquasec/trivy:0.58.1", entrypoint: [""] }
  <<: *reports
  script:
    - mkdir -p reports
    - trivy fs --scanners vuln --format json -o reports/trivy-fs.json .
    - trivy fs --scanners vuln --skip-db-update --ignorefile .trivyignore.yaml
        --severity HIGH,CRITICAL --ignore-unfixed --exit-code 1 .

image-trivy:
  stage: security
  image: { name: "${MIRROR}/aquasec/trivy:0.58.1", entrypoint: [""] }
  needs: [build]                              # job from Chapter 1
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH   # same as the build job
  <<: *reports
  script:
    - mkdir -p reports
    - IMAGE_REF="$IMAGE_NAME@$(cat image-digest.txt)"
    - trivy image --format json -o reports/trivy-image.json "$IMAGE_REF"
    - trivy convert --format sarif -o reports/trivy-image.sarif
        reports/trivy-image.json
    - trivy image --skip-db-update --ignorefile .trivyignore.yaml
        --severity CRITICAL --ignore-unfixed --exit-code 1 "$IMAGE_REF"

iac-trivy-config:
  stage: security
  image: { name: "${MIRROR}/aquasec/trivy:0.58.1", entrypoint: [""] }
  <<: *reports
  script:
    - mkdir -p reports
    - trivy config --format sarif -o reports/trivy-config.sarif .
    - trivy config --ignorefile .trivyignore.yaml
        --severity HIGH,CRITICAL --exit-code 1 .

iac-checkov:
  stage: security
  image: { name: "${MIRROR}/bridgecrew/checkov:3", entrypoint: [""] }
  <<: *reports
  script:
    - mkdir -p reports
    - checkov -d k8s/ --config-file .checkov.yaml --quiet
        -o cli -o sarif --output-file-path console,reports/

The structure always follows the same pattern: first the full report (exit code 0, all severities, as an artifact), then the gate with a threshold. That way triage has all the information, but only the agreed findings stop the pipeline. when: always ensures that the reports are saved even when a job is red.

Note the graduated thresholds: SCA and IaC abort on HIGH and CRITICAL, the image scan initially only on CRITICAL, because the base image brings many findings that the team cannot fix directly. This graduation is a deliberate decision that you document and tighten regularly.

Locally, you check the YAML syntax with:

yq '.stages' .gitlab-ci.yml

Expected: the list test, build, security. Push to your GitLab from Chapter 1 and watch the pipeline. Via needs, image-trivy waits for the build job from Chapter 1 and reads its image-digest.txt artifact. That way it scans exactly the built image, by digest instead of by tag. Because build runs only on the default branch, the same rule applies to image-trivy; otherwise GitLab rejects the pipeline in a merge request because of a missing needs job.

Tip: GitLab only shows findings in the merge request widget if they are in GitLab’s own report format under artifacts:reports (for example sast or container_scanning). The SARIF and JSON from this lab end up as normal artifacts and are ingested by your central tool (for example DefectDojo or Dependency-Track from Chapter 7).

Step 13: Clean up

rm -rf ~/seclab/kap08/cache-offline ~/seclab/kap08/demo-app/kev.json
docker image rm localhost:5001/demo-app:1.0.1

Leave the registries reg-build and reg-ziel running; Chapter 9 needs them. The mirrored artifact aquasecurity/trivy-db:2 in reg-build can stay.

Save your progress? Sign up for free

Common pitfalls

Symptom Cause Fix
Trivy hangs on startup or aborts with a download error No internet, DB download fails Point --db-repository at the mirror, or --skip-db-update with a pre-filled cache
Scan finds no Python vulnerabilities Dependencies not pinned (flask>=3) Pin versions exactly or use a lockfile
Pipeline green although findings are reported --exit-code 1 or --error missing Always call the gate with the exit code flag
Pipeline permanently red because of the base image Findings without a fix block --ignore-unfixed, rebuild the base image regularly
Entry in .trivyignore.yaml has no effect File not passed with --ignorefile, or wrong ID format Set the flag, copy the ID exactly from the output
Semgrep: p/python cannot be loaded Registry not reachable offline Put rules into the repo as YAML, --config .semgrep/
Checkov skip is ignored Annotation in the wrong place or wrong format checkov.io/skipN: ID=Begründung (ID=justification) under metadata.annotations
Many duplicate findings Trivy and Checkov check the same thing Accept it deliberately, or assign one scanner per purpose

Exercise: Threshold concept for a legacy project

A team takes over the demo-app as a legacy project. The first scan shows: 3 MEDIUM in requests, around 60 LOW/MEDIUM and 3 HIGH in the base image (2 of them without a fix), 12 Checkov findings in k8s/. The team is supposed to go live in two weeks.

Your task:

  1. Define a threshold for each scanner that is green today but stops new problems immediately.
  2. Create a .trivyignore.yaml for a justified, time-limited entry.
  3. Write down how the thresholds will become stricter over the next two quarters.

Hint 1

For each scanner, consider: is there a built-in baseline feature? If not, how do you achieve the same effect with ignore files and expiry dates?

Hint 2

Findings without a fix do not block. You can often fix HIGH findings with a fix in the base image simply by rebuilding with a current base image. Plan a concrete date for every tightening.

Solution

One possible concept:

Scanner Start (go-live) After Q1 After Q2
Semgrep --baseline-commit on the go-live tag ERROR blocks WARNING blocks
trivy fs HIGH,CRITICAL, --ignore-unfixed + MEDIUM + MEDIUM without exceptions
trivy image CRITICAL, --ignore-unfixed HIGH,CRITICAL + KEV always
Checkov --baseline baseline halved baseline empty

For Semgrep in the merge request, you use semgrep scan --baseline-commit "$CI_MERGE_REQUEST_DIFF_BASE_SHA", so that only new findings count. The 3 MEDIUM in requests are fixed before go-live, because a fix is available and the update is small. The one HIGH vulnerability with a fix in the base image disappears with a rebuild.

Example of a time-limited entry, in case an update is not possible right away after all:

vulnerabilities:
  - id: CVE-2024-35195
    paths:
      - "requirements.txt"
    statement: "Session nutzt verify=True durchgehend. Owner team-demo, SEC-150"
    expired_at: 2026-11-30

What matters in the solution is not the exact table, but the principle: green today, no new findings, every exception with owner, ticket and date, and a plan for tightening agreed in advance.

Save your progress? Sign up for free

In practice

At a public authority or an operator of critical infrastructure (KRITIS), the topic quickly becomes bigger than the tools. A few patterns that have proven themselves:

Central scanner configuration. The security stage does not live anew in every project, but as a CI template in a central repository and is pulled in via include:. Projects may tighten thresholds, but not loosen them. Exceptions go through the ignore file in the project, which is approved by the security team via a code owner rule.

Air-gapped data sources. Trivy DB, Java DB, checks bundle, EPSS CSV and KEV JSON arrive daily via the controlled zone transfer (see Chapter 4). Every import is logged: source, digest, timestamp. A monitoring check alerts when NextUpdate in metadata.json is more than 48 hours in the past. An outdated DB is a silent outage: the pipeline stays green, but no longer sees anything new.

Deadlines by severity. Many organizations follow fixed remediation deadlines, for example KEV or CRITICAL within a few days, HIGH within 30 days, MEDIUM in the regular release cycle. The exact numbers are set by information security management; what matters is that they are agreed in writing and that the expiry date in the ignore file matches them.

Continuous scanning. An image that is clean today has a new CVE tomorrow. That is why a nightly job rescans all production image digests, ideally based on the SBOM in Dependency-Track (see Chapter 7). That way you discover new vulnerabilities in software that is not being built at the moment.

Audit trail. Auditors want to see: which version of which scanner checked which image with which DB state, and who approved which exception? That is why SARIF/JSON reports are not just kept for 30 days as pipeline artifacts, but archived together with the image digest, often as a signed attestation (see Chapters 5 and 11).

Humans in the process. The best pipeline is useless if nobody looks at the findings. A fixed weekly triage meeting with one person each from development and security, a shared board and clear responsibilities are more effective than yet another scanner. How you shape this collaboration is the topic of Chapter 10.

Interview questions

Q: What is the difference between SAST and SCA? A: SAST analyzes your own source code for insecure patterns, for example injection. SCA analyzes the included dependencies and compares their versions with vulnerability databases. SAST finds your own bugs, SCA finds known bugs in other people’s code.

Q: Why is CVSS alone not enough for prioritization? A: CVSS rates the theoretical severity, not the likelihood of exploitation. EPSS provides that likelihood, and the KEV catalog shows vulnerabilities that are actually being exploited. Then there is your own context: reachability, the function in use, protective measures.

Q: Why do people often set --ignore-unfixed? A: The team cannot fix findings without an available patch. If they block the build, the pipeline stays permanently red and loses its effect. Such findings belong in the risk register and are watched until a fix appears.

Q: How do you introduce scanners in a legacy project with hundreds of findings? A: With a baseline: freeze the current state and block only new findings. Following the ratchet principle, the baseline gets smaller with every fixed finding and never bigger. Plus a plan for when the thresholds will be tightened.

Q: How do you run Trivy in an air-gapped environment? A: The Trivy DB is an OCI artifact. I mirror it, for example with oras cp, via the controlled transfer into the internal registry and set --db-repository to this mirror, or I transport a pre-filled cache directory and use --skip-db-update. I monitor freshness via metadata.json.

Q: What belongs in an exception for a scanner finding? A: The exact ID and path, a justification, an owner, a ticket and an expiry date. In Trivy this is done with exp: in .trivyignore or expired_at in .trivyignore.yaml. Exceptions are approved in review, not entered silently.

Q: Why SARIF? A: SARIF is a standardized JSON format for analysis results. With it, different scanners deliver uniform reports that central triage and review tools can ingest without their own parsers.

Q: A scan is green. Does that mean the image is secure? A: No. Green only means that the scanner, with its data and its rules, found nothing above the threshold. False negatives arise from outdated databases, unpinned dependencies or software without package metadata. On top of that come zero-days that nobody knows about yet.

Summary

Here is what you learned:

  • SAST, SCA, image, IaC and secret scans check different things and belong at different points in the pipeline.
  • CVSS says how bad it would be, EPSS how likely it is to be exploited, KEV whether it is already happening. Your own context always comes on top.
  • Thresholds such as “HIGH,CRITICAL with a fix” keep the pipeline meaningful; findings without a fix belong in the risk register.
  • The baseline and ratchet principle make scanners introducible even in legacy projects: no new findings, legacy debt only goes down.
  • Every exception needs an ID, justification, owner, ticket and expiry date: in Trivy via exp: or expired_at, in Checkov via a skip comment with the ticket number.
  • The Trivy DB is an OCI artifact that you have to mirror, verify and monitor for freshness like any other artifact.
  • A good security stage first produces complete reports (SARIF/JSON) and then checks with clear, graduated thresholds.