Chapter 8: Automated scanning tools, finding triage and thresholds (SAST, SCA, image and IaC scans with Trivy, Checkov)
In this chapter you will learn:
- which kinds of scanners exist (SAST, SCA, image, IaC and secret scans) and where they belong in the pipeline
- how to prioritize findings with CVSS, EPSS and KEV instead of drowning in lists
- how to set thresholds (“fail on CRITICAL with an available fix”) and implement the baseline principle “no new findings”
- how to document exceptions with a justification and an expiry date (
.trivyignore.yaml, Checkov skip)- how to run Trivy offline with your own database mirror and combine everything into a GitLab CI stage
security
What is this about?
Picture a food safety inspection in a large commercial kitchen. One inspector checks whether the cooks work cleanly (how is the food cooked?). A second checks the delivered ingredients against a recall list (is this batch of flour known to be contaminated?). A third looks at the finished, packaged dish. A fourth checks the kitchen’s floor plan (is the cold room set correctly?). And one looks for keys that someone left stuck to the fridge.
None of these inspectors is enough alone. And if every inspector reports every little thing, the kitchen shuts down even though nothing dangerous has happened. That is why two things are needed: rules for when operations are stopped (thresholds) and someone who assesses the reports (triage).
That is exactly what this chapter is about. You unleash five kinds of
scanners on the demo-app, learn to assess the findings, and
build a pipeline stage from them that only turns red when it really
matters.
Commit / merge request Build Release
+----------------------+ +----------------------+ +-------------------+
| Secret scan (gitleaks)| | Build image | | Gate: thresholds |
| SAST (Semgrep) | ---> | Image scan (Trivy) | --> | met? |
| SCA (trivy fs) | | SBOM (Chapter 7) | | yes -> signature |
| IaC (trivy config, | +----------------------+ | no -> triage |
| Checkov) | +-------------------+
+----------------------+ |
| v
+-------------> Reports (SARIF/JSON) -----------> Triage board / tickets
The concepts in plain terms
Five kinds of scanners at a glance
The abbreviations sound similar, but they check completely different things.
| Kind | What it finds | Example tool | When in the pipeline |
|---|---|---|---|
| SAST | Insecure patterns in your own code | Semgrep | On every commit/MR |
| SCA | Known vulnerabilities in dependencies | trivy fs, Grype | On every commit/MR |
| Image scan | Vulnerabilities in OS packages and libs in the image | trivy image | After the build, before release |
| IaC scan | Misconfiguration in manifests, Terraform, Dockerfile | trivy config, Checkov | On every commit/MR |
| Secret scan | Passwords, tokens, keys in the code | gitleaks, trivy | Pre-commit and on every MR |
SAST stands for Static Application Security Testing, that is, static analysis of the source code without running it. The scanner looks for patterns like “user input ends up unchecked in a shell command”.
SCA stands for Software Composition
Analysis, that is, analysis of what the software is composed of.
The scanner reads requirements.txt,
package-lock.json or pom.xml and compares the
versions with vulnerability databases. So it does not find bugs in your
code, but known bugs in other people’s code that you ship along.
The image scan is an SCA scan of the finished
container image. It additionally sees the operating system packages from
the base image (for example openssl or libc
from Debian).
The IaC scan (Infrastructure as Code) checks configuration files: does the container run as root? Are resource limits missing? Is an S3 bucket public?
The secret scan looks for credentials. You went deeper into this topic in Chapter 6; it belongs in the stage here for completeness.
Remember: SAST finds your bugs, SCA and image scans find other people’s known bugs, IaC scans find configuration errors. No tool replaces another.
How serious is a finding? CVSS, EPSS and KEV
A scanner quickly reports hundreds of CVEs. A CVE (Common
Vulnerabilities and Exposures) is a unique number for a known
vulnerability, for example CVE-2023-32681. To decide what
comes first, there are three ratings.
| Metric | Question it answers | Range | Publisher |
|---|---|---|---|
| CVSS | How bad would it be if it happened? | 0.0 to 10.0 | FIRST (standard), NVD/vendors |
| EPSS | How likely is it to be exploited soon? | 0 to 1 (percent) | FIRST |
| KEV | Is it demonstrably being exploited already? | yes/no | CISA (US agency) |
CVSS (Common Vulnerability Scoring System) rates the theoretical severity. Scanners derive the levels from it: LOW (0.1–3.9), MEDIUM (4.0–6.9), HIGH (7.0–8.9), CRITICAL (9.0–10.0). Version 3.1 is widespread; version 4.0 is slowly gaining ground.
EPSS (Exploit Prediction Scoring System) estimates the probability that a vulnerability will be exploited in the next 30 days. A CVSS 9.8 with EPSS 0.001 is often less urgent than a CVSS 7.5 with EPSS 0.9.
KEV (Known Exploited Vulnerabilities) is a CISA catalog of vulnerabilities that are demonstrably being exploited in the wild. If a CVE is in the KEV catalog: act immediately.
Tip: A simple prioritization to start with: (1) KEV-listed, (2) CRITICAL/HIGH with high EPSS, (3) CRITICAL/HIGH with a fix, (4) the rest in the normal patch cycle.
Context also matters. A vulnerability in a library whose vulnerable function you never call is less urgent. A vulnerability in a service reachable from the internet is more urgent than one in a batch job behind three zone boundaries.
False positives and false negatives
A false positive is a finding that is not one in your context. Examples:
- Semgrep reports
shell=True, but the command is a fixed string without user input. - Trivy reports a CVE in a package that only existed in the build stage, but the image layer was attributed incorrectly.
- Checkov demands an image digest, but the digest is only enforced at deployment via Kyverno (see Chapter 9).
A false negative is worse: a real vulnerability that is not reported. This happens, for example, when the vulnerability database is outdated, when dependencies are not pinned (the scanner sees no version), or when binaries are copied into the image without package metadata.
Warning: A green scan only means: “Nothing found that the scanner knows about.” It does not mean “secure”.
Thresholds: when does the pipeline turn red?
A threshold defines which findings abort the build. Typical levels:
| Rule | Effect | Suitable for |
|---|---|---|
| Fail on CRITICAL | only critical findings abort | getting started, legacy projects |
| Fail on HIGH,CRITICAL | stricter | new projects, production |
only with a fix (--ignore-unfixed) |
ignores findings without a patch | almost always sensible |
| KEV always fails | regardless of CVSS | regulated environments |
The argument for “only with a fix”: if there is no patch, the development team can do nothing but wait. A build that stays red for weeks gets ignored or bypassed. Findings without a fix belong in a risk register and are watched, but do not block.
Baseline and ratchet: “no new findings”
In a legacy project, the first scan often finds 200 findings. If you switched to “fail on HIGH” right away, every build would be red. The solution is called a baseline: you freeze today’s state and only require that nothing new is added.
The ratchet principle (like the tool: it turns in one direction only) complements this. Every time the team fixes a legacy finding, the baseline shrinks. It never grows again.
Findings
200 |#####
150 |##### ####
100 |##### #### ###
50 |##### #### ### ##
0 +-----------------------> time
Q1 Q2 Q3 Q4 (baseline only drops, new findings = build red)
Tools support this in different ways:
- Semgrep:
--baseline-commit <ref>reports only findings that are new since this commit. - Checkov:
--create-baselinewrites a.checkov.baseline,--baseline <datei>compares against it. - Trivy: no built-in baseline; you use an ignore file with expiry dates.
SARIF: one format for all reports
SARIF (Static Analysis Results Interchange
Format) is a JSON-based OASIS standard for scan results. Semgrep,
Trivy and Checkov can write SARIF. The advantage: a triage tool, a code
review system or a central dashboard only has to understand one format.
For your own evaluations with jq, the respective native
JSON is often more convenient. That is why you store both as artifacts
in the pipeline.
Trivy offline: mirroring the vulnerability database
On startup, Trivy downloads its vulnerability database
(trivy-db). This DB is distributed as an OCI
artifact, that is, stored in a container registry like an image
(default sources: mirror.gcr.io/aquasec/trivy-db:2 and
ghcr.io/aquasecurity/trivy-db:2). For Java dependencies
there is additionally trivy-java-db:1, and for IaC check
rules the trivy-checks bundle.
In an isolated (air-gapped) environment, there is no internet. Then you have two options:
- Mirror the DB as an OCI artifact: On the internet
side, you copy
trivy-db:2into your registry mirror (Harbor, Nexus, see Chapter 3) withoras cp. You point Trivy at this mirror with--db-repository. - Transport the cache directory: You download the DB
once with
trivy image --download-db-onlyinto a cache directory, transport it across the zone boundary (see Chapter 4) and start Trivy there with--skip-db-update.
Security: The DB is itself a supply chain artifact. Whoever tampers with it can make findings disappear. Treat it like any other artifact: verify its origin, adopt it by digest, log the transfer.
Pay attention to freshness: the DB is rebuilt several times a day. A DB that is a week old misses new CVEs. Trivy warns when the DB is too old.
Triage: from finding to decision
Triage comes from emergency medicine: who gets treated first? For every finding that breaks the threshold, there are exactly three outcomes.
+-------------------+
| New finding |
+---------+---------+
|
+----------------+------------------+
v v v
+---------------+ +----------------+ +-----------------------+
| Confirmed | | False positive | | Risk accepted |
| -> fix ticket | | -> ignore with | | -> ignore with expiry |
| + due date | | reason | | + ticket + owner |
+---------------+ +----------------+ +-----------------------+
Every exception needs four pieces of information: what (ID, path), why (justification), who (responsible person or team), until when (expiry date). Without an expiry date, the exception becomes a permanent state. Chapter 10 goes deeper into how to negotiate such exceptions organizationally with development teams.
Hands-on Lab: Scanner chain with triage and thresholds
You work on a copy of the demo-app, so that the original
version stays unchanged for other chapters.
Step 1: Prepare the working directory
mkdir -p ~/seclab/kap08 && cd ~/seclab/kap08
cp -r ~/seclab/demo-app ./demo-app
cd demo-app && git init -q && git add -A && git commit -qm "Ausgangsstand"
lsExpected output (order may differ):
Dockerfile app.py k8s requirements.txt
Check that the tools are there:
semgrep --version && trivy --version | head -1 && checkov --versionYou should see three version numbers. If one is missing, go back to Chapter 0.
Step 2: Build in a vulnerability
Now you deliberately build in a command injection. In
app.py, add the following above the
if __name__ ... block (if present, otherwise at the
end):
import subprocess
from flask import request
@app.route("/ping")
def ping():
host = request.args.get("host", "127.0.0.1")
# DELIBERATELY INSECURE - for the lab only!
out = subprocess.run(
f"ping -c 1 {host}", shell=True, capture_output=True, text=True
)
return {"output": out.stdout}Why is this dangerous? With shell=True, a shell
interprets the string. If someone calls
/ping?host=127.0.0.1;cat /etc/passwd, the shell executes
both commands.
Step 3: SAST with Semgrep
cd ~/seclab/kap08/demo-app
semgrep scan --config p/python --metrics=off .p/python is a rule pack from the Semgrep registry;
Semgrep needs internet access the first time for this.
--metrics=off prevents usage data from being sent.
Expected output (shortened; rule IDs may differ depending on the rule version):
app.py
> python.lang.security.audit.subprocess-shell-true.subprocess-shell-true
Found 'subprocess' function 'run' with 'shell=True'. This is dangerous
because this call will spawn the command using a shell process. ...
Ran ... rules on ... files: 1+ findings.
Depending on the rule version, Semgrep additionally reports a
Flask-specific rule that detects that request.args flows
into the command. This is called taint analysis: the
scanner tracks whether an input (source) reaches a dangerous
place (sink) unchecked.
Now as a report and with an exit code for the pipeline:
mkdir -p ../reports
semgrep scan --config p/python --metrics=off --error \
--sarif --output ../reports/semgrep.sarif .
echo "Exit-Code: $?"Expected: Exit-Code: 1. Without --error,
Semgrep would exit with 0 even when there are findings.
In offline operation and in the pipeline (Step 12 and Chapter 11),
there is no Semgrep registry. There you use your own rules, versioned as
YAML in the repository. For this, create the .semgrep/
directory with two rules: one for exactly the finding from before, and
one against disabled TLS verification in requests, which
the demo-app ships with:
mkdir -p .semgrep
cat > .semgrep/python-sicherheit.yaml <<'EOF'
rules:
- id: subprocess-shell-true
languages: [python]
severity: ERROR
message: >-
subprocess mit shell=True: Eingaben koennen weitere Befehle einschleusen.
Argumente als Liste uebergeben, ohne shell=True.
pattern: subprocess.$FUNC(..., shell=True, ...)
- id: requests-ohne-tls-pruefung
languages: [python]
severity: ERROR
message: >-
verify=False schaltet die Zertifikatspruefung ab.
Interne CA mit verify="/pfad/zur/ca.pem" angeben.
pattern: requests.$FUNC(..., verify=False, ...)
EOF
semgrep scan --config .semgrep/ --metrics=off --error .
echo "Exit-Code: $?"Expected: exactly one finding in app.py for the rule
subprocess-shell-true (Semgrep prefixes the ID with the
path of the rule file) and Exit-Code: 1. The rules are
reviewed like code in the merge request. Chapter 11 uses exactly this
directory.
Fix the vulnerability as a test by passing the arguments as a list:
out = subprocess.run(
["ping", "-c", "1", host], capture_output=True, text=True
)Run both scans (p/python and .semgrep/)
again. The shell=True rule disappears, and the scan with
.semgrep/ exits with exit code 0. Leave this corrected
version in place for the next steps.
Remember: Without a shell, there are no shell metacharacters such as
;or|. A real application would additionally validatehostagainst a pattern for host names or IP addresses.
Step 4: SCA with trivy fs
cd ~/seclab/kap08/demo-app
trivy fs --scanners vuln .Expected output (shortened; the number and exact CVEs depend on the DB state):
requirements.txt (pip)
Total: 3 (UNKNOWN: 0, LOW: 0, MEDIUM: 3, HIGH: 0, CRITICAL: 0)
| Library | Vulnerability | Severity | Status | Installed | Fixed Version |
|----------|----------------|----------|--------|-----------|---------------|
| requests | CVE-2023-32681 | MEDIUM | fixed | 2.25.0 | 2.31.0 |
| requests | CVE-2024-35195 | MEDIUM | fixed | 2.25.0 | 2.32.0 |
| requests | CVE-2024-47081 | MEDIUM | fixed | 2.25.0 | 2.32.4 |
The important columns are Status and
Fixed Version. fixed means: there is a
corrected version. That is exactly what --ignore-unfixed
targets later.
Now with a threshold:
trivy fs --scanners vuln --severity HIGH,CRITICAL --exit-code 1 .
echo "Exit-Code: $?"
trivy fs --scanners vuln --severity MEDIUM,HIGH,CRITICAL --exit-code 1 .
echo "Exit-Code: $?"Expected: Exit-Code: 0 for the first call (no
HIGH/CRITICAL findings in requests),
Exit-Code: 1 for the second. You see: the threshold decides
whether the same finding stops the pipeline.
Step 5: Image scan with thresholds
Build the image (with the corrected app.py) and push it
to reg-build:
docker build -t localhost:5001/demo-app:1.0.1 .
docker push localhost:5001/demo-app:1.0.1
crane digest localhost:5001/demo-app:1.0.1Expected: output of the form sha256:…. Keep the digest
in a variable:
DIGEST=$(crane digest localhost:5001/demo-app:1.0.1)
IMG="localhost:5001/demo-app@${DIGEST}"First a full scan as a report, without aborting:
trivy image --format json --output ../reports/trivy-image.json "$IMG"
jq '[.Results[].Vulnerabilities // [] | .[]] | group_by(.Severity)
| map({(.[0].Severity): length}) | add' ../reports/trivy-image.jsonExpected output (numbers vary with the base image and DB state):
{
"LOW": 50,
"MEDIUM": 12,
"HIGH": 3,
"CRITICAL": 0
}
Most findings come from Debian packages of the base image
python:3.12-slim. Now the gate:
trivy image --severity HIGH,CRITICAL --ignore-unfixed --exit-code 1 "$IMG"
echo "Exit-Code: $?"With --ignore-unfixed, all findings without an available
fix drop out. With a freshly pulled base image, the result is often
Exit-Code: 0. For comparison, run the command without
--ignore-unfixed and count how many findings would block
only because Debian does not have a patch yet.
Finally, the SARIF report from the JSON you already generated, without scanning again:
trivy convert --format sarif --output ../reports/trivy-image.sarif \
../reports/trivy-image.jsonTip:
trivy converthas existed since Trivy 0.49. With older versions, you scan a second time with--format sarif.
Step 6: IaC scan with trivy config
cd ~/seclab/kap08/demo-app
trivy config .Trivy detects the Dockerfile and Kubernetes manifests automatically. Expected output (shortened, depending on the content of your manifests from Chapter 0):
Dockerfile (dockerfile)
Tests: 2x (SUCCESSES: 2x, FAILURES: 1)
LOW: Add HEALTHCHECK instruction in your Dockerfile
...
k8s/deployment.yaml (kubernetes)
Failures: ... (LOW: ..., MEDIUM: ..., HIGH: ...)
MEDIUM: Container 'demo-app' of Deployment 'demo-app' should set
'securityContext.readOnlyRootFilesystem' to true
...
Every message has an ID, for example for the Dockerfile check
“HEALTHCHECK missing” or the Kubernetes check “readOnlyRootFilesystem”.
Depending on the Trivy version, the ID looks like
AVD-KSV-0014 or KSV014. Note down the IDs
exactly as your version prints them; you will need them shortly for
exceptions.
The gate call with a threshold:
trivy config --severity HIGH,CRITICAL --exit-code 1 .
echo "Exit-Code: $?"If your Deployment from Chapter 0 is already hardened (non-root, no privilege escalation, capabilities dropped), the exit code is 0. Otherwise, this is where you get your first real tasks.
Step 7: Checkov on Kubernetes and Terraform
Checkov checks the same manifests with its own rule set. Two scanners often see different things.
checkov -d k8s/ --compact --quiet--quiet shows only failed checks, --compact
leaves out code excerpts. Expected output (shortened):
kubernetes scan results:
Passed checks: 70, Failed checks: 12, Skipped checks: 0
Check: CKV_K8S_43: "Image should use digest"
FAILED for resource: Deployment.demo.demo-app
File: /deployment.yaml:1-40
Check: CKV_K8S_14: "Image Tag should be fixed - not latest or blank"
...
Now a small Terraform example. You need neither Terraform nor a cloud account for this; Checkov only reads the code.
mkdir -p ~/seclab/kap08/terraform
cat > ~/seclab/kap08/terraform/main.tf <<'EOF'
resource "aws_s3_bucket" "artefakte" {
bucket = "seclab-artefakte"
}
resource "aws_s3_bucket_public_access_block" "artefakte" {
bucket = aws_s3_bucket.artefakte.id
block_public_acls = false
block_public_policy = false
ignore_public_acls = false
restrict_public_buckets = false
}
EOF
checkov -d ~/seclab/kap08/terraform --framework terraform --compact --quietExpected: several failed checks, including findings about public
access (for example CKV_AWS_53 to CKV_AWS_56
for the four false values), missing access logging
(CKV_AWS_18), missing versioning (CKV_AWS_21)
and missing KMS encryption (CKV_AWS_145). The exact IDs
depend on the Checkov version.
Set the four values to true and scan again. The public
access findings disappear.
Reports for the pipeline:
cd ~/seclab/kap08/demo-app
checkov -d k8s/ --quiet -o sarif --output-file-path ../reports/
ls ../reports/Expected: next to the previous reports,
results_sarif.sarif is in the folder.
Step 8: Baseline for legacy findings
Imagine the 12 Checkov findings are legacy debt. You freeze them:
checkov -d k8s/ --create-baseline --quiet
ls -a k8s/Expected: a new file k8s/.checkov.baseline. Now you
check only against the baseline:
checkov -d k8s/ --baseline k8s/.checkov.baseline --quiet
echo "Exit-Code: $?"Expected: Failed checks: 0 and
Exit-Code: 0. Now, as a test, add
privileged: true to the container’s
securityContext and scan again. Now at least one new check
fails (CKV_K8S_16, “Container should not be privileged”),
and the exit code is 1. That is exactly the ratchet principle. Remove
the line again.
Warning: The baseline belongs in the repository and is reviewed in the merge request. Whoever regenerates the baseline file can make any finding disappear with it. In GitLab, assign a code owner from the security team for this file.
Step 9: Exceptions with justification and expiry date
Suppose the triage for CVE-2024-47081 in
requests concludes: the team will upgrade in the next
sprint, and the risk is accepted until then. The simplest form is the
classic .trivyignore:
cat > .trivyignore <<'EOF'
# Risk accepted: netrc credential leak, app does not use .netrc.
# Owner: team-demo, ticket: SEC-142, reviewed by security on 2026-10-08
CVE-2024-47081 exp:2026-12-31
EOF
trivy fs --scanners vuln .Expected: CVE-2024-47081 no longer shows up in the
table. After December 31, 2026, the entry automatically stops working
and the finding comes back.
The YAML variant is more structured. It allows paths and a
justification in the statement field:
cat > .trivyignore.yaml <<'EOF'
vulnerabilities:
- id: CVE-2024-47081
paths:
- "requirements.txt"
statement: "Risiko akzeptiert, keine .netrc im Container. Owner team-demo, SEC-142"
expired_at: 2026-12-31
misconfigurations:
- id: AVD-DS-0026
statement: "Healthcheck erfolgt per Kubernetes-Probe statt Dockerfile. SEC-143"
expired_at: 2027-03-31
EOF
trivy fs --scanners vuln,misconfig --ignorefile .trivyignore.yaml .Warning: The YAML ignore file is marked as experimental in Trivy and must be passed explicitly with
--ignorefile. Undermisconfigurations, enter the ID the way your Trivy version prints it (see Step 6). A wrong ID is silently ignored.
For Checkov, you write the exception directly on the resource. In Kubernetes manifests this is done via an annotation:
metadata:
name: demo-app
namespace: demo
annotations:
checkov.io/skip1: CKV_K8S_43=Digest wird per Kyverno erzwungen (Kap. 9), SEC-144In Terraform via a comment inside the block:
resource "aws_s3_bucket" "artefakte" {
# checkov:skip=CKV_AWS_144:Keine Cross-Region-Replikation noetig, Lab. SEC-145
bucket = "seclab-artefakte"
}
Check the result:
checkov -d k8s/ --compact | grep -A2 "SKIPPED"Expected: CKV_K8S_43 appears as SKIPPED
together with its justification (“digest is enforced by Kyverno”).
Checkov skips have no built-in expiry date. That is why the ticket
number belongs in the justification, and the ticket carries the due
date.
So that local runs and the pipeline use the same settings, you put
them into a configuration file .checkov.yaml in the root
directory of the repository. The keys are named like the long command
line options. It should contain the baseline from Step 8 and exceptions
that apply to the whole repository:
cat > .checkov.yaml <<'EOF'
# Shared Checkov settings for demo-app (Chapter 8)
# Every exception with justification and ticket; the due date is in the ticket.
baseline: k8s/.checkov.baseline
compact: true
skip-check:
# CKV_K8S_15 (imagePullPolicy Always): images are referenced
# by digest (Chapter 11), Always brings no benefit. SEC-147
- CKV_K8S_15
EOF
checkov -d k8s/ --config-file .checkov.yaml
echo "Exit-Code: $?"Expected: Failed checks: 0 and
Exit-Code: 0. CKV_K8S_15 no longer shows up at
all: checks excluded via skip-check do not run, unlike the
annotation above, which appears as SKIPPED with a
justification. That is why global skips belong only in this one file,
and, like the baseline, it gets a code owner from the security team.
Step 10: Triage of a finding with EPSS and KEV
Play through the triage for CVE-2023-32681 once. First
the details from the Trivy report:
trivy fs --scanners vuln --format json . \
| jq '.Results[].Vulnerabilities[]
| select(.VulnerabilityID=="CVE-2023-32681")
| {id: .VulnerabilityID, pkg: .PkgName, fix: .FixedVersion,
sev: .Severity, title: .Title}'Then the exploitation probability via FIRST’s public EPSS API:
curl -s "https://api.first.org/data/v1/epss?cve=CVE-2023-32681" \
| jq '.data[0] | {cve, epss, percentile}'Expected: a small epss value (well below 0.1). Finally,
the KEV check:
KEV_URL="https://www.cisa.gov/sites/default/files/feeds"
curl -s "${KEV_URL}/known_exploited_vulnerabilities.json" -o kev.json
jq '[.vulnerabilities[] | select(.cveID=="CVE-2023-32681")] | length' kev.jsonExpected: 0; the CVE is not in the KEV catalog. Your
triage note could look like this:
Finding: CVE-2023-32681, requests 2.25.0, MEDIUM
Context: app does not use requests with a proxy, reachable internally only
EPSS/KEV: low / no
Fix: requests >= 2.31.0 available
Decision: confirmed -> fix ticket SEC-146, due in 30 days (MEDIUM)
Actually fix the finding: set a current requests version
in requirements.txt (for example
requests==2.32.4 or newer) and scan again. All three
requests findings disappear.
Tip: In the air-gapped environment, you mirror the EPSS CSV file and the KEV catalog just like the Trivy DB, for example daily via the zone transfer from Chapter 4.
Step 11: Mirror the Trivy DB as an OCI artifact
Now you simulate offline operation. reg-build takes on
the role of the internal registry mirror.
oras cp --to-plain-http \
ghcr.io/aquasecurity/trivy-db:2 localhost:5001/aquasecurity/trivy-db:2
oras manifest fetch --plain-http localhost:5001/aquasecurity/trivy-db:2 | jq .Expected: an OCI manifest with one layer of media type
application/vnd.aquasec.trivy.db.layer.v1.tar+gzip. Now
Trivy uses the mirror with a fresh, empty cache:
trivy image --cache-dir ~/seclab/kap08/cache-offline \
--db-repository localhost:5001/aquasecurity/trivy-db:2 \
--download-db-only
ls ~/seclab/kap08/cache-offline/dbExpected: the files metadata.json and
trivy.db. From now on you scan without any network access
for the DB:
trivy fs --cache-dir ~/seclab/kap08/cache-offline \
--skip-db-update --scanners vuln .
jq . ~/seclab/kap08/cache-offline/db/metadata.jsonIn metadata.json you see UpdatedAt and
NextUpdate. That is your checkpoint for freshness.
Remember: For complete offline operation, you mirror
trivy-java-db:1(flag--java-db-repository) and the IaC rule bundletrivy-checks(flag--checks-bundle-repository) in the same way. With--skip-java-db-updateand--skip-check-updateyou prevent download attempts. The--offline-scanflag additionally suppresses online queries for individual language ecosystems.
Step 12: Everything in a GitLab CI stage security
Now you assemble the parts into a stage. The images come from your
mirror (see Chapter 3). The security stage extends the
pipeline from Chapter 1: the test and build
stages, the build job, the IMAGE_NAME variable
and the image-digest.txt artifact come from there. Adjust
the MIRROR variable to your setup.
stages: [test, build, security]
variables:
MIRROR: "nexus:8082" # Docker proxy from Chapter 3
IMAGE_NAME: "reg-build:5000/demo-app" # as in Chapter 1
TRIVY_NO_PROGRESS: "true"
TRIVY_INSECURE: "true" # reg-build speaks HTTP only (lab)
TRIVY_CACHE_DIR: ".trivycache"
TRIVY_DB_REPOSITORY: "reg-build:5000/aquasecurity/trivy-db:2"
.reports: &reports
artifacts:
when: always
expire_in: 30 days
paths: [reports/]
sast-semgrep:
stage: security
image: ${MIRROR}/semgrep/semgrep:1.90.0 # example version, pin by digest
<<: *reports
script:
- mkdir -p reports
- semgrep scan --config .semgrep/ --metrics=off
--json --output reports/semgrep.json .
- semgrep scan --config .semgrep/ --metrics=off --error
--severity ERROR --sarif --output reports/semgrep.sarif .
sca-trivy-fs:
stage: security
image: { name: "${MIRROR}/aquasec/trivy:0.58.1", entrypoint: [""] }
<<: *reports
script:
- mkdir -p reports
- trivy fs --scanners vuln --format json -o reports/trivy-fs.json .
- trivy fs --scanners vuln --skip-db-update --ignorefile .trivyignore.yaml
--severity HIGH,CRITICAL --ignore-unfixed --exit-code 1 .
image-trivy:
stage: security
image: { name: "${MIRROR}/aquasec/trivy:0.58.1", entrypoint: [""] }
needs: [build] # job from Chapter 1
rules:
- if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH # same as the build job
<<: *reports
script:
- mkdir -p reports
- IMAGE_REF="$IMAGE_NAME@$(cat image-digest.txt)"
- trivy image --format json -o reports/trivy-image.json "$IMAGE_REF"
- trivy convert --format sarif -o reports/trivy-image.sarif
reports/trivy-image.json
- trivy image --skip-db-update --ignorefile .trivyignore.yaml
--severity CRITICAL --ignore-unfixed --exit-code 1 "$IMAGE_REF"
iac-trivy-config:
stage: security
image: { name: "${MIRROR}/aquasec/trivy:0.58.1", entrypoint: [""] }
<<: *reports
script:
- mkdir -p reports
- trivy config --format sarif -o reports/trivy-config.sarif .
- trivy config --ignorefile .trivyignore.yaml
--severity HIGH,CRITICAL --exit-code 1 .
iac-checkov:
stage: security
image: { name: "${MIRROR}/bridgecrew/checkov:3", entrypoint: [""] }
<<: *reports
script:
- mkdir -p reports
- checkov -d k8s/ --config-file .checkov.yaml --quiet
-o cli -o sarif --output-file-path console,reports/The structure always follows the same pattern: first the full
report (exit code 0, all severities, as an artifact),
then the gate with a threshold. That way triage has all
the information, but only the agreed findings stop the pipeline.
when: always ensures that the reports are saved even when a
job is red.
Note the graduated thresholds: SCA and IaC abort on HIGH and CRITICAL, the image scan initially only on CRITICAL, because the base image brings many findings that the team cannot fix directly. This graduation is a deliberate decision that you document and tighten regularly.
Locally, you check the YAML syntax with:
yq '.stages' .gitlab-ci.ymlExpected: the list test, build,
security. Push to your GitLab from Chapter 1 and watch the
pipeline. Via needs, image-trivy waits for the
build job from Chapter 1 and reads its
image-digest.txt artifact. That way it scans exactly the
built image, by digest instead of by tag. Because build
runs only on the default branch, the same rule applies to
image-trivy; otherwise GitLab rejects the pipeline in a
merge request because of a missing needs job.
Tip: GitLab only shows findings in the merge request widget if they are in GitLab’s own report format under
artifacts:reports(for examplesastorcontainer_scanning). The SARIF and JSON from this lab end up as normal artifacts and are ingested by your central tool (for example DefectDojo or Dependency-Track from Chapter 7).
Step 13: Clean up
rm -rf ~/seclab/kap08/cache-offline ~/seclab/kap08/demo-app/kev.json
docker image rm localhost:5001/demo-app:1.0.1Leave the registries reg-build and reg-ziel
running; Chapter 9 needs them. The mirrored artifact
aquasecurity/trivy-db:2 in reg-build can
stay.
Save your progress? Sign up for free
Common pitfalls
| Symptom | Cause | Fix |
|---|---|---|
| Trivy hangs on startup or aborts with a download error | No internet, DB download fails | Point --db-repository at the mirror, or
--skip-db-update with a pre-filled cache |
| Scan finds no Python vulnerabilities | Dependencies not pinned (flask>=3) |
Pin versions exactly or use a lockfile |
| Pipeline green although findings are reported | --exit-code 1 or --error missing |
Always call the gate with the exit code flag |
| Pipeline permanently red because of the base image | Findings without a fix block | --ignore-unfixed, rebuild the base image regularly |
Entry in .trivyignore.yaml has no effect |
File not passed with --ignorefile, or wrong ID
format |
Set the flag, copy the ID exactly from the output |
Semgrep: p/python cannot be loaded |
Registry not reachable offline | Put rules into the repo as YAML,
--config .semgrep/ |
| Checkov skip is ignored | Annotation in the wrong place or wrong format | checkov.io/skipN: ID=Begründung (ID=justification)
under metadata.annotations |
| Many duplicate findings | Trivy and Checkov check the same thing | Accept it deliberately, or assign one scanner per purpose |
Exercise: Threshold concept for a legacy project
A team takes over the demo-app as a legacy project. The
first scan shows: 3 MEDIUM in requests, around 60
LOW/MEDIUM and 3 HIGH in the base image (2 of them without a fix), 12
Checkov findings in k8s/. The team is supposed to go live
in two weeks.
Your task:
- Define a threshold for each scanner that is green today but stops new problems immediately.
- Create a
.trivyignore.yamlfor a justified, time-limited entry. - Write down how the thresholds will become stricter over the next two quarters.
Hint 1
For each scanner, consider: is there a built-in baseline feature? If not, how do you achieve the same effect with ignore files and expiry dates?
Hint 2
Findings without a fix do not block. You can often fix HIGH findings with a fix in the base image simply by rebuilding with a current base image. Plan a concrete date for every tightening.
Solution
One possible concept:
| Scanner | Start (go-live) | After Q1 | After Q2 |
|---|---|---|---|
| Semgrep | --baseline-commit on the go-live tag |
ERROR blocks | WARNING blocks |
| trivy fs | HIGH,CRITICAL, --ignore-unfixed |
+ MEDIUM | + MEDIUM without exceptions |
| trivy image | CRITICAL, --ignore-unfixed |
HIGH,CRITICAL | + KEV always |
| Checkov | --baseline |
baseline halved | baseline empty |
For Semgrep in the merge request, you use
semgrep scan --baseline-commit "$CI_MERGE_REQUEST_DIFF_BASE_SHA",
so that only new findings count. The 3 MEDIUM in requests
are fixed before go-live, because a fix is available and the update is
small. The one HIGH vulnerability with a fix in the base image
disappears with a rebuild.
Example of a time-limited entry, in case an update is not possible right away after all:
vulnerabilities:
- id: CVE-2024-35195
paths:
- "requirements.txt"
statement: "Session nutzt verify=True durchgehend. Owner team-demo, SEC-150"
expired_at: 2026-11-30What matters in the solution is not the exact table, but the principle: green today, no new findings, every exception with owner, ticket and date, and a plan for tightening agreed in advance.
Save your progress? Sign up for free
In practice
At a public authority or an operator of critical infrastructure (KRITIS), the topic quickly becomes bigger than the tools. A few patterns that have proven themselves:
Central scanner configuration. The security stage
does not live anew in every project, but as a CI template in a central
repository and is pulled in via include:. Projects may
tighten thresholds, but not loosen them. Exceptions go through the
ignore file in the project, which is approved by the security team via a
code owner rule.
Air-gapped data sources. Trivy DB, Java DB, checks
bundle, EPSS CSV and KEV JSON arrive daily via the controlled zone
transfer (see Chapter 4). Every import is logged: source, digest,
timestamp. A monitoring check alerts when NextUpdate in
metadata.json is more than 48 hours in the past. An
outdated DB is a silent outage: the pipeline stays green, but no longer
sees anything new.
Deadlines by severity. Many organizations follow fixed remediation deadlines, for example KEV or CRITICAL within a few days, HIGH within 30 days, MEDIUM in the regular release cycle. The exact numbers are set by information security management; what matters is that they are agreed in writing and that the expiry date in the ignore file matches them.
Continuous scanning. An image that is clean today has a new CVE tomorrow. That is why a nightly job rescans all production image digests, ideally based on the SBOM in Dependency-Track (see Chapter 7). That way you discover new vulnerabilities in software that is not being built at the moment.
Audit trail. Auditors want to see: which version of which scanner checked which image with which DB state, and who approved which exception? That is why SARIF/JSON reports are not just kept for 30 days as pipeline artifacts, but archived together with the image digest, often as a signed attestation (see Chapters 5 and 11).
Humans in the process. The best pipeline is useless if nobody looks at the findings. A fixed weekly triage meeting with one person each from development and security, a shared board and clear responsibilities are more effective than yet another scanner. How you shape this collaboration is the topic of Chapter 10.
Interview questions
Q: What is the difference between SAST and SCA? A: SAST analyzes your own source code for insecure patterns, for example injection. SCA analyzes the included dependencies and compares their versions with vulnerability databases. SAST finds your own bugs, SCA finds known bugs in other people’s code.
Q: Why is CVSS alone not enough for prioritization? A: CVSS rates the theoretical severity, not the likelihood of exploitation. EPSS provides that likelihood, and the KEV catalog shows vulnerabilities that are actually being exploited. Then there is your own context: reachability, the function in use, protective measures.
Q: Why do people often set
--ignore-unfixed? A: The team cannot fix
findings without an available patch. If they block the build, the
pipeline stays permanently red and loses its effect. Such findings
belong in the risk register and are watched until a fix appears.
Q: How do you introduce scanners in a legacy project with hundreds of findings? A: With a baseline: freeze the current state and block only new findings. Following the ratchet principle, the baseline gets smaller with every fixed finding and never bigger. Plus a plan for when the thresholds will be tightened.
Q: How do you run Trivy in an air-gapped
environment? A: The Trivy DB is an OCI artifact. I
mirror it, for example with oras cp, via the controlled
transfer into the internal registry and set --db-repository
to this mirror, or I transport a pre-filled cache directory and use
--skip-db-update. I monitor freshness via
metadata.json.
Q: What belongs in an exception for a scanner
finding? A: The exact ID and path, a justification, an
owner, a ticket and an expiry date. In Trivy this is done with
exp: in .trivyignore or
expired_at in .trivyignore.yaml. Exceptions
are approved in review, not entered silently.
Q: Why SARIF? A: SARIF is a standardized JSON format for analysis results. With it, different scanners deliver uniform reports that central triage and review tools can ingest without their own parsers.
Q: A scan is green. Does that mean the image is secure? A: No. Green only means that the scanner, with its data and its rules, found nothing above the threshold. False negatives arise from outdated databases, unpinned dependencies or software without package metadata. On top of that come zero-days that nobody knows about yet.
Summary
Here is what you learned:
- SAST, SCA, image, IaC and secret scans check different things and belong at different points in the pipeline.
- CVSS says how bad it would be, EPSS how likely it is to be exploited, KEV whether it is already happening. Your own context always comes on top.
- Thresholds such as “HIGH,CRITICAL with a fix” keep the pipeline meaningful; findings without a fix belong in the risk register.
- The baseline and ratchet principle make scanners introducible even in legacy projects: no new findings, legacy debt only goes down.
- Every exception needs an ID, justification, owner, ticket and expiry
date: in Trivy via
exp:orexpired_at, in Checkov via a skip comment with the ticket number. - The Trivy DB is an OCI artifact that you have to mirror, verify and monitor for freshness like any other artifact.
- A good security stage first produces complete reports (SARIF/JSON) and then checks with clear, graduated thresholds.