Skip to content
Hands-on IT IT Drills

Day 3: Managers and Datasources

Goal: For every file in the repo, you know which manager reads it, which datasource supplies the versions, and what you need to do when Renovate overlooks a file. · Duration: approx. 90 min · Prerequisite: Day 2

Briefing

  • A manager finds files via managerFilePatterns and extracts dependencies from them (npm, gomod, dockerfile, helmv3, …).
  • A datasource supplies the available versions for a dependency (npm, go, docker, helm, github-tags, …).
  • A versioning decides what is greater, stable, or “in range” (semver, docker, helm, …).
  • Managers with a lockfile (go.sum, package-lock.json, Chart.lock) need the matching tool (go, npm, helm) to update it.
  • The kubernetes manager has no default file patterns. Without config it sees nothing.
 File           Manager        Datasource        Versioning
 go.mod     ──► gomod      ──► go (Proxy)    ──► semver
 values.yaml──► helm-values──► docker        ──► docker
 Chart.yaml ──► helmv3     ──► helm          ──► helm
 k8s/*.yaml ──► kubernetes ──► docker        ──► docker   (only with config!)
               extracts        supplies versions  compares and sorts

Drill 1: Two new repos

Today shop-api is joined by shop-worker (Go) and shop-deploy (Helm chart, Kubernetes manifests, docker-compose). Both already come with a renovate.json with config:recommended, so there is no onboarding PR. Each repo is created like shop-api on Day 0: copy, git init, commit, push – the loop only saves you typing.

source ~/renovate-lab/setup/env.sh
export RENOVATE_GITHUB_COM_TOKEN=$(gh auth token)   # optional, see Day 0
mkdir -p ~/renovate-lab/tag03 && cd ~/renovate-lab/tag03
for r in shop-worker shop-deploy; do
  cp -R ~/renovate-lab/code/tag03/$r start-$r
  git -C start-$r init -q -b main
  git -C start-$r add -A
  git -C start-$r commit -qm "Initialer Stand"
  git -C start-$r push -q forgejo:shop/$r.git main
done
cat start-shop-worker/go.mod

Expected:

module example.com/shop/shop-worker

go 1.21

require (
    github.com/google/uuid v1.3.0
    github.com/joho/godotenv v1.4.0
    github.com/pkg/errors v0.9.0
)

The interesting spots in shop-deploy (all files under ~/renovate-lab/code/tag03/shop-deploy):

# charts/shop/Chart.yaml (excerpt)
dependencies:
  - name: podinfo
    version: 6.5.0
    repository: https://stefanprodan.github.io/podinfo
# charts/shop/values.yaml (excerpt)
image:
  repository: nginx
  tag: 1.25.3
# docker-compose.yml: postgres:15.4, redis:7.0.12
# k8s/deployment.yaml: image: busybox:1.36.0
# k8s/kustomization.yaml: images: [{name: memcached, newTag: 1.6.21}]
# examples/docker-compose.yml: mailhog/mailhog:v1.0.0

go.sum is real, generated with go mod tidy. Renovate has to update it with every update (Drill 7).

Save your progress? Sign up for free

Drill 2: Which manager finds what?

--dry-run=extract stops after reading the files. With --report-type=file, Renovate also writes a JSON report that you evaluate with jq.

cd ~/renovate-lab/tag03
renovate --dry-run=extract --report-type=file --report-path=extract.json \
  shop/shop-api shop/shop-worker shop/shop-deploy > extract.log 2>&1
jq -c '.repositories[].packageFiles | keys' extract.json

Expected:

R03_KEYS

Per repo, the managers that kicked in. Under each manager is the list of files it found:

jq -r '.repositories[].packageFiles[][].packageFile' extract.json

Expected:

R03_FILES

Two files are missing: k8s/deployment.yaml (no pattern for kubernetes, Drill 4) and examples/docker-compose.yml (ignored, Drill 5). The default patterns are in the source code of each manager, for example:

M=~/renovate-lab/tools/node_modules/renovate/dist/modules/manager
for m in kubernetes helm-values docker-compose kustomize; do
  p=$(grep -o 'managerFilePatterns: .*' "$M/$m/index.js" | head -1)
  printf "%-15s %s\n" "$m" "$p"
done | cut -c1-88

Expected:

kubernetes      managerFilePatterns: [] };
helm-values     managerFilePatterns: ["/(^|/)k8s/.+\\.yaml$/"]
|/)values\\.ya?ml$/"],
docker-compose  managerFilePatterns: ["/(^|/)(?:docker-)?compose[^/]*\\.ya?ml$/"] };
kustomize       managerFilePatterns: ["/(^|/)kustomization\\.ya?ml$/"],

The complete list of all managers and their patterns is at docs.renovatebot.com/modules/manager/.

Tip: helm-values picks up every values.yaml and recognizes image: with repository/tag (optionally registry). It does not see other structures; for those there are custom managers (Day 7).

Save your progress? Sign up for free

Drill 3: Datasources and versioning

--dry-run=lookup additionally queries the datasources, but does not create branches.

renovate --dry-run=lookup --report-type=file --report-path=lookup.json \
  shop/shop-api shop/shop-worker shop/shop-deploy > lookup.log 2>&1
jq -c '.repositories["shop/shop-api"].packageFiles.npm[0].deps[0]
  | {depName, currentValue, updates: [.updates[].newVersion]}' lookup.json

Expected:

{"depName":"express","currentValue":"4.18.2","updates":["4.22.3","5.3.0"]}

Every dependency carries the updates found for it. The same for all repos, one line per dependency:

jq -r '.repositories[].packageFiles[][].deps[]
  | "\(.depName) \(.currentValue) \([.updates[]?.newVersion])"' lookup.json

Expected (your target versions will be newer):

node 20.11.0-alpine ["20.20.2","26.11.1"]
actions/checkout v3 ["v7.0.1"]
actions/setup-node v3 ["v7.1.0"]
ubuntu latest []
node 20 ["24.21.0"]
express 4.18.2 ["4.22.3","5.3.0"]
lodash ^4.17.20 []
prettier 3.0.0 ["3.9.10"]
go 1.21 []
github.com/google/uuid v1.3.0 ["v1.6.0"]
github.com/joho/godotenv v1.4.0 ["v1.5.1"]
github.com/pkg/errors v0.9.0 ["v0.9.1"]
postgres 15.4 ["15.19","18.6"]
redis 7.0.12 ["7.4.11","8.10.2"]
nginx 1.25.3 ["1.31.6"]
podinfo 6.5.0 ["6.15.0"]
memcached 1.6.21 ["1.6.45"]

Two target versions, as with postgres, mean: one minor PR and one major PR (Day 4). lodash gets none because 4.17.21 is already within the range ^4.17.20 (Day 6). Which datasource and which versioning each dependency has is in the same entries; sort -u turns that into the list of combinations:

jq -r '.repositories[].packageFiles[][].deps[] | "\(.datasource) \(.versioning)"' \
  lookup.json | sort -u

Expected:

R03_DSRC

Four managers (dockerfile, docker-compose, helm-values, kustomize) share the docker datasource. One manager can use several datasources: github-actions queries the tags for uses:, and for node-version the releases of actions/node-versions. Why does the go directive not get an update?

jq '.repositories["shop/shop-worker"].packageFiles.gomod[0].deps[0]
  | {depName, currentValue, datasource, versioning, currentVersion, updates}' \
  lookup.json

Expected:

{
  "depName": "go",
  "currentValue": "1.21",
  "datasource": "golang-version",
  "versioning": "go-mod-directive",
  "currentVersion": "1.27.2",
  "updates": []
}

The go-mod-directive versioning reads go 1.21 as a minimum version (^1.21). Every newer 1.x satisfies that, so there is nothing to replace. Day 6 shows how to raise it anyway with rangeStrategy.

Save your progress? Sign up for free

Drill 4: kubernetes needs managerFilePatterns

From now on you change shop-deploy in a local clone and push as lab-admin.

cd ~/renovate-lab/tag03
git clone -q forgejo:shop/shop-deploy.git
cd shop-deploy
cat > renovate.json <<'EOF'
{
  "$schema": "https://docs.renovatebot.com/renovate-schema.json",
  "extends": ["config:recommended"],
  "kubernetes": {
    "managerFilePatterns": ["/(^|/)k8s/.+\\.yaml$/"]
  }
}
EOF
renovate-config-validator
git commit -qam "kubernetes-Manager für k8s/ einschalten"
git push -q
renovate --dry-run=lookup --report-type=file --report-path=../k8s.json \
  shop/shop-deploy > ../k8s.log 2>&1
jq -c '.repositories[].packageFiles.kubernetes[].deps[]
  | {depName, currentValue, datasource}' ../k8s.json

Expected:

 INFO: Validating renovate.json
 INFO: Config validated successfully against 1 file(s)
R03_K8S

Between /…/ is a regex (re2); without slashes it is a glob. In JSON, \. becomes \\.. The manager reads only files with apiVersion and kind, and besides images it also finds the API version (datasource kubernetes-api). kustomization.yaml does match the pattern, but gives the kubernetes manager nothing; that is kustomize’s job.

Warning: In older Renovate versions the option was called fileMatch and was always a regex without slashes. Old guides are full of it. The validator migrates it, but aborts with --strict.

mkdir -p ~/renovate-lab/tag03/alt && cd ~/renovate-lab/tag03/alt
cat > renovate.json <<'EOF'
{
  "kubernetes": {
    "fileMatch": ["(^|/)k8s/.+\\.yaml$"]
  }
}
EOF
renovate-config-validator --strict > val.log 2>&1; echo "Exit: $?"
grep -A2 "migration necessary" val.log | cut -c1-88
cd ~/renovate-lab/tag03/shop-deploy

Expected:

Exit: 1
 WARN: Config migration necessary
       "oldConfig": {"kubernetes": {"fileMatch": ["(^|/)k8s/.+\\.yaml$"]}},
       "newConfig": {"kubernetes": {"managerFilePatterns": ["/(^|/)k8s/.+\\.yaml$/"]}}

Save your progress? Sign up for free

Drill 5: ignorePaths

Why is examples/docker-compose.yml missing? The debug log shows the effective ignorePaths.

LOG_LEVEL=debug renovate --dry-run=extract shop/shop-deploy 2>&1 \
  | grep -A9 "Found repo ignorePaths"

Expected:

DEBUG: Found repo ignorePaths (repository=shop/shop-deploy)
       "ignorePaths": [
         "**/node_modules/**",
         "**/bower_components/**",
         "**/vendor/**",
         "**/examples/**",
         "**/__tests__/**",
         "**/test/**",
         "**/tests/**",
         "**/__fixtures__/**"

Renovate’s own default is only **/node_modules/** and **/bower_components/**. The rest comes from config:recommended through the preset :ignoreModulesAndTests. ignorePaths is not mergeable: your own list completely replaces the inherited one. Try it out locally without pushing. --platform=local reads the working directory including unsaved changes (Day 2).

jq '.ignorePaths = ["charts/**"]' renovate.json > tmp.json
mv tmp.json renovate.json
renovate --platform=local --dry-run=extract --report-type=file \
  --report-path=../local.json > ../local.log 2>&1
jq -r '.repositories[].packageFiles[][].packageFile' ../local.json
git checkout -q renovate.json

Expected:

docker-compose.yml
examples/docker-compose.yml
k8s/deployment.yaml
k8s/kustomization.yaml

The chart is gone, but examples/ is back. If you want to extend instead of replace, include the inherited entries in your list.

Save your progress? Sign up for free

Drill 6: enabledManagers

enabledManagers is an allowlist: all other managers are off.

jq '.enabledManagers = ["kubernetes", "kustomize"]' renovate.json > tmp.json
mv tmp.json renovate.json
renovate --platform=local --dry-run=extract --report-type=file \
  --report-path=../local.json > ../local.log 2>&1
jq -r '.repositories[].packageFiles[][].packageFile' ../local.json
jq '.enabledManagers = ["helm", "kubernetes"]' renovate.json > tmp.json
mv tmp.json renovate.json
renovate-config-validator > ../val.log 2>&1; echo "Exit: $?"
grep -o 'enabledManagers are not supported: .*' ../val.log
git checkout -q renovate.json

Expected:

k8s/deployment.yaml
k8s/kustomization.yaml
Exit: 1
enabledManagers are not supported: \"helm\""

There is no manager called helm; it is called helmv3 (Chart.yaml) or helm-values. enabledManagers does not magically turn on the kubernetes manager either. Without managerFilePatterns it stays blind.

Tip: You can turn off individual managers more precisely with "<manager>": {"enabled": false}, for example "docker-compose": {"enabled": false}. That stays readable when Renovate gets new managers.

Save your progress? Sign up for free

Drill 7: Lockfile update with gomod

Now a real run against shop-worker. Renovate changes go.mod, then calls go and commits the new go.sum along with it.

cd ~/renovate-lab/tag03
LOG_LEVEL=debug renovate shop/shop-worker > worker.log 2>&1
grep -E "^ INFO: PR created|Falling back|\"command\"|updatedArtifacts" worker.log \
  | cut -c1-80
curl -s -u "$ADMIN" "$FORGEJO/api/v1/repos/shop/shop-worker/pulls?state=open" \
  | jq -r '.[] | "#\(.number) \(.title)"'
curl -s -u "$ADMIN" "$FORGEJO/api/v1/repos/shop/shop-worker/pulls/1.diff" \
  | grep -E '^[-+][^-+]' | cut -c1-72

Expected:

DEBUG: Falling back to binarySource=global (repository=shop/shop-worker, branch=
       "command": "go get -t ./...",
       "updatedArtifacts": ["go.sum"]
 INFO: PR created (repository=shop/shop-worker, branch=renovate/github.com-pkg-e
DEBUG: Falling back to binarySource=global (repository=shop/shop-worker, branch=
       "command": "go get -t ./...",
       "updatedArtifacts": ["go.sum"]
 INFO: PR created (repository=shop/shop-worker, branch=renovate/github.com-googl
#2 Update module github.com/google/uuid to v1.6.0
#1 Update module github.com/pkg/errors to v0.9.1
-   github.com/pkg/errors v0.9.0
+   github.com/pkg/errors v0.9.1
+github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLG
+github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDR

Three observations:

  • Only two PRs, although three updates are pending: that is prHourlyLimit 2 from the defaults (Day 4). Issue #3 is the Dependency Dashboard.
  • binarySource defaults to install. Outside the official Renovate image (containerbase), Renovate falls back to global and uses the go from your PATH, no matter which Go version go.mod requires.
  • Renovate calls go get, not go mod tidy. The old checksums for v0.9.0 stay in go.sum. Day 13 shows how to clean up with postUpdateOptions: ["gomodTidy"].

Save your progress? Sign up for free

Drill 8: No tool, no lockfile

What happens if go is missing? A placeholder script in front of the real go in your PATH simulates that. --pr-hourly-limit=0 lifts the limit for this one run.

mkdir -p nogo
cat > nogo/go <<'EOF'
#!/bin/sh
echo "go: command not found (simuliert)" >&2
exit 127
EOF
chmod +x nogo/go
PATH="$PWD/nogo:$PATH" renovate --pr-hourly-limit=0 shop/shop-worker 2>&1 \
  | grep -A1 -E "PR created|WARN" | cut -c1-88
PR=$(curl -s -u "$ADMIN" "$FORGEJO/api/v1/repos/shop/shop-worker/pulls?state=open" \
  | jq '[.[] | select(.title | test("godotenv"))][0].number'); echo "PR #$PR"
SHA=$(curl -s -u "$ADMIN" "$FORGEJO/api/v1/repos/shop/shop-worker/pulls/$PR" \
  | jq -r .head.sha)
curl -s -u "$ADMIN" "$FORGEJO/api/v1/repos/shop/shop-worker/commits/$SHA/status" \
  | jq -r '.statuses[] | "\(.context): \(.status)"'
curl -s -u "$ADMIN" "$FORGEJO/api/v1/repos/shop/shop-worker/issues/$PR/comments" \
  | jq -r '.[0].body' | head -1

Expected:

 INFO: PR created (repository=shop/shop-worker, branch=renovate/github.com-joho-godotenv
       "pr": 4,
--
 WARN: artifactErrors (repository=shop/shop-worker, branch=renovate/github.com-joho-godo
       "artifactErrors": [{"fileName": "go.sum", "stderr": "go: command not found (simul
PR #4
renovate/artifacts: failure
### ⚠️ Artifact update problem

The PR exists anyway, but with a red renovate/artifacts status and a comment with the error message. Renovate retries only on certain occasions, for example when the PR title starts with rebase!:

curl -s -u "$ADMIN" -X PATCH -H 'Content-Type: application/json' \
  -d '{"title":"rebase!"}' "$FORGEJO/api/v1/repos/shop/shop-worker/pulls/$PR" \
  | jq -r .title
renovate shop/shop-worker 2>&1 | grep -E "Branch updated|PR updated" | cut -c1-60
R="$FORGEJO/api/v1/repos/shop/shop-worker"
curl -s -u "$ADMIN" "$R/pulls/$PR" | jq -r .title
curl -s -u "$ADMIN" "$R/raw/go.sum?ref=renovate/github.com-joho-godotenv-1.x" \
  | grep -c 'godotenv v1.5.1'
curl -s -u "$ADMIN" "$R/issues/$PR/comments" | jq length

Expected:

rebase!
 INFO: Branch updated (repository=shop/shop-worker, branch=r
 INFO: PR updated (repository=shop/shop-worker, branch=renov
Update module github.com/joho/godotenv to v1.5.1
2
0

Title repaired, go.sum contains two new lines, and the error comment is gone. Which managers need a tool at all:

Manager Artifact Tool
npm package-lock.json npm (with Node)
gomod go.sum, possibly vendor/ go
helmv3 Chart.lock, only if present helm
kustomize Helm charts, only with kustomizeInflateHelmCharts helm
dockerfile, docker-compose, kubernetes, helm-values, github-actions none none

binarySource is a global option (Day 9): global uses the PATH, install installs the matching version on demand in the official image, and docker starts sidecar containers.

Save your progress? Sign up for free

Checkpoint

go is in your PATH, and shop-deploy has a pattern for the kubernetes manager:

cd ~/renovate-lab/tag03
R="$FORGEJO/api/v1/repos/shop"
command -v go
curl -s -u "$ADMIN" "$R/shop-deploy/raw/renovate.json" \
  | jq -c .kubernetes.managerFilePatterns

Expected:

R03_KP1

PR #1 in shop-worker also updates go.sum:

curl -s -u "$ADMIN" "$R/shop-worker/pulls/1/files" | jq -r '.[].filename'

Expected:

go.mod
go.sum

PR #4 has no failed status (artifact error):

SHA=$(curl -s -u "$ADMIN" "$R/shop-worker/pulls/4" | jq -r .head.sha)
curl -s -u "$ADMIN" "$R/shop-worker/commits/$SHA/status" | jq -r .state

Expected:

R03_KP2

Challenge

From now on, the platform team stores CronJobs under deploy/manifests/*.yml (extension .yml!). Copy cronjob.yml from the companion code there and make sure Renovate finds the image alpine:3.18.4 without losing the existing pattern for k8s/.

Hint: managerFilePatterns takes several entries, regex and glob mixed.

Solution

Check the starting point:

cd ~/renovate-lab/tag03/shop-deploy
mkdir -p deploy/manifests
cp ~/renovate-lab/code/tag03/challenge/deploy/manifests/cronjob.yml deploy/manifests/
renovate --platform=local --dry-run=extract --report-type=file \
  --report-path=../local.json > ../local.log 2>&1
jq -r '.repositories[].packageFiles[][].packageFile' ../local.json | grep -c manifests

Expected: no match.

0

Add a second pattern, this time as a glob:

cat > renovate.json <<'EOF'
{
  "$schema": "https://docs.renovatebot.com/renovate-schema.json",
  "extends": ["config:recommended"],
  "kubernetes": {
    "managerFilePatterns": [
      "/(^|/)k8s/.+\\.yaml$/",
      "deploy/manifests/*.yml"
    ]
  }
}
EOF
renovate-config-validator
git add -A
git commit -qm "CronJob in deploy/manifests erfassen"
git push -q
renovate --dry-run=lookup --report-type=file --report-path=../ch.json \
  shop/shop-deploy > ../ch.log 2>&1
jq -c '.repositories[].packageFiles.kubernetes[].deps[]
  | {depName, currentValue, datasource}' ../ch.json

Expected:

 INFO: Validating renovate.json
 INFO: Config validated successfully against 1 file(s)
R03_CH

A pattern "/\\.ya?ml$/" for everything would be more convenient, but would make the manager parse every YAML file, including Helm templates with {{ … }}. Keep the patterns narrow.

Save your progress? Sign up for free

Enterprise Note

  • For Kubernetes, Kustomize, and Argo CD repos, create an org preset with managerFilePatterns (Day 11) instead of copying the patterns into every repo.
  • After onboarding, check with --dry-run=extract whether really all files are detected. Otherwise nobody notices an overlooked manifest folder.
  • Run Renovate in the official image with binarySource=install. Then the Go, Node, or Helm version matches the project instead of the runner.
  • Your own ignorePaths replace the inherited list. Deliberately add vendor/, test/, and so on back in, otherwise Renovate opens PRs against test data.
  • enabledManagers is suitable for pilot phases (“Docker images only at first”). Keep the list in the preset so that expanding it remains a central step.