Day 3: Filling and Maintaining the Catalog
Goal: Fill the catalog from configuration, collection locations and automatic repo discovery, and confidently handle errors, orphans and deletions. · Duration: approx. 90 min · Prerequisite: Day 2
Briefing
- Every entity comes from a Location. Locations come
from three sources: registered via the API/GUI (Day 1), statically from
catalog.locations, or from an entity provider such as the Gitea provider. - The catalog processes everything in loops: read, check
(
catalog.rules), store. Errors end up on the entity, not in the author’s terminal. - If an entity disappears from its source, it becomes an orphan.
- Prerequisite: the portal is running, the repos
kranich/sendungs-api,kranich/tracking-webandkranich/sendungsverfolgungare registered via the API, andcatalog.rulesfrom Day 2 additionally allowsDomain.
app-config.local.yaml Forgejo (kranich/*)
┌──────────────────────┐ ┌──────────────────────────────┐
│ catalog.locations ───┼────────►│ katalog/locations.yaml │
│ │ │ └─ extern/*.yaml │
│ catalog.providers │ 1/min │ sendungs-api/catalog-info │
│ gitea.kranich ─────┼────────►│ tracking-web/catalog-info │
└──────────────────────┘ │ sendungsverfolgung/… … │
POST /api/catalog/locations └──────────────────────────────┘
(Day 1/2, replaced today)
Drill 1: Where Does an Entity Come From?
source ~/backstage-lab/setup/env.sh
mkdir -p ~/backstage-lab/tag03
bs catalog/locations | jq -r '.[].data | "\(.type) \(.target)"'Expected:
url http://127.0.0.1:3300/kranich/sendungs-api/src/branch/main/catalog-info.yaml
url http://127.0.0.1:3300/kranich/tracking-web/src/branch/main/catalog-info.yaml
url http://127.0.0.1:3300/kranich/sendungsverfolgung/src/branch/main/catalog-info.yaml
/locations only shows the locations created via the API
or Register Existing Component – the example data from
app-config.yaml is missing. The origin of each entity is in
its annotations; ancestry gives you the chain leading
there:
bs catalog/entities/by-name/component/default/tracking-web \
| jq '.metadata.annotations | with_entries(select(.key | test("managed-by")))'
bs catalog/entities/by-name/component/default/tracking-web/ancestry \
| jq -r '.items[] | "\(.entity.kind):\(.entity.metadata.name)"'Expected (shortened):
{
"backstage.io/managed-by-location": "url:http://127.0.0.1:3300/kranich/tracking-web…",
"backstage.io/managed-by-origin-location": "url:http://127.0.0.1:3300/kranich/track…"
}
Component:tracking-web
Location:generated-312510c429b29856fc16cb8e02bb87db6e1c7ad3
In the browser:
- Open
http://localhost:3000/catalog/default/component/tracking-web. - Click ⋮ at the top right.
Expected: the menu with Unregister entity, Inspect entity and Copy entity URL.
- Choose Inspect entity, then the Ancestry tab.
Expected: above tracking-web hangs a location
generated-… that the catalog created during
registration.
Why? Ancestry is the processing chain (who created whom), not the relations from Day 2. Raw YAML shows the entity the way the catalog stored it – including all annotations the catalog added.
Save your progress? Sign up for free
Drill 2: Collection Location in the Repo kranich/katalog
External services have no repo of their own. They end up as
individual files in a catalog repo, and a Location
entity with spec.targets collects them:
cp -R ~/backstage-lab/code/tag03/katalog ~/backstage-lab/tag03/katalog
cat ~/backstage-lab/tag03/katalog/locations.yaml
cat ~/backstage-lab/tag03/katalog/extern/paketdienst.yamlExpected:
apiVersion: backstage.io/v1alpha1
kind: Location
metadata:
name: kranich-extern
description: Externe Dienste ohne eigenes Repo
spec:
targets:
- ./extern/paketdienst.yaml
- ./extern/kartendienst.yaml
apiVersion: backstage.io/v1alpha1
kind: Resource
metadata:
name: paketdienst
description: Externer Paketdienst (SaaS), liefert Zustellereignisse
links:
- url: https://status.paketdienst.example
title: Statusseite
spec:
type: external-service
owner: team-sendung
system: sendungsverfolgungkartendienst.yaml (map service) works the same way
(owner team-web). Relative targets resolve
relative to the file locations.yaml. The file is
deliberately not called catalog-info.yaml – you will see
why in Drill 4.
~/backstage-lab/setup/mkrepo.sh kranich/katalog ~/backstage-lab/tag03/katalogExpected:
✔ kranich/katalog angelegt (http://127.0.0.1:3300/kranich/katalog)
Now put the location statically into the configuration. The file is
rewritten completely (state of Day 2 plus the locations
block):
cat > ~/backstage-lab/portal/app-config.local.yaml <<'EOF'
app:
title: Kranich Entwicklerportal
organization:
name: Kranich Logistik
backend:
auth:
externalAccess:
- type: static
options:
token: ${BACKSTAGE_TOKEN}
subject: lab-cli
database:
client: better-sqlite3
connection:
directory: ${LAB}/data
integrations:
gitea:
- host: 127.0.0.1:3300
baseUrl: http://127.0.0.1:3300
username: backstage-bot
password: ${FORGEJO_TOKEN}
catalog:
rules:
- allow: [Component, System, API, Resource, Location, Domain]
locations:
# example data from create-app (arrays are replaced, not merged!)
- type: file
target: ../../examples/entities.yaml
- type: file
target: ../../examples/template/template.yaml
rules:
- allow: [Template]
- type: file
target: ../../examples/org.yaml
rules:
- allow: [User, Group]
# collection location in the repo kranich/katalog
- type: url
target: http://127.0.0.1:3300/kranich/katalog/src/branch/main/locations.yaml
EOF
~/backstage-lab/setup/portal.sh restartExpected:
✔ gestoppt
✔ Backstage läuft: http://localhost:3000 (Backend http://localhost:7007)
From here on I leave out the two ✔ lines of
portal.sh restart in the expected output.
Warning: Backstage merges configuration files, but not lists:
catalog.locationsfromapp-config.local.yamlcompletely replaces the list fromapp-config.yaml. Without the threeexamplesentries, the example data would be gone after the restart.
bs 'catalog/entities?filter=kind=resource' | jq -r '.[] | [.metadata.name,
.metadata.annotations["backstage.io/managed-by-location"]] | @tsv'Expected:
kartendienst url:http://127.0.0.1:3300/kranich/katalog/src/branch/main/extern/karte…
paketdienst url:http://127.0.0.1:3300/kranich/katalog/src/branch/main/extern/paket…
sendungs-db url:http://127.0.0.1:3300/kranich/sendungsverfolgung/src/branch/main/ca…
In the browser, open
http://localhost:3000/?filters[kind]=resource. Expected:
three resources, two of them from the catalog repo.
Save your progress? Sign up for free
Drill 3: Local File and catalog.rules
type: file reads from the backend’s file system – handy
for trying things out. Relative paths are resolved from
packages/backend; ${LAB} is set by
env.sh. The draft contains the missing group
team-sendung from Day 1:
cp ~/backstage-lab/code/tag03/entwurf.yaml ~/backstage-lab/tag03/entwurf.yaml
cat ~/backstage-lab/tag03/entwurf.yaml
cat >> ~/backstage-lab/portal/app-config.local.yaml <<'EOF'
# local file for trying things out (Day 3, Drill 3)
- type: file
target: ${LAB}/tag03/entwurf.yaml
EOF
~/backstage-lab/setup/portal.sh restartExpected:
apiVersion: backstage.io/v1alpha1
kind: Group
metadata:
name: team-sendung
description: Entwurf – zieht an Tag 4 nach kranich/org um
spec:
type: team
children: []The cat >> appends a fourth list entry to
catalog.locations – this works because
locations is currently the last block in the file. Does the
group arrive?
bs catalog/entities/by-name/group/default/team-sendung || echo "nicht da"
~/backstage-lab/setup/portal.sh logs 300 \
| grep -o 'Entity group:[^ ]*\|is not of an allowed kind for that location'Expected:
nicht da
Entity group:default/team-sendung
is not of an allowed kind for that location
Why? Globally, catalog.rules only allows the six kinds
from Day 2 – Group is only unlocked for
examples/org.yaml via a rule on that entry. The rejected
entity appears nowhere in the GUI, only in the log (“nicht da” means
“not there”). Unlock it for this location only:
ref=$(~/backstage-lab/setup/portal.sh logs 400 | grep 'allowed kind' \
| grep -o 'location:default/generated-[0-9a-f]*' | tail -1); echo "$ref"
cat >> ~/backstage-lab/portal/app-config.local.yaml <<'EOF'
rules:
- allow: [Group]
EOF
~/backstage-lab/setup/portal.sh restart
bs catalog/refresh -X POST -H 'Content-Type: application/json' \
-d "{\"entityRef\":\"$ref\"}"
sleep 5
bs catalog/entities/by-name/group/default/team-sendung | jq -r .metadata.name
bs catalog/entities/by-name/component/default/sendungs-api | jq -c .relationsExpected (the hash depends on the file path and will differ):
location:default/generated-df36173cc9451c4a4b72bd4335d84580ef5f747f
team-sendung
[{"type":"ownedBy","targetRef":"group:default/team-sendung"},{"type":"providesApi"…
You fetch the location ID before the restart, because
portal.sh starts a fresh log with every start. Without the
refresh, the group only appears on the next regular run
(every 100–150 s). The relations warning on sendungs-api is
now gone.
Save your progress? Sign up for free
Drill 4: Auto-Discovery with the Gitea Provider
Instead of registering every repo individually, an entity provider
regularly asks Forgejo for all repos of the org and creates a location
for each one that has a catalog-info.yaml. create-app does
not ship the module:
cd ~/backstage-lab/portal
yarn --cwd packages/backend add @backstage/plugin-catalog-backend-module-gitea
grep module-gitea packages/backend/package.jsonExpected (shortened):
➤ YN0000: · Yarn 4.13.0
...
➤ YN0000: · Done with warnings in 10s 243ms
"@backstage/plugin-catalog-backend-module-gitea": "^0.1.16",
Register it in the backend, directly after the catalog’s logs module:
sed -i '' "/plugin-catalog-backend-module-logs/a\\
backend.add(import('@backstage/plugin-catalog-backend-module-gitea'));
" packages/backend/src/index.ts
git diff packages/backend/src/index.ts | grep '^+b'Expected:
+backend.add(import('@backstage/plugin-catalog-backend-module-gitea'));
The configuration keys are defined in the module itself
(dist/providers/config.cjs.js): host (must
match integrations.gitea), organization,
optionally branch and catalogPath (defaults
main and catalog-info.yaml) and a mandatory
schedule. providers is appended as a new
sub-block of catalog:
cat >> app-config.local.yaml <<'EOF'
providers:
gitea:
kranich:
host: 127.0.0.1:3300
organization: kranich
branch: main
catalogPath: catalog-info.yaml
schedule:
frequency: { minutes: 1 }
timeout: { minutes: 1 }
EOF
~/backstage-lab/setup/portal.sh restart
~/backstage-lab/setup/portal.sh logs 300 | sed 's/\x1b\[[0-9;]*m//g' \
| grep -E 'gitea-provider' | cut -c26-110Expected (shortened):
catalog info Registered scheduled task: gitea-provider:kranich:refresh, {"version":…
catalog warn Repo http://127.0.0.1:3300/kranich/katalog does not contain a catalog…
catalog warn Source gitea-provider:kranich detected conflicting entityRef location:…
catalog warn Source gitea-provider:kranich detected conflicting entityRef location:…
catalog warn Source gitea-provider:kranich detected conflicting entityRef location:…
catalog info Found 3 locations from 4 repos target="gitea-provider:kranich" class=…
Four repos, three with catalog-info.yaml –
kranich/katalog is skipped; otherwise it would deliver its
resources twice. The three conflicts are deliberately instructive: for
each repo the provider generates the same location
(generated-…) as your registration from Day 1/2. Whoever
was there first wins; the provider is rejected.
Tip: The provider reads
catalog-info.yamlvia the Forgejo API with the bot token. It won’t find repos thatbackstage-botis not allowed to see.
Save your progress? Sign up for free
Drill 5: Unregister or Delete?
First Delete: deleting an entity only removes the stored state – its location delivers it again on the next run.
uid=$(bs catalog/entities/by-name/component/default/sendungs-worker \
| jq -r .metadata.uid); echo "$uid"
bs catalog/entities/by-uid/$uid -X DELETE -w '%{http_code}\n'
bs catalog/entities/by-name/component/default/sendungs-worker || echo "weg"
until bs catalog/entities/by-name/component/default/sendungs-worker >/dev/null
do sleep 5; done
bs catalog/entities/by-name/component/default/sendungs-worker | jq -r .metadata.uidExpected (UIDs differ; waiting takes up to about 2 min):
fd0f37b3-8ef8-431f-9ed9-f5d7fb7ce7de
204
weg
fed6dc38-bec1-4d2d-a772-f2b34137fc8b
New UID, same entity (“weg” means “gone”). Something is only gone for good once its location disappears. In the browser:
- Open
http://localhost:3000/catalog/default/component/tracking-web. - ⋮ → Unregister entity.
- Expand Advanced Options.
Expected: the dialog names all entities of the location and offers two ways.
- Click Unregister Location.
Delete Entity would be the approach from the first attempt above – only useful if the file has already been deleted at the source. You remove the two remaining locations from Day 1/2 via the API:
for id in $(bs catalog/locations | jq -r '.[].data.id'); do
bs catalog/locations/$id -X DELETE -w '%{http_code}\n'
done
bs catalog/locations | jq length
until bs catalog/entities/by-name/component/default/tracking-web >/dev/null; do
sleep 5; done; echo "tracking-web ist zurück"Expected (waiting takes up to one minute):
204
204
0
tracking-web ist zurück
Why? After the unregister, the way is clear for the Gitea provider: its next run creates the same locations, this time without a conflict. From now on, no repo belongs to a manual registration anymore.
Save your progress? Sign up for free
Drill 6: Finding Processing Errors
A typical error: someone renames the OpenAPI file, and
$text points nowhere.
cd ~/backstage-lab/tag03
git clone -q http://lab-admin:[email protected]:3300/kranich/sendungs-api.git
cd sendungs-api
sed -i '' 's#\$text: ./openapi.yaml#$text: ./openapi-v2.yaml#' catalog-info.yaml
git -c user.name="Lab Admin" -c [email protected] \
commit -qam "API-Definition umbenannt" && git push -q
bs catalog/refresh -X POST -H 'Content-Type: application/json' \
-d '{"entityRef":"api:default/sendungs-api"}'
sleep 5
bs 'catalog/entities?fields=kind,metadata.name,status' | jq -r '.[]
| select(.status.items) | "\(.kind):\(.metadata.name) \(.status.items[0].message)"' \
| cut -c1-85Expected:
API:sendungs-api InputError: Processor PlaceholderProcessor threw an error while pre
The error is in the entity’s status.items. In the
browser, open
http://localhost:3000/catalog/default/api/sendungs-api.
Expected: a red error box above the page; the arrow on the right expands the stack trace.
Below it you still see the old OpenAPI definition: if processing fails, the catalog keeps the last good state. That is why an error is often only noticed days later. Repair it:
git -c user.name="Lab Admin" -c [email protected] revert --no-edit HEAD \
>/dev/null && git push -q
bs catalog/refresh -X POST -H 'Content-Type: application/json' \
-d '{"entityRef":"api:default/sendungs-api"}'
sleep 5
bs catalog/entities/by-name/api/default/sendungs-api | jq -c '.status // "ok"'Expected:
"ok"
Save your progress? Sign up for free
Drill 7: Orphans and the Refresh Interval
In Backstage 1.55, catalog.orphanStrategy: delete is the
default (it used to be keep): a job cleans up orphaned
entities every 30 s. To see an orphan, you briefly turn that off and
shorten the processing loop with catalog.processingInterval
(default: random 100–150 s):
cd ~/backstage-lab/portal
cat >> app-config.local.yaml <<'EOF'
processingInterval: { seconds: 20 }
orphanStrategy: keep
EOF
echo '# Gruppen kommen ab Tag 4 aus kranich/org' > ~/backstage-lab/tag03/entwurf.yaml
~/backstage-lab/setup/portal.sh restart
until bs 'catalog/entities?filter=metadata.annotations.backstage.io/orphan=true' \
| jq -e 'length > 0' >/dev/null; do sleep 5; done
bs 'catalog/entities?filter=metadata.annotations.backstage.io/orphan=true' \
| jq -r '.[] | "\(.kind):\(.metadata.name)"'Expected (after about 30 s):
Group:team-sendung
The location for entwurf.yaml still exists, but no
longer delivers the group. The catalog marks it with
backstage.io/orphan: "true". In the browser, open
http://localhost:3000/catalog/default/group/team-sendung.
Expected: a warning with Delete entity on the right.
Instead of clicking, you restore the default strategy and let the catalog clean up:
sed -i '' '/orphanStrategy: keep/d' app-config.local.yaml
~/backstage-lab/setup/portal.sh restart
until ! bs catalog/entities/by-name/group/default/team-sendung >/dev/null; do
sleep 5; done
~/backstage-lab/setup/portal.sh logs 300 | sed 's/\x1b\[[0-9;]*m//g' \
| grep -E 'Registered.*orphan|orphaned entities' | cut -c26-108Expected:
catalog info Registered scheduled task: catalog_orphan_cleanup, {"version":2,"caden…
catalog info Deleted 1 orphaned entities
It also works by clicking: Delete entity in the
warning, then Delete in the dialog.
processingInterval: { seconds: 20 } stays in for the lab –
that way changes in Forgejo arrive faster.
Warning: If you remove a provider from the configuration, the catalog deletes all of its entities on the next start (
catalog.orphanProviderStrategy, whose default is alsodelete). Migrate first, then remove the old provider.
Save your progress? Sign up for free
Checkpoint
The file is also available at
~/backstage-lab/code/tag03/kontrollpunkt.sh.
source ~/backstage-lab/setup/env.sh
ok() { echo "✔ $1"; }; nok() { echo "✘ $1"; }
grep -q catalog-backend-module-gitea "$PORTAL/packages/backend/src/index.ts" \
&& ok "Gitea-Provider im Backend" || nok "backend.add(...gitea) fehlt"
n=$(bs catalog/locations | jq length)
[ "$n" = 0 ] && ok "keine per API registrierten Locations mehr" \
|| nok "$n Location(s) noch per API registriert"
for c in sendungs-api tracking-web sendungs-worker; do
bs "catalog/entities/by-name/component/default/$c" >/dev/null \
&& ok "Component $c im Katalog" || nok "Component $c fehlt"
done
bs catalog/entities/by-name/location/default/kranich-extern >/dev/null \
&& ok "Sammel-Location kranich-extern" || nok "Location kranich-extern fehlt"
bs catalog/entities/by-name/resource/default/paketdienst >/dev/null \
&& ok "Resource paketdienst aus kranich/katalog" || nok "paketdienst fehlt"
e=$(bs 'catalog/entities?fields=kind,metadata.name,status' \
| jq '[.[] | select(.status.items)] | length')
[ "$e" = 0 ] && ok "keine Processing-Fehler" || nok "$e Entity(s) mit Fehlern"
o=$(bs 'catalog/entities?filter=metadata.annotations.backstage.io/orphan=true' \
| jq length)
[ "$o" = 0 ] && ok "keine Orphans" || nok "$o Orphan(s)"Expected:
✔ Gitea-Provider im Backend
✔ keine per API registrierten Locations mehr
✔ Component sendungs-api im Katalog
✔ Component tracking-web im Katalog
✔ Component sendungs-worker im Katalog
✔ Sammel-Location kranich-extern
✔ Resource paketdienst aus kranich/katalog
✔ keine Processing-Fehler
✔ keine Orphans
The complete app-config.local.yaml after this day is
available at
~/backstage-lab/code/tag03/app-config.local.yaml.
Challenge
Create the repo kranich/zustell-app with a
catalog-info.yaml (Component zustell-app,
System sendungsverfolgung) and prove via the API that it
appears in the catalog without you registering it.
Hint: The files are ready in
~/backstage-lab/code/tag03/zustell-app, and the provider
runs once per minute.
Solution
source ~/backstage-lab/setup/env.sh
~/backstage-lab/setup/mkrepo.sh kranich/zustell-app \
~/backstage-lab/code/tag03/zustell-app
s=$SECONDS; until bs catalog/entities/by-name/component/default/zustell-app \
>/dev/null; do sleep 5; done; echo "nach $((SECONDS - s)) s im Katalog"
bs catalog/entities/by-name/component/default/zustell-app \
| jq -c '{name: .metadata.name, system: .spec.system}'
bs catalog/locations | jq length
~/backstage-lab/setup/portal.sh logs 400 | sed 's/\x1b\[[0-9;]*m//g' \
| grep 'gitea-provider:kranich"' | tail -1 | cut -c26-101Expected (the waiting time is between a few seconds and a little over a minute):
✔ kranich/zustell-app angelegt (http://127.0.0.1:3300/kranich/zustell-app)
nach 35 s im Katalog
{"name":"zustell-app","system":"sendungsverfolgung"}
0
catalog info Found 4 locations from 5 repos target="gitea-provider:kranich"
bs catalog/locations stays at 0: the new
location comes from the provider. In the catalog you find the app via
the search field.
Save your progress? Sign up for free
Enterprise Note
- Discovery instead of manual registration: for GitHub/GitLab the
modules are called
catalog-backend-module-githuband-gitlaband can additionally filter by topics, branches or groups; the Gitea provider only knowsorganization. - Polling scales poorly: with hundreds of repos, use webhooks/events
(the GitHub and GitLab providers support this) and choose a generous
schedule, otherwise API rate limits kick in. - Keep
catalog.rulesrestrictive (e.g.Template,User,Grouponly from central repos) – otherwise every team can smuggle in templates or groups viacatalog-info.yaml. - Actively monitor processing errors (Day 14:
catalog-backend-module-logs, metrics), because the catalog silently shows the last good state. - Decide
orphanStrategyandorphanProviderStrategydeliberately and take a database backup before restructuring providers (Day 13).