Skip to content
Hands-on IT IT Drills

Day 3: Filling and Maintaining the Catalog

Goal: Fill the catalog from configuration, collection locations and automatic repo discovery, and confidently handle errors, orphans and deletions. · Duration: approx. 90 min · Prerequisite: Day 2

Briefing

  • Every entity comes from a Location. Locations come from three sources: registered via the API/GUI (Day 1), statically from catalog.locations, or from an entity provider such as the Gitea provider.
  • The catalog processes everything in loops: read, check (catalog.rules), store. Errors end up on the entity, not in the author’s terminal.
  • If an entity disappears from its source, it becomes an orphan.
  • Prerequisite: the portal is running, the repos kranich/sendungs-api, kranich/tracking-web and kranich/sendungsverfolgung are registered via the API, and catalog.rules from Day 2 additionally allows Domain.
 app-config.local.yaml            Forgejo (kranich/*)
 ┌──────────────────────┐         ┌──────────────────────────────┐
 │ catalog.locations ───┼────────►│ katalog/locations.yaml       │
 │                      │         │   └─ extern/*.yaml           │
 │ catalog.providers    │  1/min  │ sendungs-api/catalog-info    │
 │   gitea.kranich ─────┼────────►│ tracking-web/catalog-info    │
 └──────────────────────┘         │ sendungsverfolgung/…  …      │
 POST /api/catalog/locations      └──────────────────────────────┘
   (Day 1/2, replaced today)

Drill 1: Where Does an Entity Come From?

source ~/backstage-lab/setup/env.sh
mkdir -p ~/backstage-lab/tag03
bs catalog/locations | jq -r '.[].data | "\(.type)  \(.target)"'

Expected:

url  http://127.0.0.1:3300/kranich/sendungs-api/src/branch/main/catalog-info.yaml
url  http://127.0.0.1:3300/kranich/tracking-web/src/branch/main/catalog-info.yaml
url  http://127.0.0.1:3300/kranich/sendungsverfolgung/src/branch/main/catalog-info.yaml

/locations only shows the locations created via the API or Register Existing Component – the example data from app-config.yaml is missing. The origin of each entity is in its annotations; ancestry gives you the chain leading there:

bs catalog/entities/by-name/component/default/tracking-web \
  | jq '.metadata.annotations | with_entries(select(.key | test("managed-by")))'
bs catalog/entities/by-name/component/default/tracking-web/ancestry \
  | jq -r '.items[] | "\(.entity.kind):\(.entity.metadata.name)"'

Expected (shortened):

{
  "backstage.io/managed-by-location": "url:http://127.0.0.1:3300/kranich/tracking-web…",
  "backstage.io/managed-by-origin-location": "url:http://127.0.0.1:3300/kranich/track…"
}
Component:tracking-web
Location:generated-312510c429b29856fc16cb8e02bb87db6e1c7ad3

In the browser:

  1. Open http://localhost:3000/catalog/default/component/tracking-web.
  2. Click ⋮ at the top right.

Expected: the menu with Unregister entity, Inspect entity and Copy entity URL.

The context menu of an entity
  1. Choose Inspect entity, then the Ancestry tab.

Expected: above tracking-web hangs a location generated-… that the catalog created during registration.

Entity Inspector with the ancestry of tracking-web

Why? Ancestry is the processing chain (who created whom), not the relations from Day 2. Raw YAML shows the entity the way the catalog stored it – including all annotations the catalog added.

Save your progress? Sign up for free

Drill 2: Collection Location in the Repo kranich/katalog

External services have no repo of their own. They end up as individual files in a catalog repo, and a Location entity with spec.targets collects them:

cp -R ~/backstage-lab/code/tag03/katalog ~/backstage-lab/tag03/katalog
cat ~/backstage-lab/tag03/katalog/locations.yaml
cat ~/backstage-lab/tag03/katalog/extern/paketdienst.yaml

Expected:

apiVersion: backstage.io/v1alpha1
kind: Location
metadata:
  name: kranich-extern
  description: Externe Dienste ohne eigenes Repo
spec:
  targets:
    - ./extern/paketdienst.yaml
    - ./extern/kartendienst.yaml
apiVersion: backstage.io/v1alpha1
kind: Resource
metadata:
  name: paketdienst
  description: Externer Paketdienst (SaaS), liefert Zustellereignisse
  links:
    - url: https://status.paketdienst.example
      title: Statusseite
spec:
  type: external-service
  owner: team-sendung
  system: sendungsverfolgung

kartendienst.yaml (map service) works the same way (owner team-web). Relative targets resolve relative to the file locations.yaml. The file is deliberately not called catalog-info.yaml – you will see why in Drill 4.

~/backstage-lab/setup/mkrepo.sh kranich/katalog ~/backstage-lab/tag03/katalog

Expected:

✔ kranich/katalog angelegt (http://127.0.0.1:3300/kranich/katalog)

Now put the location statically into the configuration. The file is rewritten completely (state of Day 2 plus the locations block):

cat > ~/backstage-lab/portal/app-config.local.yaml <<'EOF'
app:
  title: Kranich Entwicklerportal
organization:
  name: Kranich Logistik
backend:
  auth:
    externalAccess:
      - type: static
        options:
          token: ${BACKSTAGE_TOKEN}
          subject: lab-cli
  database:
    client: better-sqlite3
    connection:
      directory: ${LAB}/data
integrations:
  gitea:
    - host: 127.0.0.1:3300
      baseUrl: http://127.0.0.1:3300
      username: backstage-bot
      password: ${FORGEJO_TOKEN}
catalog:
  rules:
    - allow: [Component, System, API, Resource, Location, Domain]
  locations:
    # example data from create-app (arrays are replaced, not merged!)
    - type: file
      target: ../../examples/entities.yaml
    - type: file
      target: ../../examples/template/template.yaml
      rules:
        - allow: [Template]
    - type: file
      target: ../../examples/org.yaml
      rules:
        - allow: [User, Group]
    # collection location in the repo kranich/katalog
    - type: url
      target: http://127.0.0.1:3300/kranich/katalog/src/branch/main/locations.yaml
EOF
~/backstage-lab/setup/portal.sh restart

Expected:

✔ gestoppt
✔ Backstage läuft: http://localhost:3000 (Backend http://localhost:7007)

From here on I leave out the two ✔ lines of portal.sh restart in the expected output.

Warning: Backstage merges configuration files, but not lists: catalog.locations from app-config.local.yaml completely replaces the list from app-config.yaml. Without the three examples entries, the example data would be gone after the restart.

bs 'catalog/entities?filter=kind=resource' | jq -r '.[] | [.metadata.name,
  .metadata.annotations["backstage.io/managed-by-location"]] | @tsv'

Expected:

kartendienst    url:http://127.0.0.1:3300/kranich/katalog/src/branch/main/extern/karte…
paketdienst url:http://127.0.0.1:3300/kranich/katalog/src/branch/main/extern/paket…
sendungs-db url:http://127.0.0.1:3300/kranich/sendungsverfolgung/src/branch/main/ca…

In the browser, open http://localhost:3000/?filters[kind]=resource. Expected: three resources, two of them from the catalog repo.

Catalog filtered to resources

Save your progress? Sign up for free

Drill 3: Local File and catalog.rules

type: file reads from the backend’s file system – handy for trying things out. Relative paths are resolved from packages/backend; ${LAB} is set by env.sh. The draft contains the missing group team-sendung from Day 1:

cp ~/backstage-lab/code/tag03/entwurf.yaml ~/backstage-lab/tag03/entwurf.yaml
cat ~/backstage-lab/tag03/entwurf.yaml
cat >> ~/backstage-lab/portal/app-config.local.yaml <<'EOF'
    # local file for trying things out (Day 3, Drill 3)
    - type: file
      target: ${LAB}/tag03/entwurf.yaml
EOF
~/backstage-lab/setup/portal.sh restart

Expected:

apiVersion: backstage.io/v1alpha1
kind: Group
metadata:
  name: team-sendung
  description: Entwurf – zieht an Tag 4 nach kranich/org um
spec:
  type: team
  children: []

The cat >> appends a fourth list entry to catalog.locations – this works because locations is currently the last block in the file. Does the group arrive?

bs catalog/entities/by-name/group/default/team-sendung || echo "nicht da"
~/backstage-lab/setup/portal.sh logs 300 \
  | grep -o 'Entity group:[^ ]*\|is not of an allowed kind for that location'

Expected:

nicht da
Entity group:default/team-sendung
is not of an allowed kind for that location

Why? Globally, catalog.rules only allows the six kinds from Day 2 – Group is only unlocked for examples/org.yaml via a rule on that entry. The rejected entity appears nowhere in the GUI, only in the log (“nicht da” means “not there”). Unlock it for this location only:

ref=$(~/backstage-lab/setup/portal.sh logs 400 | grep 'allowed kind' \
  | grep -o 'location:default/generated-[0-9a-f]*' | tail -1); echo "$ref"
cat >> ~/backstage-lab/portal/app-config.local.yaml <<'EOF'
      rules:
        - allow: [Group]
EOF
~/backstage-lab/setup/portal.sh restart
bs catalog/refresh -X POST -H 'Content-Type: application/json' \
  -d "{\"entityRef\":\"$ref\"}"
sleep 5
bs catalog/entities/by-name/group/default/team-sendung | jq -r .metadata.name
bs catalog/entities/by-name/component/default/sendungs-api | jq -c .relations

Expected (the hash depends on the file path and will differ):

location:default/generated-df36173cc9451c4a4b72bd4335d84580ef5f747f
team-sendung
[{"type":"ownedBy","targetRef":"group:default/team-sendung"},{"type":"providesApi"…

You fetch the location ID before the restart, because portal.sh starts a fresh log with every start. Without the refresh, the group only appears on the next regular run (every 100–150 s). The relations warning on sendungs-api is now gone.

Save your progress? Sign up for free

Drill 4: Auto-Discovery with the Gitea Provider

Instead of registering every repo individually, an entity provider regularly asks Forgejo for all repos of the org and creates a location for each one that has a catalog-info.yaml. create-app does not ship the module:

cd ~/backstage-lab/portal
yarn --cwd packages/backend add @backstage/plugin-catalog-backend-module-gitea
grep module-gitea packages/backend/package.json

Expected (shortened):

➤ YN0000: · Yarn 4.13.0
...
➤ YN0000: · Done with warnings in 10s 243ms
    "@backstage/plugin-catalog-backend-module-gitea": "^0.1.16",

Register it in the backend, directly after the catalog’s logs module:

sed -i '' "/plugin-catalog-backend-module-logs/a\\
backend.add(import('@backstage/plugin-catalog-backend-module-gitea'));
" packages/backend/src/index.ts
git diff packages/backend/src/index.ts | grep '^+b'

Expected:

+backend.add(import('@backstage/plugin-catalog-backend-module-gitea'));

The configuration keys are defined in the module itself (dist/providers/config.cjs.js): host (must match integrations.gitea), organization, optionally branch and catalogPath (defaults main and catalog-info.yaml) and a mandatory schedule. providers is appended as a new sub-block of catalog:

cat >> app-config.local.yaml <<'EOF'
  providers:
    gitea:
      kranich:
        host: 127.0.0.1:3300
        organization: kranich
        branch: main
        catalogPath: catalog-info.yaml
        schedule:
          frequency: { minutes: 1 }
          timeout: { minutes: 1 }
EOF
~/backstage-lab/setup/portal.sh restart
~/backstage-lab/setup/portal.sh logs 300 | sed 's/\x1b\[[0-9;]*m//g' \
  | grep -E 'gitea-provider' | cut -c26-110

Expected (shortened):

catalog info Registered scheduled task: gitea-provider:kranich:refresh, {"version":…
catalog warn Repo http://127.0.0.1:3300/kranich/katalog does not contain a catalog…
catalog warn Source gitea-provider:kranich detected conflicting entityRef location:…
catalog warn Source gitea-provider:kranich detected conflicting entityRef location:…
catalog warn Source gitea-provider:kranich detected conflicting entityRef location:…
catalog info Found 3 locations from 4 repos target="gitea-provider:kranich" class=…

Four repos, three with catalog-info.yaml – kranich/katalog is skipped; otherwise it would deliver its resources twice. The three conflicts are deliberately instructive: for each repo the provider generates the same location (generated-…) as your registration from Day 1/2. Whoever was there first wins; the provider is rejected.

Tip: The provider reads catalog-info.yaml via the Forgejo API with the bot token. It won’t find repos that backstage-bot is not allowed to see.

Save your progress? Sign up for free

Drill 5: Unregister or Delete?

First Delete: deleting an entity only removes the stored state – its location delivers it again on the next run.

uid=$(bs catalog/entities/by-name/component/default/sendungs-worker \
  | jq -r .metadata.uid); echo "$uid"
bs catalog/entities/by-uid/$uid -X DELETE -w '%{http_code}\n'
bs catalog/entities/by-name/component/default/sendungs-worker || echo "weg"
until bs catalog/entities/by-name/component/default/sendungs-worker >/dev/null
do sleep 5; done
bs catalog/entities/by-name/component/default/sendungs-worker | jq -r .metadata.uid

Expected (UIDs differ; waiting takes up to about 2 min):

fd0f37b3-8ef8-431f-9ed9-f5d7fb7ce7de
204
weg
fed6dc38-bec1-4d2d-a772-f2b34137fc8b

New UID, same entity (“weg” means “gone”). Something is only gone for good once its location disappears. In the browser:

  1. Open http://localhost:3000/catalog/default/component/tracking-web.
  2. ⋮ → Unregister entity.
  3. Expand Advanced Options.

Expected: the dialog names all entities of the location and offers two ways.

Unregister dialog with Advanced Options expanded
  1. Click Unregister Location.

Delete Entity would be the approach from the first attempt above – only useful if the file has already been deleted at the source. You remove the two remaining locations from Day 1/2 via the API:

for id in $(bs catalog/locations | jq -r '.[].data.id'); do
  bs catalog/locations/$id -X DELETE -w '%{http_code}\n'
done
bs catalog/locations | jq length
until bs catalog/entities/by-name/component/default/tracking-web >/dev/null; do
  sleep 5; done; echo "tracking-web ist zurück"

Expected (waiting takes up to one minute):

204
204
0
tracking-web ist zurück

Why? After the unregister, the way is clear for the Gitea provider: its next run creates the same locations, this time without a conflict. From now on, no repo belongs to a manual registration anymore.

Save your progress? Sign up for free

Drill 6: Finding Processing Errors

A typical error: someone renames the OpenAPI file, and $text points nowhere.

cd ~/backstage-lab/tag03
git clone -q http://lab-admin:[email protected]:3300/kranich/sendungs-api.git
cd sendungs-api
sed -i '' 's#\$text: ./openapi.yaml#$text: ./openapi-v2.yaml#' catalog-info.yaml
git -c user.name="Lab Admin" -c [email protected] \
  commit -qam "API-Definition umbenannt" && git push -q
bs catalog/refresh -X POST -H 'Content-Type: application/json' \
  -d '{"entityRef":"api:default/sendungs-api"}'
sleep 5
bs 'catalog/entities?fields=kind,metadata.name,status' | jq -r '.[]
  | select(.status.items) | "\(.kind):\(.metadata.name)  \(.status.items[0].message)"' \
  | cut -c1-85

Expected:

API:sendungs-api  InputError: Processor PlaceholderProcessor threw an error while pre

The error is in the entity’s status.items. In the browser, open http://localhost:3000/catalog/default/api/sendungs-api.

Expected: a red error box above the page; the arrow on the right expands the stack trace.

Processing error on the entity page

Below it you still see the old OpenAPI definition: if processing fails, the catalog keeps the last good state. That is why an error is often only noticed days later. Repair it:

git -c user.name="Lab Admin" -c [email protected] revert --no-edit HEAD \
  >/dev/null && git push -q
bs catalog/refresh -X POST -H 'Content-Type: application/json' \
  -d '{"entityRef":"api:default/sendungs-api"}'
sleep 5
bs catalog/entities/by-name/api/default/sendungs-api | jq -c '.status // "ok"'

Expected:

"ok"

Save your progress? Sign up for free

Drill 7: Orphans and the Refresh Interval

In Backstage 1.55, catalog.orphanStrategy: delete is the default (it used to be keep): a job cleans up orphaned entities every 30 s. To see an orphan, you briefly turn that off and shorten the processing loop with catalog.processingInterval (default: random 100–150 s):

cd ~/backstage-lab/portal
cat >> app-config.local.yaml <<'EOF'
  processingInterval: { seconds: 20 }
  orphanStrategy: keep
EOF
echo '# Gruppen kommen ab Tag 4 aus kranich/org' > ~/backstage-lab/tag03/entwurf.yaml
~/backstage-lab/setup/portal.sh restart
until bs 'catalog/entities?filter=metadata.annotations.backstage.io/orphan=true' \
  | jq -e 'length > 0' >/dev/null; do sleep 5; done
bs 'catalog/entities?filter=metadata.annotations.backstage.io/orphan=true' \
  | jq -r '.[] | "\(.kind):\(.metadata.name)"'

Expected (after about 30 s):

Group:team-sendung

The location for entwurf.yaml still exists, but no longer delivers the group. The catalog marks it with backstage.io/orphan: "true". In the browser, open http://localhost:3000/catalog/default/group/team-sendung.

Expected: a warning with Delete entity on the right.

Orphan warning on the group page

Instead of clicking, you restore the default strategy and let the catalog clean up:

sed -i '' '/orphanStrategy: keep/d' app-config.local.yaml
~/backstage-lab/setup/portal.sh restart
until ! bs catalog/entities/by-name/group/default/team-sendung >/dev/null; do
  sleep 5; done
~/backstage-lab/setup/portal.sh logs 300 | sed 's/\x1b\[[0-9;]*m//g' \
  | grep -E 'Registered.*orphan|orphaned entities' | cut -c26-108

Expected:

catalog info Registered scheduled task: catalog_orphan_cleanup, {"version":2,"caden…
catalog info Deleted 1 orphaned entities

It also works by clicking: Delete entity in the warning, then Delete in the dialog. processingInterval: { seconds: 20 } stays in for the lab – that way changes in Forgejo arrive faster.

Warning: If you remove a provider from the configuration, the catalog deletes all of its entities on the next start (catalog.orphanProviderStrategy, whose default is also delete). Migrate first, then remove the old provider.

Save your progress? Sign up for free

Checkpoint

The file is also available at ~/backstage-lab/code/tag03/kontrollpunkt.sh.

source ~/backstage-lab/setup/env.sh
ok() { echo "✔ $1"; }; nok() { echo "✘ $1"; }
grep -q catalog-backend-module-gitea "$PORTAL/packages/backend/src/index.ts" \
  && ok "Gitea-Provider im Backend" || nok "backend.add(...gitea) fehlt"
n=$(bs catalog/locations | jq length)
[ "$n" = 0 ] && ok "keine per API registrierten Locations mehr" \
  || nok "$n Location(s) noch per API registriert"
for c in sendungs-api tracking-web sendungs-worker; do
  bs "catalog/entities/by-name/component/default/$c" >/dev/null \
    && ok "Component $c im Katalog" || nok "Component $c fehlt"
done
bs catalog/entities/by-name/location/default/kranich-extern >/dev/null \
  && ok "Sammel-Location kranich-extern" || nok "Location kranich-extern fehlt"
bs catalog/entities/by-name/resource/default/paketdienst >/dev/null \
  && ok "Resource paketdienst aus kranich/katalog" || nok "paketdienst fehlt"
e=$(bs 'catalog/entities?fields=kind,metadata.name,status' \
  | jq '[.[] | select(.status.items)] | length')
[ "$e" = 0 ] && ok "keine Processing-Fehler" || nok "$e Entity(s) mit Fehlern"
o=$(bs 'catalog/entities?filter=metadata.annotations.backstage.io/orphan=true' \
  | jq length)
[ "$o" = 0 ] && ok "keine Orphans" || nok "$o Orphan(s)"

Expected:

✔ Gitea-Provider im Backend
✔ keine per API registrierten Locations mehr
✔ Component sendungs-api im Katalog
✔ Component tracking-web im Katalog
✔ Component sendungs-worker im Katalog
✔ Sammel-Location kranich-extern
✔ Resource paketdienst aus kranich/katalog
✔ keine Processing-Fehler
✔ keine Orphans

The complete app-config.local.yaml after this day is available at ~/backstage-lab/code/tag03/app-config.local.yaml.

Challenge

Create the repo kranich/zustell-app with a catalog-info.yaml (Component zustell-app, System sendungsverfolgung) and prove via the API that it appears in the catalog without you registering it.

Hint: The files are ready in ~/backstage-lab/code/tag03/zustell-app, and the provider runs once per minute.

Solution

source ~/backstage-lab/setup/env.sh
~/backstage-lab/setup/mkrepo.sh kranich/zustell-app \
  ~/backstage-lab/code/tag03/zustell-app
s=$SECONDS; until bs catalog/entities/by-name/component/default/zustell-app \
  >/dev/null; do sleep 5; done; echo "nach $((SECONDS - s)) s im Katalog"
bs catalog/entities/by-name/component/default/zustell-app \
  | jq -c '{name: .metadata.name, system: .spec.system}'
bs catalog/locations | jq length
~/backstage-lab/setup/portal.sh logs 400 | sed 's/\x1b\[[0-9;]*m//g' \
  | grep 'gitea-provider:kranich"' | tail -1 | cut -c26-101

Expected (the waiting time is between a few seconds and a little over a minute):

✔ kranich/zustell-app angelegt (http://127.0.0.1:3300/kranich/zustell-app)
nach 35 s im Katalog
{"name":"zustell-app","system":"sendungsverfolgung"}
0
catalog info Found 4 locations from 5 repos target="gitea-provider:kranich"

bs catalog/locations stays at 0: the new location comes from the provider. In the catalog you find the app via the search field.

The zustell-app found via discovery in the catalog

Save your progress? Sign up for free

Enterprise Note

  • Discovery instead of manual registration: for GitHub/GitLab the modules are called catalog-backend-module-github and -gitlab and can additionally filter by topics, branches or groups; the Gitea provider only knows organization.
  • Polling scales poorly: with hundreds of repos, use webhooks/events (the GitHub and GitLab providers support this) and choose a generous schedule, otherwise API rate limits kick in.
  • Keep catalog.rules restrictive (e.g. Template, User, Group only from central repos) – otherwise every team can smuggle in templates or groups via catalog-info.yaml.
  • Actively monitor processing errors (Day 14: catalog-backend-module-logs, metrics), because the catalog silently shows the last good state.
  • Decide orphanStrategy and orphanProviderStrategy deliberately and take a database backup before restructuring providers (Day 13).