How We Build
Applies to: every delivery, from a two-week GenAI PoC to a multi-year platform build · Owner: the delivery tech lead
Every service page on this site assumes the standards below. A PoC uses the same repo template, pipeline, and IaC as a production system; it simply turns fewer knobs. That is how a successful PoC becomes production without a rewrite.
The standard in one picture
Repository template
Every project starts from the Lieber repo template. Layout:
.
├── .github/workflows/ # or .gitea/workflows/ or .gitlab-ci.yml
├── docs/
│ ├── adr/ # 0001-record-architecture-decisions.md, ...
│ ├── architecture/ # C4 diagrams (Structurizr DSL or Mermaid)
│ └── runbooks/ # one file per alert
├── deploy/
│ ├── base/ # Kustomize base or Helm chart
│ └── overlays/{dev,staging,prod}/
├── infra/ # OpenTofu root modules per environment
├── evals/ # AI projects: datasets, graders, thresholds
├── prompts/ # AI projects: versioned prompt templates
├── src/ ...
├── Dockerfile
├── renovate.json
├── CODEOWNERS
└── README.md # run locally in 3 commands
Required repo settings:
| Setting | Value |
|---|---|
| Default branch | main, protected |
| Merge | squash or rebase only, linear history |
| Required checks | CI workflow, at least one review from CODEOWNERS |
| Signed commits | required on main for regulated clients |
| Secrets | none in the repo; secret scanning (gitleaks) in CI and pre-commit |
| README | "run locally" in three commands or fewer |
Trunk-based development
- Branches live hours to two days, then merge to
main. No long-runningdevelopor release branches. - Unfinished work ships behind feature flags (OpenFeature SDK with a flag provider such as flagd).
mainis always deployable; every merge builds an immutable artifact that is promoted through environments, never rebuilt.- Releases are tags on
mainproduced by release automation (release-please or semantic-release) from commit history.
Conventional Commits
Format: <type>[(scope)]: <description> in the imperative, lowercase, no trailing period. Types: feat, fix, docs, style, refactor, perf, test, chore, ci, build, revert. feat!: or a BREAKING CHANGE: footer marks a major bump.
feat(api): add pagination to /documents
fix(ingest): retry S3 reads on throttling
ci: pin all actions to commit SHAs
Enforce with commitlint in CI (@commitlint/config-conventional) and a commit-msg hook.
CI/CD
The pipeline has the same stages everywhere; only the syntax changes.
| Stage | Tooling | Gate |
|---|---|---|
| Lint + format | language linters, hadolint, tofu fmt -check, kubeconform | zero errors |
| Unit + contract tests | language test runner, Pact | pass, coverage trend not falling |
| Build | docker buildx, multi-arch where needed | reproducible, digest recorded |
| SBOM | Syft | SPDX or CycloneDX attached to image |
| Scan | Grype or Trivy, gitleaks, Semgrep | no unwaived High/Critical |
| Sign + attest | cosign keyless, build provenance | signature + provenance verifiable |
| AI evals | eval harness in evals/ | thresholds met |
| Deploy | GitOps commit (image digest bump) | Argo CD / Flux sync healthy |
Minimal GitHub Actions workflow
Gitea Actions runs the same YAML (place it under .gitea/workflows/), with runner labels adjusted.
name: ci
on:
pull_request:
push:
branches: [main]
permissions:
contents: read
jobs:
build-test-scan-sign:
runs-on: ubuntu-24.04
permissions:
contents: read
packages: write
id-token: write # keyless signing (Sigstore OIDC)
attestations: write
env:
IMAGE: ghcr.io/${{ github.repository }}
steps:
# Pin every third-party action to a full commit SHA; Renovate keeps them current.
- uses: actions/checkout@<sha> # v7
- name: Test
run: make lint test
- uses: docker/setup-buildx-action@<sha> # v4
- uses: docker/login-action@<sha> # v4
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- id: build
uses: docker/build-push-action@<sha> # v7
with:
push: ${{ github.ref == 'refs/heads/main' }}
tags: ${{ env.IMAGE }}:${{ github.sha }}
- name: SBOM (Syft)
run: syft "${IMAGE}@${{ steps.build.outputs.digest }}" -o spdx-json=sbom.spdx.json
- name: Scan (Grype)
run: grype sbom:sbom.spdx.json --fail-on high
- uses: sigstore/cosign-installer@<sha> # v4
if: github.ref == 'refs/heads/main'
- name: Sign and attest SBOM
if: github.ref == 'refs/heads/main'
run: |
REF="${IMAGE}@${{ steps.build.outputs.digest }}"
cosign sign --yes "$REF"
cosign attest --yes --type spdxjson --predicate sbom.spdx.json "$REF"
- uses: actions/attest@<sha> # v4 (provenance attestation)
if: github.ref == 'refs/heads/main'
with:
subject-name: ${{ env.IMAGE }}
subject-digest: ${{ steps.build.outputs.digest }}
push-to-registry: true
On 19 March 2026 the Trivy v0.69.4 release, 76 of 77 trivy-action tags, and all setup-trivy tags were briefly replaced with credential-stealing code (CVE-2026-33634); malicious v0.69.5/v0.69.6 images followed on Docker Hub. Pipelines that pinned to mutable tags ran it. Our rule: third-party actions pinned to full commit SHAs, scanner binaries and images pinned by version and checksum or digest, least-privilege permissions: on every job, and no pull_request_target with checkout of untrusted code.
GitLab CI equivalent
stages: [test, build, secure, sign]
test:
stage: test
image: $CI_REGISTRY/base/toolchain:1.4.2
script: [make lint test]
build:
stage: build
image: docker:27
services: [docker:27-dind]
script:
- docker build -t $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA .
- docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
sbom-scan:
stage: secure
image: $CI_REGISTRY/base/sectools:2026.10 # internal image with syft + grype pinned
script:
- syft $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA -o spdx-json=sbom.spdx.json
- grype sbom:sbom.spdx.json --fail-on high
artifacts: { paths: [sbom.spdx.json] }
sign:
stage: sign
image: $CI_REGISTRY/base/sectools:2026.10
id_tokens:
SIGSTORE_ID_TOKEN: { aud: sigstore }
rules: [{ if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH }]
script:
- cosign sign --yes $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
Infrastructure as Code: OpenTofu / Terraform
- Default tool: OpenTofu (MPL-2.0, Linux Foundation; current stable line 1.13, as of 2026-10). Terraform is used where the client has standardised on it; modules stay compatible by avoiding tool-specific features unless needed.
- Layout: reusable modules in
infra/modules/, one root module per environment and per blast radius (network, cluster, data, app). Small states plan faster and fail smaller. - State: remote, versioned, locked, encrypted. Never local state outside a sandbox.
terraform {
backend "s3" {
bucket = "acme-tfstate-prod"
key = "network/terraform.tfstate"
region = "eu01" # S3-compatible: STACKIT, Ceph RGW, AWS
use_lockfile = true # native S3 locking, no DynamoDB table
}
encryption { # OpenTofu-only: client-side state encryption
key_provider "openbao" "state" {
address = "https://bao.internal:8200"
key_name = "tofu-state-prod"
}
method "aes_gcm" "default" {
keys = key_provider.openbao.state
}
state { method = method.aes_gcm.default }
plan { method = method.aes_gcm.default }
}
}
- Enable bucket versioning with a lifecycle rule; locking adds object versions.
tofu planruns on every PR and posts the plan as a comment;tofu applyruns only frommainwith a reviewed plan file.- Drift detection: a nightly
tofu plan -detailed-exitcode; exit code 2 opens an issue. - Back up state-encryption keys before enabling encryption; without the key OpenTofu cannot read the state.
GitOps: Argo CD or Flux
Cluster state is declared in Git and reconciled by a controller. CI never holds cluster credentials.
| Choose | When |
|---|---|
| Argo CD (3.x; 3.3 to 3.5 supported, as of 2026-10) | Teams want a UI, multi-cluster fleets via ApplicationSets, SSO-based RBAC |
| Flux (2.7 to 2.9 supported, as of 2026-10) | Lean, CLI-first, controller-per-concern, strong Helm and OCI-artifact support |
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: api-prod
namespace: argocd
spec:
project: acme
source:
repoURL: https://git.example.com/acme/api.git
targetRevision: main
path: deploy/overlays/prod
destination:
server: https://kubernetes.default.svc
namespace: api
syncPolicy:
automated: { prune: true, selfHeal: true }
syncOptions: [ServerSideApply=true]
Promotion is a pull request that bumps the image digest in the next overlay (Renovate, Argo CD Image Updater, or Flux image automation can open it).
Secrets
Secrets never sit in Git in plaintext, CI variables are short-lived, and workloads get identities instead of static keys.
| Need | Tool |
|---|---|
| Secrets in Git (GitOps) | SOPS with age or KMS keys; Flux decrypts natively, Argo CD via a plugin or KSOPS |
| Sync from a central store into Kubernetes | External Secrets Operator (ExternalSecret → Secret) |
| Central store, dynamic DB creds, PKI, transit encryption | OpenBao (open-source Vault fork under the Linux Foundation / OpenSSF; 2.7 current, as of 2026-10) or HashiCorp Vault where licensed |
| Cloud-native store | AWS Secrets Manager, STACKIT Secrets Manager, OCI Vault, via ESO providers |
| CI to cloud | OIDC federation; no long-lived cloud keys in CI |
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: api-db
spec:
refreshInterval: 1h
secretStoreRef: { kind: ClusterSecretStore, name: openbao }
target: { name: api-db }
data:
- secretKey: DATABASE_URL
remoteRef: { key: apps/api/prod, property: database_url }
OpenBao 2.7 adds PostgreSQL horizontal scalability, HSM/KMS-backed keys, and post-quantum algorithms (ML-DSA, ML-KEM); it moved several vendor auto-unseal mechanisms out of the core binary, so check the release notes before upgrading.
Supply chain security
Target SLSA Build L2 on every project and L3 for regulated clients (SLSA v1.2 is current and adds a Source track, as of 2026-10).
| SLSA Build level | Meaning | How we meet it |
|---|---|---|
| L1 | Provenance exists | Build provenance attestation generated in CI |
| L2 | Signed provenance from a hosted build platform | GitHub/GitLab hosted runners + Sigstore signing |
| L3 | Hardened build platform | Isolated ephemeral runners, reusable workflows, no secrets reachable by build steps |
Controls:
- SBOM: Syft generates SPDX or CycloneDX for every image and attaches it as a cosign attestation.
- Signing: cosign v3 signs keylessly with the Sigstore bundle format by default; verify with
cosign verify --certificate-identity-regexp ... --certificate-oidc-issuer .... Air-gapped clients use a key in OpenBao or KMS instead. - Admission: Kyverno or Sigstore policy-controller rejects unsigned images in production namespaces.
- Vulnerability scanning: Grype or Trivy on the SBOM in CI and nightly against what runs in the cluster. Waivers go in a reviewed
.grype.yaml/.trivyignorewith expiry dates. - Dependencies: Renovate with grouped minor/patch updates, automerge for passing patch updates, digest pinning for images and actions.
{
"$schema": "https://docs.renovatebot.com/renovate-schema.json",
"extends": ["config:recommended", "helpers:pinGitHubActionDigests", ":semanticCommits"],
"packageRules": [
{ "matchUpdateTypes": ["patch", "digest"], "automerge": true },
{ "matchUpdateTypes": ["minor"], "groupName": "minor dependencies" }
],
"vulnerabilityAlerts": { "labels": ["security"] }
}
Observability baseline
Every service emits OpenTelemetry from day one: traces, metrics, and logs correlated by trace ID.
- Instrumentation: OTel auto-instrumentation where available, manual spans around business operations. Resource attributes
service.name,service.version,deployment.environment.nameare mandatory. - Collector: deploy as DaemonSet (node agent) plus a gateway Deployment for tail sampling and export. Keep 100 % of error traces, sample the rest.
- Metrics: RED (rate, errors, duration) per endpoint, USE (utilisation, saturation, errors) per resource.
- Logs: structured JSON to stdout, no PII, trace ID on every line.
- Dashboards as code: Grafana dashboards in Git (JSON or Grafonnet), provisioned, never hand-edited in production.
receivers:
otlp: { protocols: { grpc: {}, http: {} } }
processors:
memory_limiter: { check_interval: 1s, limit_percentage: 80 }
batch: {}
exporters:
otlphttp/tempo: { endpoint: http://tempo:4318 }
otlphttp/loki: { endpoint: http://loki:3100/otlp }
prometheusremotewrite: { endpoint: http://prometheus:9090/api/v1/write }
service:
pipelines:
traces: { receivers: [otlp], processors: [memory_limiter, batch], exporters: [otlphttp/tempo] }
logs: { receivers: [otlp], processors: [memory_limiter, batch], exporters: [otlphttp/loki] }
metrics: { receivers: [otlp], processors: [memory_limiter, batch], exporters: [prometheusremotewrite] }
SLOs and runbooks
- Define 2 to 4 SLOs per user-facing service before go-live, agreed with the client: availability (good requests / valid requests) and latency (share of requests under a threshold).
- Alert on error-budget burn rate, not on raw thresholds: page on fast burn (for example 14.4x over 1 h and 5 min windows), ticket on slow burn (for example 6x over 6 h). Generate rules with Sloth or Pyrra.
- Every alert has a
runbook_urlannotation pointing todocs/runbooks/<alert>.mdwith: symptom, impact, first three checks, mitigation, escalation.
version: prometheus/v1
service: api
slos:
- name: requests-availability
objective: 99.9
sli:
events:
error_query: sum(rate(http_server_request_duration_seconds_count{service="api",http_response_status_code=~"5.."}[{{.window}}]))
total_query: sum(rate(http_server_request_duration_seconds_count{service="api"}[{{.window}}]))
alerting:
name: ApiHighErrorRate
annotations: { runbook_url: https://git.example.com/acme/api/-/blob/main/docs/runbooks/ApiHighErrorRate.md }
DORA metrics
We report the five DORA software delivery metrics monthly to the client from pipeline and incident data:
| Group | Metric | Source |
|---|---|---|
| Throughput | Change lead time (commit to production) | Git + deploy events |
| Throughput | Deployment frequency | GitOps sync events |
| Throughput | Failed deployment recovery time | Incident tracker |
| Instability | Change fail rate | Deploys needing rollback or hotfix |
| Instability | Deployment rework rate (unplanned deploys caused by incidents) | Deploys labelled hotfix |
Track trends per team, never as individual performance metrics.
Security baseline
- Application: OWASP ASVS 5.0 Level 1 for every app, Level 2 for anything handling personal or financial data. Map requirements to tests where possible; document security decisions in ADRs.
- Least privilege: workload identity per service (Kubernetes ServiceAccount bound to cloud IAM), no shared admin accounts, break-glass accounts audited.
- Network: default-deny NetworkPolicies, TLS everywhere, mTLS between services where the client requires it.
- Kubernetes: Pod Security Standards
restrictedon application namespaces, non-root containers, read-only root filesystem. - Static analysis: Semgrep and gitleaks in CI; dependency and image scanning as above.
- Threat model: a STRIDE pass on the C4 container diagram at design time, revisited on major changes.
AI-specific standards
AI features are software and follow everything above, plus:
- Prompts are code. Prompt templates live in
prompts/with a version in the file name or front matter, reviewed through PRs. The running service logs the prompt version and model ID on every call. - Evals in CI. A PR that touches prompts, retrieval, model config, or tools runs the eval suite in
evals/; the job fails when a metric drops below its threshold. - Model pinning. Pin exact model versions or image digests (self-hosted) and treat a model change like a dependency upgrade: PR, evals, staged rollout.
- Tracing. Instrument LLM calls with the OpenTelemetry GenAI semantic conventions (
gen_ai.*attributes; still at Development stability as of 2026-10; the repository move is not independently confirmed). Prompt and response capture is opt-in and follows the data-classification rules.
on:
pull_request:
paths: ["prompts/**", "src/rag/**", "evals/**", "config/models.yaml"]
jobs:
evals:
runs-on: ubuntu-24.04
steps:
- uses: actions/checkout@<sha> # v7
- run: make evals EVAL_SET=evals/golden.jsonl
- run: python evals/check_thresholds.py --min-faithfulness 0.85 --min-answer-relevance 0.80
Documentation
- ADRs in
docs/adr/using the Nygard format (context, decision, consequences). One ADR per significant decision; superseded ADRs stay and link forward. - C4 diagrams: System Context and Container diagrams for every delivery, Component diagrams where they help. Keep them as code (Structurizr DSL or Mermaid) next to the source.
- Runbooks per alert, README for local development, handover doc listing environments, owners, credentials locations, and on-call.
# 4. Use CloudNativePG for PostgreSQL
Status: Accepted (2026-10-02)
## Context
We need HA PostgreSQL on Kubernetes in STACKIT and on-prem with the same tooling.
## Decision
Run PostgreSQL with the CloudNativePG operator, backups to S3-compatible object storage.
## Consequences
One operational model across clouds; the team owns DB upgrades and backup testing.
Definition of done for any delivery
- Repo created from the template;
mainprotected; CODEOWNERS and required checks active - Commits follow Conventional Commits; releases and changelog generated automatically
- CI runs lint, tests, build, SBOM, scan, sign, and provenance; all third-party actions pinned by SHA
- No unwaived High/Critical vulnerabilities; waivers have owners and expiry dates
- Production admission verifies image signatures
- All infrastructure in OpenTofu/Terraform; remote, locked, encrypted state; drift check scheduled
- Deployments via GitOps; no
kubectl applyor console changes in production - Secrets in OpenBao/Vault or a cloud secret manager, synced by ESO or SOPS; no static cloud keys in CI
- OpenTelemetry traces, metrics, and logs flowing; dashboards in Git
- SLOs agreed and burn-rate alerts live; every alert has a runbook
- DORA metrics collection wired up
- ASVS level agreed and verified; threat model recorded
- AI projects: prompts versioned, evals gate PRs, model versions pinned, GenAI spans emitted
- ADRs for key decisions; C4 context and container diagrams current
- Backup and restore tested (not just configured), with the restore time recorded
- Handover document reviewed with the client's operations team
Sources
- Conventional Commits 1.0.0
- OpenTofu releases (v1.13.1 is the current 1.13 patch)
- OpenTofu S3 backend (
use_lockfile) - OpenTofu state and plan encryption
- Argo CD supported versions (endoflife.date)
- Argo CD release process and cadence
- Flux releases and supported versions
- Announcing Flux 2.7 GA
- OpenBao 2.7 release announcement
- OpenBao 2.7.x release notes
- OpenSSF: Announcing OpenBao v2.6
- SLSA v1.2 Build track basics
- Sigstore: Cosign v3 is now available
- Cosign releases
- actions/attest (successor to attest-build-provenance)
- Trivy supply chain compromise advisory GHSA-69fq-xp46-6x23 / CVE-2026-33634
- Aqua: Trivy supply chain attack, what you need to know
- DORA metrics guide
- OpenTelemetry GenAI semantic conventions (spans)
- OWASP ASVS 5.0 release (Global AppSec EU 2025)
- CloudNativePG releases