Ugrás a fő tartalomhoz

How We Build

Basic Advanced

Applies to: every delivery, from a two-week GenAI PoC to a multi-year platform build · Owner: the delivery tech lead

Every service page on this site assumes the standards below. A PoC uses the same repo template, pipeline, and IaC as a production system; it simply turns fewer knobs. That is how a successful PoC becomes production without a rewrite.

The standard in one picture​

Repository template​

Every project starts from the Lieber repo template. Layout:

repo layout
.
├── .github/workflows/ # or .gitea/workflows/ or .gitlab-ci.yml
├── docs/
│ ├── adr/ # 0001-record-architecture-decisions.md, ...
│ ├── architecture/ # C4 diagrams (Structurizr DSL or Mermaid)
│ └── runbooks/ # one file per alert
├── deploy/
│ ├── base/ # Kustomize base or Helm chart
│ └── overlays/{dev,staging,prod}/
├── infra/ # OpenTofu root modules per environment
├── evals/ # AI projects: datasets, graders, thresholds
├── prompts/ # AI projects: versioned prompt templates
├── src/ ...
├── Dockerfile
├── renovate.json
├── CODEOWNERS
└── README.md # run locally in 3 commands

Required repo settings:

SettingValue
Default branchmain, protected
Mergesquash or rebase only, linear history
Required checksCI workflow, at least one review from CODEOWNERS
Signed commitsrequired on main for regulated clients
Secretsnone in the repo; secret scanning (gitleaks) in CI and pre-commit
README"run locally" in three commands or fewer

Trunk-based development​

  • Branches live hours to two days, then merge to main. No long-running develop or release branches.
  • Unfinished work ships behind feature flags (OpenFeature SDK with a flag provider such as flagd).
  • main is always deployable; every merge builds an immutable artifact that is promoted through environments, never rebuilt.
  • Releases are tags on main produced by release automation (release-please or semantic-release) from commit history.

Conventional Commits​

Format: <type>[(scope)]: <description> in the imperative, lowercase, no trailing period. Types: feat, fix, docs, style, refactor, perf, test, chore, ci, build, revert. feat!: or a BREAKING CHANGE: footer marks a major bump.

feat(api): add pagination to /documents
fix(ingest): retry S3 reads on throttling
ci: pin all actions to commit SHAs

Enforce with commitlint in CI (@commitlint/config-conventional) and a commit-msg hook.

CI/CD​

The pipeline has the same stages everywhere; only the syntax changes.

StageToolingGate
Lint + formatlanguage linters, hadolint, tofu fmt -check, kubeconformzero errors
Unit + contract testslanguage test runner, Pactpass, coverage trend not falling
Builddocker buildx, multi-arch where neededreproducible, digest recorded
SBOMSyftSPDX or CycloneDX attached to image
ScanGrype or Trivy, gitleaks, Semgrepno unwaived High/Critical
Sign + attestcosign keyless, build provenancesignature + provenance verifiable
AI evalseval harness in evals/thresholds met
DeployGitOps commit (image digest bump)Argo CD / Flux sync healthy

Minimal GitHub Actions workflow​

Gitea Actions runs the same YAML (place it under .gitea/workflows/), with runner labels adjusted.

.github/workflows/ci.yml
name: ci
on:
pull_request:
push:
branches: [main]

permissions:
contents: read

jobs:
build-test-scan-sign:
runs-on: ubuntu-24.04
permissions:
contents: read
packages: write
id-token: write # keyless signing (Sigstore OIDC)
attestations: write
env:
IMAGE: ghcr.io/${{ github.repository }}
steps:
# Pin every third-party action to a full commit SHA; Renovate keeps them current.
- uses: actions/checkout@<sha> # v7
- name: Test
run: make lint test
- uses: docker/setup-buildx-action@<sha> # v4
- uses: docker/login-action@<sha> # v4
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- id: build
uses: docker/build-push-action@<sha> # v7
with:
push: ${{ github.ref == 'refs/heads/main' }}
tags: ${{ env.IMAGE }}:${{ github.sha }}
- name: SBOM (Syft)
run: syft "${IMAGE}@${{ steps.build.outputs.digest }}" -o spdx-json=sbom.spdx.json
- name: Scan (Grype)
run: grype sbom:sbom.spdx.json --fail-on high
- uses: sigstore/cosign-installer@<sha> # v4
if: github.ref == 'refs/heads/main'
- name: Sign and attest SBOM
if: github.ref == 'refs/heads/main'
run: |
REF="${IMAGE}@${{ steps.build.outputs.digest }}"
cosign sign --yes "$REF"
cosign attest --yes --type spdxjson --predicate sbom.spdx.json "$REF"
- uses: actions/attest@<sha> # v4 (provenance attestation)
if: github.ref == 'refs/heads/main'
with:
subject-name: ${{ env.IMAGE }}
subject-digest: ${{ steps.build.outputs.digest }}
push-to-registry: true
Pin actions and scanners by digest

On 19 March 2026 the Trivy v0.69.4 release, 76 of 77 trivy-action tags, and all setup-trivy tags were briefly replaced with credential-stealing code (CVE-2026-33634); malicious v0.69.5/v0.69.6 images followed on Docker Hub. Pipelines that pinned to mutable tags ran it. Our rule: third-party actions pinned to full commit SHAs, scanner binaries and images pinned by version and checksum or digest, least-privilege permissions: on every job, and no pull_request_target with checkout of untrusted code.

GitLab CI equivalent​

.gitlab-ci.yml
stages: [test, build, secure, sign]

test:
stage: test
image: $CI_REGISTRY/base/toolchain:1.4.2
script: [make lint test]

build:
stage: build
image: docker:27
services: [docker:27-dind]
script:
- docker build -t $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA .
- docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA

sbom-scan:
stage: secure
image: $CI_REGISTRY/base/sectools:2026.10 # internal image with syft + grype pinned
script:
- syft $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA -o spdx-json=sbom.spdx.json
- grype sbom:sbom.spdx.json --fail-on high
artifacts: { paths: [sbom.spdx.json] }

sign:
stage: sign
image: $CI_REGISTRY/base/sectools:2026.10
id_tokens:
SIGSTORE_ID_TOKEN: { aud: sigstore }
rules: [{ if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH }]
script:
- cosign sign --yes $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA

Infrastructure as Code: OpenTofu / Terraform​

  • Default tool: OpenTofu (MPL-2.0, Linux Foundation; current stable line 1.13, as of 2026-10). Terraform is used where the client has standardised on it; modules stay compatible by avoiding tool-specific features unless needed.
  • Layout: reusable modules in infra/modules/, one root module per environment and per blast radius (network, cluster, data, app). Small states plan faster and fail smaller.
  • State: remote, versioned, locked, encrypted. Never local state outside a sandbox.
infra/prod/backend.tf
terraform {
backend "s3" {
bucket = "acme-tfstate-prod"
key = "network/terraform.tfstate"
region = "eu01" # S3-compatible: STACKIT, Ceph RGW, AWS
use_lockfile = true # native S3 locking, no DynamoDB table
}

encryption { # OpenTofu-only: client-side state encryption
key_provider "openbao" "state" {
address = "https://bao.internal:8200"
key_name = "tofu-state-prod"
}
method "aes_gcm" "default" {
keys = key_provider.openbao.state
}
state { method = method.aes_gcm.default }
plan { method = method.aes_gcm.default }
}
}
  • Enable bucket versioning with a lifecycle rule; locking adds object versions.
  • tofu plan runs on every PR and posts the plan as a comment; tofu apply runs only from main with a reviewed plan file.
  • Drift detection: a nightly tofu plan -detailed-exitcode; exit code 2 opens an issue.
  • Back up state-encryption keys before enabling encryption; without the key OpenTofu cannot read the state.

GitOps: Argo CD or Flux​

Cluster state is declared in Git and reconciled by a controller. CI never holds cluster credentials.

ChooseWhen
Argo CD (3.x; 3.3 to 3.5 supported, as of 2026-10)Teams want a UI, multi-cluster fleets via ApplicationSets, SSO-based RBAC
Flux (2.7 to 2.9 supported, as of 2026-10)Lean, CLI-first, controller-per-concern, strong Helm and OCI-artifact support
deploy/argocd/app-prod.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: api-prod
namespace: argocd
spec:
project: acme
source:
repoURL: https://git.example.com/acme/api.git
targetRevision: main
path: deploy/overlays/prod
destination:
server: https://kubernetes.default.svc
namespace: api
syncPolicy:
automated: { prune: true, selfHeal: true }
syncOptions: [ServerSideApply=true]

Promotion is a pull request that bumps the image digest in the next overlay (Renovate, Argo CD Image Updater, or Flux image automation can open it).

Secrets​

Secrets never sit in Git in plaintext, CI variables are short-lived, and workloads get identities instead of static keys.

NeedTool
Secrets in Git (GitOps)SOPS with age or KMS keys; Flux decrypts natively, Argo CD via a plugin or KSOPS
Sync from a central store into KubernetesExternal Secrets Operator (ExternalSecret → Secret)
Central store, dynamic DB creds, PKI, transit encryptionOpenBao (open-source Vault fork under the Linux Foundation / OpenSSF; 2.7 current, as of 2026-10) or HashiCorp Vault where licensed
Cloud-native storeAWS Secrets Manager, STACKIT Secrets Manager, OCI Vault, via ESO providers
CI to cloudOIDC federation; no long-lived cloud keys in CI
deploy/base/externalsecret.yaml
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: api-db
spec:
refreshInterval: 1h
secretStoreRef: { kind: ClusterSecretStore, name: openbao }
target: { name: api-db }
data:
- secretKey: DATABASE_URL
remoteRef: { key: apps/api/prod, property: database_url }

OpenBao 2.7 adds PostgreSQL horizontal scalability, HSM/KMS-backed keys, and post-quantum algorithms (ML-DSA, ML-KEM); it moved several vendor auto-unseal mechanisms out of the core binary, so check the release notes before upgrading.

Supply chain security​

Target SLSA Build L2 on every project and L3 for regulated clients (SLSA v1.2 is current and adds a Source track, as of 2026-10).

SLSA Build levelMeaningHow we meet it
L1Provenance existsBuild provenance attestation generated in CI
L2Signed provenance from a hosted build platformGitHub/GitLab hosted runners + Sigstore signing
L3Hardened build platformIsolated ephemeral runners, reusable workflows, no secrets reachable by build steps

Controls:

  • SBOM: Syft generates SPDX or CycloneDX for every image and attaches it as a cosign attestation.
  • Signing: cosign v3 signs keylessly with the Sigstore bundle format by default; verify with cosign verify --certificate-identity-regexp ... --certificate-oidc-issuer .... Air-gapped clients use a key in OpenBao or KMS instead.
  • Admission: Kyverno or Sigstore policy-controller rejects unsigned images in production namespaces.
  • Vulnerability scanning: Grype or Trivy on the SBOM in CI and nightly against what runs in the cluster. Waivers go in a reviewed .grype.yaml / .trivyignore with expiry dates.
  • Dependencies: Renovate with grouped minor/patch updates, automerge for passing patch updates, digest pinning for images and actions.
renovate.json
{
"$schema": "https://docs.renovatebot.com/renovate-schema.json",
"extends": ["config:recommended", "helpers:pinGitHubActionDigests", ":semanticCommits"],
"packageRules": [
{ "matchUpdateTypes": ["patch", "digest"], "automerge": true },
{ "matchUpdateTypes": ["minor"], "groupName": "minor dependencies" }
],
"vulnerabilityAlerts": { "labels": ["security"] }
}

Observability baseline​

Every service emits OpenTelemetry from day one: traces, metrics, and logs correlated by trace ID.

  • Instrumentation: OTel auto-instrumentation where available, manual spans around business operations. Resource attributes service.name, service.version, deployment.environment.name are mandatory.
  • Collector: deploy as DaemonSet (node agent) plus a gateway Deployment for tail sampling and export. Keep 100 % of error traces, sample the rest.
  • Metrics: RED (rate, errors, duration) per endpoint, USE (utilisation, saturation, errors) per resource.
  • Logs: structured JSON to stdout, no PII, trace ID on every line.
  • Dashboards as code: Grafana dashboards in Git (JSON or Grafonnet), provisioned, never hand-edited in production.
otel-collector.yaml (excerpt)
receivers:
otlp: { protocols: { grpc: {}, http: {} } }
processors:
memory_limiter: { check_interval: 1s, limit_percentage: 80 }
batch: {}
exporters:
otlphttp/tempo: { endpoint: http://tempo:4318 }
otlphttp/loki: { endpoint: http://loki:3100/otlp }
prometheusremotewrite: { endpoint: http://prometheus:9090/api/v1/write }
service:
pipelines:
traces: { receivers: [otlp], processors: [memory_limiter, batch], exporters: [otlphttp/tempo] }
logs: { receivers: [otlp], processors: [memory_limiter, batch], exporters: [otlphttp/loki] }
metrics: { receivers: [otlp], processors: [memory_limiter, batch], exporters: [prometheusremotewrite] }

SLOs and runbooks​

  • Define 2 to 4 SLOs per user-facing service before go-live, agreed with the client: availability (good requests / valid requests) and latency (share of requests under a threshold).
  • Alert on error-budget burn rate, not on raw thresholds: page on fast burn (for example 14.4x over 1 h and 5 min windows), ticket on slow burn (for example 6x over 6 h). Generate rules with Sloth or Pyrra.
  • Every alert has a runbook_url annotation pointing to docs/runbooks/<alert>.md with: symptom, impact, first three checks, mitigation, escalation.
slo/api.yaml (Sloth)
version: prometheus/v1
service: api
slos:
- name: requests-availability
objective: 99.9
sli:
events:
error_query: sum(rate(http_server_request_duration_seconds_count{service="api",http_response_status_code=~"5.."}[{{.window}}]))
total_query: sum(rate(http_server_request_duration_seconds_count{service="api"}[{{.window}}]))
alerting:
name: ApiHighErrorRate
annotations: { runbook_url: https://git.example.com/acme/api/-/blob/main/docs/runbooks/ApiHighErrorRate.md }

DORA metrics​

We report the five DORA software delivery metrics monthly to the client from pipeline and incident data:

GroupMetricSource
ThroughputChange lead time (commit to production)Git + deploy events
ThroughputDeployment frequencyGitOps sync events
ThroughputFailed deployment recovery timeIncident tracker
InstabilityChange fail rateDeploys needing rollback or hotfix
InstabilityDeployment rework rate (unplanned deploys caused by incidents)Deploys labelled hotfix

Track trends per team, never as individual performance metrics.

Security baseline​

  • Application: OWASP ASVS 5.0 Level 1 for every app, Level 2 for anything handling personal or financial data. Map requirements to tests where possible; document security decisions in ADRs.
  • Least privilege: workload identity per service (Kubernetes ServiceAccount bound to cloud IAM), no shared admin accounts, break-glass accounts audited.
  • Network: default-deny NetworkPolicies, TLS everywhere, mTLS between services where the client requires it.
  • Kubernetes: Pod Security Standards restricted on application namespaces, non-root containers, read-only root filesystem.
  • Static analysis: Semgrep and gitleaks in CI; dependency and image scanning as above.
  • Threat model: a STRIDE pass on the C4 container diagram at design time, revisited on major changes.

AI-specific standards​

AI features are software and follow everything above, plus:

  • Prompts are code. Prompt templates live in prompts/ with a version in the file name or front matter, reviewed through PRs. The running service logs the prompt version and model ID on every call.
  • Evals in CI. A PR that touches prompts, retrieval, model config, or tools runs the eval suite in evals/; the job fails when a metric drops below its threshold.
  • Model pinning. Pin exact model versions or image digests (self-hosted) and treat a model change like a dependency upgrade: PR, evals, staged rollout.
  • Tracing. Instrument LLM calls with the OpenTelemetry GenAI semantic conventions (gen_ai.* attributes; still at Development stability as of 2026-10; the repository move is not independently confirmed). Prompt and response capture is opt-in and follows the data-classification rules.
.github/workflows/evals.yml (excerpt)
on:
pull_request:
paths: ["prompts/**", "src/rag/**", "evals/**", "config/models.yaml"]
jobs:
evals:
runs-on: ubuntu-24.04
steps:
- uses: actions/checkout@<sha> # v7
- run: make evals EVAL_SET=evals/golden.jsonl
- run: python evals/check_thresholds.py --min-faithfulness 0.85 --min-answer-relevance 0.80

Documentation​

  • ADRs in docs/adr/ using the Nygard format (context, decision, consequences). One ADR per significant decision; superseded ADRs stay and link forward.
  • C4 diagrams: System Context and Container diagrams for every delivery, Component diagrams where they help. Keep them as code (Structurizr DSL or Mermaid) next to the source.
  • Runbooks per alert, README for local development, handover doc listing environments, owners, credentials locations, and on-call.
docs/adr/0004-use-cloudnativepg.md
# 4. Use CloudNativePG for PostgreSQL

Status: Accepted (2026-10-02)

## Context
We need HA PostgreSQL on Kubernetes in STACKIT and on-prem with the same tooling.

## Decision
Run PostgreSQL with the CloudNativePG operator, backups to S3-compatible object storage.

## Consequences
One operational model across clouds; the team owns DB upgrades and backup testing.

Definition of done for any delivery​

  • Repo created from the template; main protected; CODEOWNERS and required checks active
  • Commits follow Conventional Commits; releases and changelog generated automatically
  • CI runs lint, tests, build, SBOM, scan, sign, and provenance; all third-party actions pinned by SHA
  • No unwaived High/Critical vulnerabilities; waivers have owners and expiry dates
  • Production admission verifies image signatures
  • All infrastructure in OpenTofu/Terraform; remote, locked, encrypted state; drift check scheduled
  • Deployments via GitOps; no kubectl apply or console changes in production
  • Secrets in OpenBao/Vault or a cloud secret manager, synced by ESO or SOPS; no static cloud keys in CI
  • OpenTelemetry traces, metrics, and logs flowing; dashboards in Git
  • SLOs agreed and burn-rate alerts live; every alert has a runbook
  • DORA metrics collection wired up
  • ASVS level agreed and verified; threat model recorded
  • AI projects: prompts versioned, evals gate PRs, model versions pinned, GenAI spans emitted
  • ADRs for key decisions; C4 context and container diagrams current
  • Backup and restore tested (not just configured), with the restore time recorded
  • Handover document reviewed with the client's operations team

Sources​