Cloud-native Product and Modern Software Development
Service: Cloud-native Product Development · Modern Software Development · Typical shape: dedicated squad (architect, senior engineers, project manager) in agile sprints; or a modernization track that adopts CI/CD, containers, and Kubernetes in an existing team
What we deliver
- New products built cloud-native from the first sprint: architecture, code, CI/CD, IaC, observability, and runbooks, all following How We Build.
- Modernization of existing teams and systems: containerization, a Kubernetes production baseline, GitOps, and a paved road (internal developer platform) so teams deploy many times per day with automated recovery.
- Managed operations of the resulting platform where the client wants it.
Reference architecture
Architecture choices
Start with the simplest architecture that meets the requirements and keep the seams clean so it can split later.
| Option | Choose when | Watch for |
|---|---|---|
| Modular monolith (default) | One team or a few teams, domain still being discovered, need for fast iteration | Enforce module boundaries (package visibility, ArchUnit/import-linter rules); one database schema per module |
| Microservices | Independent teams that must deploy independently, parts with very different scaling or availability needs, clear bounded contexts | Distributed transactions (use outbox + events), contract drift (Pact), operational cost per service |
| Serverless functions (Lambda, Cloudflare Workers) | Spiky, event-driven, or edge-latency workloads; glue code; webhooks | Cold starts, execution limits, vendor-specific APIs; keep domain logic in portable libraries |
| Edge + containers (Cloudflare Workers + Containers) | Global low-latency front with heavier Linux workloads behind it | Cloudflare Containers became GA in April 2026 (as of 2026-10); design for scale-to-zero start-up time |
Decision rule we apply: a module becomes a separate service only when at least one of these holds and is recorded in an ADR: a different team owns it, it needs an independent release cadence, it has a different scaling profile, or it has a different security/compliance boundary.
12-factor, refreshed
The Twelve-Factor App was open-sourced by Heroku in November 2024 and is being refreshed by the community (logs as telemetry, workload identity). We apply it with the common extensions often called "15-factor" (API first, telemetry, authentication and authorization):
| Factor | Our implementation |
|---|---|
| Codebase, build/release/run | One repo per deployable, immutable image per commit, promoted by digest |
| Dependencies | Lockfiles committed, Renovate-managed |
| Config | Environment variables and mounted files from ConfigMaps / ExternalSecrets; no environment-specific builds |
| Backing services | Addressed by URL from config; swappable between managed and in-cluster |
| Processes, concurrency | Stateless pods, horizontal scaling via HPA/KEDA |
| Disposability | Fast start, graceful SIGTERM handling, readiness gates |
| Dev/prod parity | Same image and Helm/Kustomize base everywhere; Testcontainers locally |
| Logs → telemetry | OpenTelemetry traces, metrics, logs |
| Admin processes | Kubernetes Jobs from the same image (migrations, backfills) |
| API first | OpenAPI contract written and reviewed before implementation |
| Security | OIDC at the edge, workload identity inside |
Implementation steps
1. Containers
Rules: multi-stage builds, minimal base (distroless or Chainguard), non-root, no shell in production images, pinned base by digest, one process per container.
# syntax=docker/dockerfile:1
FROM golang:1.27 AS build
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN \
CGO_ENABLED=0 go build -trimpath -ldflags="-s -w" -o /out/api ./cmd/api
FROM gcr.io/distroless/static-debian12:nonroot
COPY /out/api /api
USER 65532:65532
EXPOSE 8080
ENTRYPOINT ["/api"]
# build and runtime share the same Chainguard Python so the venv's interpreter path matches
FROM cgr.dev/chainguard/python:latest-dev AS build
USER root
COPY /uv /usr/local/bin/uv
ENV UV_PYTHON_DOWNLOADS=never UV_LINK_MODE=copy
WORKDIR /app
COPY pyproject.toml uv.lock ./
RUN uv sync --frozen --no-dev --no-install-project
COPY src ./src
RUN uv sync --frozen --no-dev
FROM cgr.dev/chainguard/python:latest
WORKDIR /app
COPY /app /app
ENV PATH="/app/.venv/bin:$PATH"
ENTRYPOINT ["python", "-m", "app"]
Pin :latest tags to digests in the real file (Renovate maintains them). Debug distroless pods with kubectl debug --profile=restricted and an ephemeral container whose security context matches the target.
2. Kubernetes production baseline
Target a supported Kubernetes minor (1.35 to 1.37 maintained upstream; 1.34 reaches end of life on 2026-10-27, as of 2026-10). Managed offerings (STACKIT SKE, EKS, OKE) or Cluster API on OpenStack (see Hybrid Cloud and OpenStack).
Every workload ships with:
apiVersion: apps/v1
kind: Deployment
metadata:
name: api
labels: { app.kubernetes.io/name: api }
spec:
replicas: 3
selector: { matchLabels: { app.kubernetes.io/name: api } }
template:
metadata:
labels: { app.kubernetes.io/name: api }
spec:
serviceAccountName: api
automountServiceAccountToken: false
securityContext:
runAsNonRoot: true
seccompProfile: { type: RuntimeDefault }
topologySpreadConstraints:
- maxSkew: 1
topologyKey: topology.kubernetes.io/zone
whenUnsatisfiable: ScheduleAnyway
labelSelector: { matchLabels: { app.kubernetes.io/name: api } }
containers:
- name: api
image: registry.example.com/acme/api@sha256:...
ports: [{ name: http, containerPort: 8080 }]
resources:
requests: { cpu: 250m, memory: 256Mi }
limits: { memory: 256Mi } # memory limit = request; no CPU limit by default
readinessProbe: { httpGet: { path: /readyz, port: http } }
livenessProbe: { httpGet: { path: /livez, port: http }, periodSeconds: 20 }
startupProbe: { httpGet: { path: /livez, port: http }, failureThreshold: 30, periodSeconds: 2 }
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities: { drop: [ALL] }
envFrom: [{ secretRef: { name: api-env } }]
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata: { name: api }
spec:
minAvailable: 2
selector: { matchLabels: { app.kubernetes.io/name: api } }
---
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata: { name: api }
spec:
scaleTargetRef: { apiVersion: apps/v1, kind: Deployment, name: api }
minReplicas: 3
maxReplicas: 20
metrics:
- type: Resource
resource: { name: cpu, target: { type: Utilization, averageUtilization: 70 } }
Namespace-level guardrails:
apiVersion: v1
kind: Namespace
metadata:
name: api
labels:
pod-security.kubernetes.io/enforce: restricted
pod-security.kubernetes.io/audit: restricted
pod-security.kubernetes.io/warn: restricted
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: { name: default-deny, namespace: api }
spec:
podSelector: {}
policyTypes: [Ingress, Egress]
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: { name: api-allow, namespace: api }
spec:
podSelector: { matchLabels: { app.kubernetes.io/name: api } }
ingress:
- from: [{ namespaceSelector: { matchLabels: { kubernetes.io/metadata.name: gateway } } }]
ports: [{ port: 8080 }]
egress:
- to: [{ namespaceSelector: { matchLabels: { kubernetes.io/metadata.name: kube-system } } }]
ports: [{ port: 53, protocol: UDP }]
- to: [{ podSelector: { matchLabels: { cnpg.io/cluster: api-db } } }]
ports: [{ port: 5432 }]
Baseline checklist per cluster:
| Area | Standard |
|---|---|
| Scheduling | Requests on every container, memory limits, topology spread across zones, PDBs |
| Scaling | HPA for CPU/memory, KEDA for queue/event-driven scaling (including scale to zero), cluster autoscaler or Karpenter |
| Security | Pod Security Standards restricted (enforced via namespace labels), default-deny NetworkPolicies, Kyverno policies for signed images and required labels |
| Ingress | Gateway API (see below) with cert-manager for TLS |
| Secrets | External Secrets Operator from OpenBao or the cloud secret manager |
| Observability | OTel Collector, kube-state-metrics, node exporter, Grafana dashboards in Git |
| Delivery | Argo CD or Flux, Argo Rollouts for critical services |
| Backup | Velero (cluster resources and volumes) plus database-native backups |
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata: { name: worker }
spec:
scaleTargetRef: { name: worker }
minReplicaCount: 0
maxReplicaCount: 50
triggers:
- type: rabbitmq
metadata: { queueName: jobs, mode: QueueLength, value: "20" }
authenticationRef: { name: rabbitmq-auth }
3. Gateway API instead of Ingress
The community ingress-nginx controller was retired in March 2026 and its repository archived (2026-03-24): no further releases or security fixes. The Ingress API itself is feature-frozen but not removed. New clusters use Gateway API; existing ingress-nginx installs get a migration plan (ingress2gateway, 1.0 since March 2026 and 1.2 as of 2026-10, converts manifests; annotation-specific behaviour needs manual mapping).
Gateway API v1.5 (released 2026-02-27) moved existing experimental features to the Standard channel. The current release is v1.6.3 (2026-10-06), with v1.7.0-rc.1 in pre-release (as of 2026-10). Implementations we use: Envoy Gateway, Cilium, Istio, kgateway, NGINX Gateway Fabric, Traefik; check each implementation's conformance report for the features you need.
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: public
namespace: gateway
annotations: { cert-manager.io/cluster-issuer: letsencrypt }
spec:
gatewayClassName: envoy
listeners:
- name: https
protocol: HTTPS
port: 443
hostname: "*.example.com"
tls: { certificateRefs: [{ name: wildcard-example-com }] }
allowedRoutes: { namespaces: { from: Selector, selector: { matchLabels: { gateway-access: "true" } } } }
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata: { name: api, namespace: api }
spec:
parentRefs: [{ name: public, namespace: gateway }]
hostnames: ["api.example.com"]
rules:
- matches: [{ path: { type: PathPrefix, value: /v1 } }]
backendRefs: [{ name: api-stable, port: 8080 }]
4. Platform engineering: the paved road
An internal developer platform turns this baseline into self-service. We build it in layers and stop at the layer the client needs.
- Backstage Software Catalog: every service has a
catalog-info.yaml(owner, system, API, links to dashboards and runbooks). - Software Templates (
scaffolder.backstage.io/v1beta3) create a repo from our template with CI, Dockerfile, manifests, and an Argo CD app already wired. - TechDocs renders
docs/(ADRs, runbooks) next to the service. - Measure platform success with DORA metrics and time-to-first-deploy for a new service (target: same day).
apiVersion: scaffolder.backstage.io/v1beta3
kind: Template
metadata: { name: go-service, title: Go service (paved road) }
spec:
owner: platform-team
type: service
parameters:
- title: Service
required: [name, owner]
properties:
name: { type: string }
owner:
type: string
ui:field: OwnerPicker
steps:
- id: fetch
action: fetch:template
input: { url: ./skeleton, values: { name: "${{ parameters.name }}", owner: "${{ parameters.owner }}" } }
- id: publish
action: publish:github
input: { repoUrl: "github.com?owner=acme&repo=${{ parameters.name }}", defaultBranch: main }
- id: register
action: catalog:register
input: { repoContentsUrl: "${{ steps.publish.output.repoContentsUrl }}", catalogInfoPath: /catalog-info.yaml }
5. Progressive delivery
Critical services deploy as canaries with automated analysis against SLO metrics; everything else uses rolling updates with readiness gates.
| Tool | Fit |
|---|---|
| Argo Rollouts | Pairs with Argo CD; Gateway API traffic routing via the argoproj-labs/gatewayAPI plugin; AnalysisTemplates on Prometheus |
| Flagger | Pairs with Flux; supports Gateway API and service meshes |
| Feature flags (OpenFeature) | Decouple release from deploy for business features |
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata: { name: api }
spec:
replicas: 3
selector: { matchLabels: { app.kubernetes.io/name: api } }
workloadRef: { apiVersion: apps/v1, kind: Deployment, name: api, scaleDown: progressively }
strategy:
canary:
stableService: api-stable
canaryService: api-canary
trafficRouting:
plugins:
argoproj-labs/gatewayAPI: { httpRoute: api, namespace: api }
analysis:
templates: [{ templateName: success-rate }]
startingStep: 1
steps:
- setWeight: 10
- pause: { duration: 5m }
- setWeight: 50
- pause: { duration: 10m }
---
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata: { name: success-rate }
spec:
metrics:
- name: success-rate
interval: 1m
failureLimit: 2
successCondition: result[0] >= 0.995
provider:
prometheus:
address: http://prometheus.monitoring:9090
query: |
sum(rate(http_server_request_duration_seconds_count{service="api-canary",http_response_status_code!~"5.."}[2m]))
/ sum(rate(http_server_request_duration_seconds_count{service="api-canary"}[2m]))
6. Testing strategy
| Layer | Tooling | Runs |
|---|---|---|
| Unit | Language runner (Go testing, pytest, JUnit 5, Vitest) | Every commit |
| Integration | Testcontainers for real Postgres, Kafka, Redis, S3 (MinIO/LocalStack) | Every PR |
| Contract | Pact consumer-driven contracts with a Pact Broker and can-i-deploy; OpenAPI schema validation (Schemathesis) for public APIs | Every PR on both sides |
| E2E | Playwright on 5 to 10 critical journeys against an ephemeral or staging environment | Merge to main |
| Performance | k6 against staging with SLO-derived thresholds | Nightly or before releases |
| Production | Canary analysis, synthetic probes (Grafana Synthetic Monitoring or blackbox exporter) | Continuous |
func TestStore(t *testing.T) {
ctx := context.Background()
pg, err := postgres.Run(ctx, "postgres:17-alpine",
postgres.WithDatabase("app"),
postgres.BasicWaitStrategies(),
)
require.NoError(t, err)
t.Cleanup(func() { _ = pg.Terminate(ctx) })
dsn, err := pg.ConnectionString(ctx, "sslmode=disable")
require.NoError(t, err)
s := store.MustOpen(dsn)
require.NoError(t, s.Migrate(ctx))
// ... exercise the store against a real Postgres
}
pact-broker can-i-deploy --pacticipant api --version "$GIT_SHA" --to-environment production
7. API design
- Contract first: OpenAPI 3.1 in
api/openapi.yaml, reviewed in PRs, linted with Spectral, used to generate server stubs and clients. AsyncAPI for event contracts. - Versioning: major version in the path (
/v1), additive changes only within a major; breaking changes get a new major and a deprecation window announced withDeprecationandSunsetheaders. - Conventions: RFC 9457 problem details for errors, cursor-based pagination, idempotency keys on unsafe retries (
Idempotency-Keyheader), consistent resource naming. - Breaking-change check in CI:
oasdiff breaking main.yaml pr.yaml --fail-on ERR. - Internal service calls: gRPC with protobuf and
buf breakingwhere latency and typing matter; REST/JSON for public APIs.
openapi: 3.1.0
info: { title: Acme API, version: 1.4.0 }
paths:
/v1/documents:
get:
operationId: listDocuments
parameters:
- { name: cursor, in: query, schema: { type: string } }
- { name: limit, in: query, schema: { type: integer, maximum: 100, default: 20 } }
responses:
"200":
description: A page of documents
content:
application/json:
schema: { $ref: "#/components/schemas/DocumentPage" }
default:
description: Error
content:
application/problem+json:
schema: { $ref: "#/components/schemas/Problem" }
8. Data: PostgreSQL on Kubernetes with CloudNativePG
Use the cloud's managed Postgres when available and acceptable; use CloudNativePG for on-prem, OpenStack, and sovereign setups that need one operational model everywhere. CloudNativePG 1.30 is current (1.30.1 is the latest patch; supports Kubernetes 1.34 to 1.36 and Postgres 14 to 18; adds a declarative DatabaseRole CRD). In-tree Barman Cloud backup is deprecated and slated for removal in 1.31.0; use the Barman Cloud Plugin (as of 2026-10).
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata: { name: api-db }
spec:
instances: 3
imageName: ghcr.io/cloudnative-pg/postgresql:17
primaryUpdateStrategy: unsupervised
storage: { size: 50Gi, storageClass: fast-rbd }
postgresql:
parameters: { max_connections: "200", shared_buffers: "1GB" }
bootstrap:
initdb: { database: app, owner: app }
affinity: { topologyKey: topology.kubernetes.io/zone }
monitoring: { enablePodMonitor: true }
plugins:
- name: barman-cloud.cloudnative-pg.io
isWALArchiver: true
parameters: { barmanObjectName: api-db-backup }
---
apiVersion: barmancloud.cnpg.io/v1
kind: ObjectStore
metadata: { name: api-db-backup }
spec:
configuration:
destinationPath: s3://acme-pg-backups/api-db/
endpointURL: https://object.storage.eu01.onstackit.cloud
s3Credentials:
accessKeyId: { name: pg-backup-s3, key: ACCESS_KEY_ID }
secretAccessKey: { name: pg-backup-s3, key: ACCESS_SECRET_KEY }
wal: { compression: gzip }
retentionPolicy: "30d"
---
apiVersion: postgresql.cnpg.io/v1
kind: ScheduledBackup
metadata: { name: api-db-daily }
spec:
schedule: "0 0 2 * * *"
cluster: { name: api-db }
method: plugin
pluginConfiguration: { name: barman-cloud.cloudnative-pg.io }
- Schema migrations run as a Kubernetes Job (or Argo CD PreSync hook) using Flyway, Atlas, or the framework's migrator; migrations are backward compatible for one release (expand/contract).
- Connection pooling via the CNPG
Pooler(PgBouncer). - Test point-in-time recovery into a scratch cluster every quarter and record the restore time.
- Track CNPG security releases; 1.30.0 and 1.29.2 (June 2026) fixed CVE-2026-55769 and CVE-2026-55765.
Best practices
- Default to a modular monolith; split along bounded contexts only with an ADR.
- One immutable image per commit, promoted by digest across environments.
- Memory limits equal to requests; avoid CPU limits unless the client needs hard isolation (they cause throttling without improving fairness over requests).
- Graceful shutdown: handle
SIGTERM, stop accepting traffic, drain in-flight requests withinterminationGracePeriodSeconds. - Expand/contract for every schema change so rollbacks remain possible.
- Keep serverless functions thin; domain logic lives in a library that also runs in containers.
- Run the paved road as a product: a platform backlog, docs in Backstage, and feedback from the teams using it.
Pitfalls to design around
- Ingress controller choice: start new clusters on Gateway API and plan existing ingress-nginx migrations explicitly.
- Chatty microservices: keep synchronous call chains to two hops or fewer; use events and the outbox pattern for cross-service workflows.
- Probes: keep liveness probes independent of downstream dependencies so a database blip does not restart every pod; readiness can check dependencies.
- Scale-to-zero: pair KEDA scale-to-zero with a start-up time budget the users accept, or keep
minReplicaCount: 1for interactive paths. - Shared databases between services: give each service its own schema and credentials from the start.
- Pod Security
restricted: audit images for root users before enforcing; runwarnandauditfirst.
Definition of done
- Architecture choice and service boundaries recorded in ADRs; C4 context and container diagrams current
- OpenAPI/AsyncAPI contracts in the repo, linted, with breaking-change checks in CI
- Images multi-stage, distroless or Chainguard based, non-root, pinned by digest, signed, with SBOM
- Every workload has requests, memory limits, probes, PDB, topology spread, and an HPA or KEDA ScaledObject
- Namespaces enforce Pod Security
restricted; default-deny NetworkPolicies with explicit allows - Traffic enters through Gateway API with automated TLS; no dependency on ingress-nginx
- Deployments through GitOps; critical services use canary analysis with automatic rollback
- Test pyramid in CI: unit, Testcontainers integration, Pact contracts with
can-i-deploy, E2E smoke - Postgres via managed service or CloudNativePG with scheduled backups, WAL archiving, and a tested PITR
- OpenTelemetry instrumentation, SLOs, burn-rate alerts, and runbooks in place
- Service registered in the Backstage catalog with owner, docs, and links (when a portal is in scope)
- DORA metrics reported; handover to the client team completed
Sources
- Twelve-Factor App methodology is now open source
- twelve-factor/twelve-factor on GitHub
- Kubernetes releases and support windows
- Kubernetes v1.36 release blog
- Ingress NGINX retirement: what you need to know
- Ingress NGINX: statement from the Steering and Security Response Committees
- Gateway API v1.5: moving features to Stable
- Gateway API v1.5.0 release notes
- Gateway API v1.6.0 release notes
- ingress2gateway releases
- Cloudflare: Containers and Sandboxes are now generally available
- Chainguard Containers overview
- Chainguard Academy: debugging distroless images
- KEDA external scalers
- Backstage Software Templates
- Backstage: adding templates
- Argo Rollouts analysis and progressive delivery
- Argo Rollouts Gateway API plugin quick start
- testcontainers-go PostgreSQL module
- CloudNativePG releases
- CloudNativePG 1.30.0 released
- CloudNativePG 1.29.1 and 1.28.3: critical CVE fix
- CloudNativePG supported releases
- Barman Cloud Plugin usage
- testcontainers-go Postgres module
- Chainguard Python image