Hybrid Cloud and OpenStack Private Cloud
Service: Hybrid Cloud Architecture · OpenStack Private Cloud · Typical shape: design sprint (2 to 4 weeks), pilot region (6 to 10 weeks), then migration waves and managed day-2 operations
What we deliver
- OpenStack private cloud on commodity hardware and Ubuntu: compute, software-defined networking, block/object storage on Ceph, load balancing, and Kubernetes-as-a-service, deployed and upgraded by automation.
- VMware exit path: discovery, wave planning, and automated VM conversion onto OpenStack.
- Hybrid architecture: sensitive data and steady workloads on-premise, burst and managed services in STACKIT, AWS, or OCI, connected by private, encrypted networking.
- Unified management: one IaC codebase, one Kubernetes lifecycle model (Cluster API), one observability plane across all environments.
- GPU capacity for private AI via PCI passthrough or vGPU flavors.
Reference architecture
Core OpenStack services
| Service | Role | Our default |
|---|---|---|
| Keystone | Identity, projects, app credentials | Federated to client IdP via OIDC/SAML |
| Nova | Compute (KVM/libvirt) | Host aggregates per hardware class |
| Neutron + OVN | SDN, routers, security groups, floating IPs | ML2/OVN, distributed routing |
| Cinder | Block volumes | Ceph RBD backend, volume types per pool |
| Glance | Images | Ceph RBD backend (copy-on-write clones) |
| Swift API / Ceph RGW | Object storage, S3 API | Ceph RGW with Keystone auth |
| Octavia | Load balancers (L4/L7) | Amphora or OVN provider |
| Barbican | Secrets, certificates | Needed by Octavia TLS and Magnum CAPI |
| Magnum + Cluster API | Kubernetes as a service | magnum-capi-helm driver, or CAPO directly |
| Designate | DNS as a service | Optional, for tenant self-service |
| Horizon / Skyline | Web UI | Horizon; API and IaC preferred |
Release landscape (as of 2026-10)
| Series | Released | SLURP | Status |
|---|---|---|---|
| 2026.2 Hibiscus | 2026-09-30 | No | Maintained |
| 2026.1 Gazpacho | 2026-04-01 | Yes | Maintained |
| 2025.2 Flamingo | 2025-10-01 | No | Maintained |
| 2025.1 Epoxy | 2025-04-02 | Yes | Maintained |
| 2027.1 Indri | 2027-03-24 (est.) | Yes | In development |
SLURP (Skip Level Upgrade Release Process) releases can be upgraded directly to the next SLURP, skipping the intermediate .2 release. For most clients we deploy SLURP releases only and upgrade yearly: 2025.1 → 2026.1 → 2027.1.
Choosing a deployment method
| Option | What it is | Choose when | Notes |
|---|---|---|---|
| Canonical OpenStack (Sunbeam) | Snap-based, OpenStack control plane on Kubernetes managed by Juju | Ubuntu shop, wants vendor support (Ubuntu Pro), small to mid-size clouds | LTS releases 2024.1 and 2026.1; 2024.1 runs to Apr 2029 (standard), Apr 2034 (Ubuntu Pro), Apr 2036 (Legacy Support). 2024.1/stable is the snap stable channel (as of 2026-10); the 2026.1 snap channel is not listed on the snap page, so check channels before committing |
| Kolla-Ansible | Upstream, OpenStack services in containers, deployed with Ansible | Teams that want upstream, full control, any size | kolla-ansible 22.x is the current stable series (22.2.0, 2026-09-08); check its release notes for the Ansible versions it needs |
| OpenStack-Helm | Upstream Helm charts for OpenStack on Kubernetes | Client already runs Kubernetes well and wants one operational model | Most Kubernetes-native; most moving parts |
| Red Hat OpenStack Services on OpenShift (RHOSO) 18 | OpenStack control plane as operators on OpenShift | Red Hat subscription estate | Requires an OpenShift cluster and an external Ceph cluster; check the RHOSO 18 planning guide for supported versions |
Our default for "enterprise OpenStack on Ubuntu" is Canonical OpenStack when the client wants a commercial support contract, and Kolla-Ansible on Ubuntu 24.04 when the client wants upstream releases on day one or a feature not yet in the Canonical stable channel. Check the Canonical snap channel for the target release before committing to a date.
Implementation steps
1. Discovery and sizing
Collect: VM inventory (vCPU, RAM, disk, IOPS, growth), network segments, storage tiers, GPU needs, RPO/RTO, compliance zones. Export from vCenter with RVTools or govc.
Sizing rules of thumb we start from, then validate against the inventory:
| Item | Starting rule |
|---|---|
| Control plane | 3 nodes (odd number for Galera/RabbitMQ/OVN DB quorum), 16+ cores, 128 GB RAM, NVMe |
| CPU overcommit | 4:1 general purpose, 1:1 for latency-sensitive and GPU flavors (hw:cpu_policy=dedicated) |
| RAM overcommit | 1:1 (no memory overcommit in production) |
| Ceph raw capacity | usable needed × 3 (replica 3) ÷ 0.75 (stay below ~75 % full for recovery headroom) |
| Ceph OSD nodes | at least 4 for replica 3 with one-node failure headroom; NVMe for RBD pools |
| Network | 2 × 25 GbE bonded per node minimum; separate VLANs for management, storage public, storage cluster, tenant overlay, provider |
| Failure domains | at least 3 racks or power zones mapped to Ceph CRUSH and Nova availability zones |
2. Bare metal and OS
- Provision with MAAS (Canonical) or Ironic/Bifrost; Ubuntu 24.04 LTS on all nodes.
- Netplan bonds with LACP, jumbo frames on storage and overlay networks, NTP (chrony) on every node.
- Firmware, BIOS (VT-d/IOMMU on for GPU nodes), and RAID settings captured as code.
3a. Deploy with Canonical OpenStack (Sunbeam)
# On the first node (Ubuntu 24.04). Pick the channel that matches the target release.
sudo snap install openstack --channel 2024.1/stable
sunbeam prepare-node-script --bootstrap | bash -x && newgrp snap_daemon
sunbeam cluster bootstrap --role control,compute,storage # interactive or with --manifest
sunbeam cluster add <node-fqdn> # prints a join token
# on the new node:
sunbeam cluster join --role compute,storage <token>
sunbeam configure --openrc demo-openrc
# cluster add/join and enable subcommands not confirmed against the snap docs; check `sunbeam <cmd> --help`
sunbeam enable orchestration
sunbeam enable loadbalancer
sunbeam enable vault && sunbeam enable secrets
sunbeam enable dns "ns1.example.com."
sunbeam enable observability embedded
Keep the deployment manifest (manifest.yaml) in Git; it carries network ranges, Ceph disks, and feature settings.
3b. Deploy with Kolla-Ansible
python3 -m venv ~/kolla && . ~/kolla/bin/activate
pip install 'kolla-ansible==22.*' # 2026.1 Gazpacho series
kolla-ansible install-deps
cp -r ~/kolla/share/kolla-ansible/etc_examples/kolla /etc/kolla
cp ~/kolla/share/kolla-ansible/ansible/inventory/multinode .
kolla-genpwd # then move passwords.yml into SOPS/OpenBao
kolla-ansible bootstrap-servers -i multinode
kolla-ansible prechecks -i multinode
kolla-ansible deploy -i multinode
kolla-ansible post-deploy -i multinode
kolla_base_distro: "ubuntu"
openstack_release: "2026.1"
kolla_internal_vip_address: "10.10.0.250"
kolla_external_vip_address: "203.0.113.10"
network_interface: "bond0.100"
neutron_external_interface: "bond1"
neutron_plugin_agent: "ovn"
enable_cinder: "yes"
enable_octavia: "yes"
enable_barbican: "yes"
enable_magnum: "yes"
enable_designate: "yes"
glance_backend_ceph: "yes"
cinder_backend_ceph: "yes"
nova_backend_ceph: "yes"
kolla_enable_tls_internal: "yes"
kolla_enable_tls_external: "yes"
4. Ceph storage
Run Ceph Tentacle (20.2.x, latest 20.2.4 as of 2026-10) deployed by cephadm, MicroCeph (Sunbeam), or Rook (OpenStack-Helm).
cephadm bootstrap --mon-ip 10.20.0.11 --cluster-network 10.30.0.0/24
ceph orch host add stor-02 10.20.0.12
ceph orch apply osd --all-available-devices
ceph osd pool create volumes 256 && ceph osd pool application enable volumes rbd
ceph osd pool create images 64 && ceph osd pool application enable images rbd
ceph osd pool create vms 256 && ceph osd pool application enable vms rbd
ceph osd pool set volumes pg_autoscale_mode on
ceph auth get-or-create client.cinder mon 'profile rbd' osd 'profile rbd pool=volumes, profile rbd pool=vms, profile rbd-read-only pool=images'
- Map CRUSH failure domain to
rack(orhoston small clusters). - Use erasure coding for RGW/object and backup pools; keep RBD for VMs on replica 3.
- Apply Ceph point releases promptly; 20.2.4 (2026-08-19) is the latest Tentacle point release.
5. Tenancy, quotas, and IaC
Everything after the platform install is OpenTofu with the openstack provider: domains, projects, quotas, flavors, networks, images.
resource "openstack_identity_project_v3" "team_a" {
name = "team-a-prod"
domain_id = var.domain_id
}
resource "openstack_compute_quotaset_v2" "team_a" {
project_id = openstack_identity_project_v3.team_a.id
cores = 256
ram = 1048576
instances = 100
}
resource "openstack_compute_flavor_v2" "gp_8x32" {
name = "gp.8x32"
vcpus = 8
ram = 32768
disk = 0 # boot from Cinder volume
is_public = true
}
resource "openstack_networking_network_v2" "app" {
name = "app-net"
tenant_id = openstack_identity_project_v3.team_a.id
}
6. Kubernetes on OpenStack
Two supported paths, both based on Cluster API:
- Magnum with
magnum-capi-helm: tenants request clusters through the Magnum API; the driver renders Helm charts that create Cluster API resources on a management cluster. Needs Octavia (API load balancers) and Barbican (certificates). - Cluster API Provider OpenStack (CAPO) directly: platform team manages clusters as YAML in Git. Same model works for AWS (CAPA) and other providers, which is what makes hybrid management uniform. CAPO v0.15 (released 2026-09-17) is the current line: it adds the v1beta2 API types alongside v1beta1 and implements the Cluster API v1beta2 contract; v0.14 still gets patch releases (as of 2026-10). Core Cluster API v1beta1 is on track to be unserved in CAPI v1.16, so write new manifests against v1beta2.
apiVersion: cluster.x-k8s.io/v1beta2
kind: Cluster
metadata:
name: onprem-prod
spec:
clusterNetwork:
pods: { cidrBlocks: ["10.244.0.0/16"] }
infrastructureRef:
apiGroup: infrastructure.cluster.x-k8s.io
kind: OpenStackCluster
name: onprem-prod
controlPlaneRef:
apiGroup: controlplane.cluster.x-k8s.io
kind: KubeadmControlPlane
name: onprem-prod-cp
Inside each cluster: OpenStack Cloud Controller Manager, Cinder CSI, and Octavia ingress or a Gateway API implementation (see Cloud-native Development).
7. GPU flavors for private AI
| Mode | Use for | Nova mechanism |
|---|---|---|
| PCI passthrough | Full GPU per VM: LLM inference, fine-tuning | [pci] device_spec + alias, flavor pci_passthrough:alias |
| vGPU (mdev) | Shared GPU slices: notebooks, small models, VDI | [devices] enabled_mdev_types, flavor resources:VGPU=1 |
| SR-IOV vGPU (Ampere and newer NVIDIA) | Shared slices on modern GPUs | Not covered here; follow the current Nova vGPU docs (unverified) |
[pci]
device_spec = { "vendor_id": "10de", "product_id": "2330" }
alias = { "vendor_id": "10de", "product_id": "2330", "device_type": "type-PCI", "name": "h100", "numa_policy": "preferred" }
[filter_scheduler]
enabled_filters = ComputeFilter,ImagePropertiesFilter,ServerGroupAntiAffinityFilter,ServerGroupAffinityFilter,PciPassthroughFilter,NUMATopologyFilter
openstack flavor create gpu.h100x1 --vcpus 24 --ram 196608 --disk 0 \
--property "pci_passthrough:alias"="h100:1" \
--property hw:cpu_policy=dedicated \
--property hw:mem_page_size=large
Enable IOMMU on the host kernel command line (intel_iommu=on iommu=pt or amd_iommu=on), bind the GPU to vfio-pci, and put GPU hosts in their own aggregate. The serving stack on top is covered in Private and Sovereign AI and Serving at Scale.
8. VMware exit migration
Tooling (open source first):
| Tool | Notes |
|---|---|
os-migrate vmware-migration-kit | Ansible collection (os_migrate.vmware_migration_kit), maintained by the Red Hat OpenStack migration team; default path is nbdkit on a conversion host with optional VMware CBT for near-zero-downtime two-phase cutover, alternatively virt-v2v with or without a conversion host; result lands as Cinder volumes or Glance images |
| virt-v2v + nbdkit | Converts guests (drivers, bootloader, VirtIO) and streams VMDK data without full download first |
| Coriolis, VEXXHOST MigrateKit, vJailbreak | Listed in the OpenInfra VMware migration guide; evaluate per estate size and support needs |
# Variable names from the collection's vars.yaml example; check them against the pinned collection version.
vcenter_hostname: vcenter.corp.local
vcenter_datacenter: DC1
vddk_libdir: /usr/lib/vmware-vix-disklib
already_deploy_conversion_host: true
os_migrate_vmw_data_dir: /opt/os-migrate
use_existing_flavor: true
vms_list: [app-db-01, app-web-01, app-web-02]
ansible-galaxy collection install os_migrate.vmware_migration_kit
ansible-playbook -i inventory.yml os_migrate.vmware_migration_kit.migration \
-e @secrets.yml -e @migrate/wave-01.vars.yml # vCenter + dst_cloud credentials in secrets.yml (SOPS)
Per-wave checklist: VirtIO drivers present (Windows: install virtio-win before conversion), static IPs mapped to Neutron ports, MAC addresses preserved where licences depend on them, application owner sign-off after smoke tests, rollback (power the source VM back on) rehearsed. Use incremental/warm migration for large disks to keep cutover windows short.
9. Hybrid connectivity
| Need | Pattern |
|---|---|
| Site-to-site private network to a hyperscaler | IPsec VPN (strongSwan or a VPN appliance on OpenStack) to the cloud VPN gateway, BGP for route exchange, two tunnels for redundancy |
| Low-overhead links between our own sites and clusters | WireGuard (kernel module), routed with BGP (FRR or BIRD) |
| Dedicated bandwidth | Provider interconnect (AWS Direct Connect, STACKIT and OCI interconnect offerings) with BGP |
| Publishing internal apps without inbound firewall holes | Cloudflare Tunnel (cloudflared) plus Cloudflare Access for identity-aware access |
| Kubernetes service-to-service across clouds | Cilium Cluster Mesh or a service mesh with multi-cluster support |
[Interface]
Address = 169.254.10.1/30
ListenPort = 51820
PrivateKey = <from OpenBao>
[Peer]
PublicKey = <peer public key>
Endpoint = 198.51.100.20:51820
AllowedIPs = 169.254.10.2/32, 10.60.0.0/16
PersistentKeepalive = 25
router bgp 65010
neighbor 169.254.10.2 remote-as 65020
neighbor 169.254.10.2 bfd
address-family ipv4 unicast
network 10.10.0.0/16
neighbor 169.254.10.2 prefix-list FROM-CLOUD in
exit-address-family
tunnel: 6f1c... # tunnel UUID
credentials-file: /etc/cloudflared/6f1c.json
ingress:
- hostname: grafana.example.com
service: http://grafana.monitoring.svc:3000
- service: http_status:404
Plan IP space for all environments up front (non-overlapping RFC 1918 ranges per site and cloud); renumbering later is the most expensive hybrid change.
10. Unified management
- One IaC repo, one root module per environment, providers for
openstack,stackit,aws, andcloudflare. Shared modules expose the same inputs (network, cluster, DNS) regardless of the target. - One cluster lifecycle: a Cluster API management cluster with CAPO for on-prem and the matching provider (or the managed service's API via OpenTofu) for public clouds; Argo CD ApplicationSets deploy the same add-ons (CNI, CSI, cert-manager, ESO, OTel Collector) to every cluster.
- One observability plane: each site runs Prometheus/OTel Collector agents that remote-write to a central Mimir (or Thanos) and ship logs to Loki and traces to Tempo; Grafana is the single pane. OpenStack itself is monitored with
openstack-exporter,node_exporter, Ceph's Prometheus module, libvirt and OVN exporters. - Placement policy in an ADR: which data classes may run where (for example personal data on-prem or STACKIT only; stateless burst in public cloud).
11. Day-2 operations
| Task | Practice |
|---|---|
| Upgrades | SLURP to SLURP yearly; rehearse on a staging region built from the same code; upgrade control plane, then computes in batches with live migration |
| Patching | Monthly OS patching with rolling host evacuation (openstack compute service set --disable, live-migrate, reboot) |
| Capacity | Dashboards for allocation vs. physical per aggregate; alert at 70 % CPU/RAM allocation and 70 % Ceph raw usage |
| Backups | Database dumps (Galera/MariaDB) and config nightly; tenant backups via Cinder backup to RGW or an external backup tool |
| Health | Rally/Tempest smoke tests nightly; alert on failed API probes, RabbitMQ queue growth, OVN DB leader changes, Ceph HEALTH_WARN |
| Certificates | cert-manager or the deployer's TLS automation; alert 21 days before expiry |
# Ceph nearing full
ceph_cluster_total_used_bytes / ceph_cluster_total_bytes > 0.70
# Nova compute service down
openstack_nova_agent_state{adminState="enabled"} == 0
Best practices
- Deploy only SLURP releases unless a client needs a feature from a
.2release. - Keep the platform reproducible: if a staging region cannot be rebuilt from Git in a day, fix that before production.
- Separate control plane and data plane networks physically or by VLAN with QoS; storage traffic should never compete with tenant traffic.
- Federate Keystone to the client IdP and use application credentials for automation instead of user passwords.
- Treat GPU hosts as a separate failure and maintenance domain (no live migration with passthrough devices; drain by rescheduling).
- Document data placement and connectivity in ADRs and C4 diagrams per How We Build.
Pitfalls to design around
- Quorum: run three control nodes and three Ceph monitors so a single failure keeps the cloud writeable.
- MTU: set the physical MTU (9000) high enough that the Geneve overlay inside still offers 1500 to tenants, and test it end to end.
- Ceph headroom: plan to stay below roughly 75 % raw usage so a node failure can re-replicate without hitting the full ratio.
- Release availability: confirm the target release exists in the chosen distribution's stable channel before scheduling the install.
- Windows guests: install VirtIO drivers and check licence activation behaviour before the migration wave, not during it.
- Overlapping IP ranges: agree an address plan across all clouds during design.
Definition of done
- Hardware, network, and IP plan documented; failure domains mapped to Ceph CRUSH and Nova AZs
- OpenStack deployed from Git (Sunbeam manifest or Kolla
globals.yml+ inventory); secrets in OpenBao or SOPS - Release is a SLURP release (or an ADR records why not); upgrade path documented
- Keystone federated to the client IdP; RBAC roles and quotas in OpenTofu
- Ceph healthy, pools sized, alerts at 70 % raw; restore from backup tested
- Octavia, Barbican, and Kubernetes-as-a-service working; a test cluster created and deleted via the API
- GPU flavors (if in scope) schedule correctly and pass a CUDA smoke test in a guest
- VMware migration: pilot wave converted, validated, signed off; runbook for later waves
- Hybrid links redundant (two tunnels or paths), BGP with BFD, failover tested
- Central observability shows every site and cloud; SLOs for API availability and VM provisioning time
- Rally/Tempest smoke tests scheduled; on-call runbooks written for the top alerts
- Day-2 handbook (patching, upgrades, capacity, backup) handed over and walked through with the operations team
Sources
- OpenStack releases and SLURP status
- OpenStack 2026.1 Gazpacho release page
- Canonical OpenStack: release cycle and supported versions
- Canonical OpenStack snap channels
- Canonical OpenStack: configuring GPU passthrough
- Kolla-Ansible 2026.1 release notes
- Red Hat OpenStack Services on OpenShift 18.0 documentation
- RHOSO 18.0 Planning your deployment
- Ceph releases index
- Ceph Tentacle release notes
- Canonical OpenStack: install with the manual bare metal provider
- snap-openstack (sunbeam) source
- Cluster API v1.14.0 release notes (v1beta1 deprecation)
- os-migrate vmware-migration-kit example vars.yaml
- Nova: attaching physical PCI devices to guests
- Nova: attaching virtual GPU devices to guests
- magnum-capi-helm documentation
- StackHPC: Magnum Cluster API Helm driver deep dive
- Cluster API Provider OpenStack releases
- os-migrate vmware-migration-kit
- OpenInfra: VMware to OpenStack migration guide
- OpenTofu state encryption