Cloud AI Platforms and AI Tool Integration
Service: Cloud AI Platform Services · AI Tool Integration · Typical shape: 1–2 week platform selection and landing zone, then iterative delivery per workload; tool rollouts as a 4–8 week pilot-to-org programme
Managed AI platforms let a customer use LLMs, RAG and agents without running a GPU fleet. Our job is to pick the platform that matches data residency, model and cost constraints, implement it with IaC and CI/CD, and keep the application portable so the next move (another region, another cloud, or self-hosted per Private and Sovereign AI) is a configuration change.
What we deliver
- Platform decision record: decision matrix filled for the customer, with residency and contract notes.
- Landing zone for AI: accounts/projects, IAM, private networking, logging, budgets — in Terraform.
- Model access layer: an OpenAI-compatible gateway (LiteLLM or the platform's own gateway) with per-team keys, quotas and spend reporting.
- Workload implementation: RAG (managed or custom), guardrails, agents, evaluation pipeline.
- AI tool rollout: coding assistants and workspace copilots with SSO, data controls, MCP allowlists, telemetry and usage policy.
Reference architecture
The gateway is the portability seam: applications speak the OpenAI Chat Completions (or Responses) protocol to one URL; routing, credentials, fallbacks and budgets live in gateway config.
Platform implementation notes
AWS: Amazon Bedrock and SageMaker AI
| Capability | Service (as of 2026-10) | Notes |
|---|---|---|
| Model invocation | Bedrock Converse / ConverseStream | One request shape across model providers; tool use; guardrailConfig |
| OpenAI-compatible | https://bedrock-runtime.{region}.amazonaws.com/openai/v1 (recommended); https://bedrock-mantle.{region}.api.aws/v1 as compatibility endpoint for models or features not on bedrock-runtime | SigV4 or Bedrock API key (bedrock-mantle: API key or AWS credentials) |
| Residency | In-Region inference, or EU geographic cross-Region inference profiles (eu. prefix) that keep routing inside EU Regions | Global profiles route worldwide — exclude them via SCP when residency matters |
| RAG | Bedrock Knowledge Bases with S3 Vectors, OpenSearch Serverless, Aurora PostgreSQL and partner vector stores | RetrieveAndGenerate or Retrieve + own prompt |
| Safety | Bedrock Guardrails (content filters, denied topics, PII, contextual grounding) | Attach per call or per agent |
| Agents | Amazon Bedrock AgentCore: Runtime, Gateway, Identity, Memory, Policy (Cedar, GA 2026-03), Evaluations, Observability | Framework-agnostic (Strands, LangGraph, CrewAI) |
| Custom models / training | Amazon SageMaker AI (training jobs, HyperPod, real-time and async endpoints) | Use for fine-tunes and open models needing dedicated endpoints |
import os
import boto3
brt = boto3.client("bedrock-runtime", region_name="eu-central-1")
resp = brt.converse(
modelId=os.environ["BEDROCK_MODEL_ID"], # EU inference profile ID from `aws bedrock list-inference-profiles`
system=[{"text": "Answer from the provided context only."}],
messages=[{"role": "user", "content": [{"text": "Summarise our travel policy."}]}],
inferenceConfig={"maxTokens": 512, "temperature": 0.2},
guardrailConfig={
"guardrailIdentifier": os.environ["GUARDRAIL_ID"],
"guardrailVersion": "1",
"trace": "enabled",
},
)
print(resp["output"]["message"]["content"][0]["text"])
print(resp["usage"]) # inputTokens / outputTokens for chargeback
import boto3
agent_rt = boto3.client("bedrock-agent-runtime", region_name="eu-central-1")
resp = agent_rt.retrieve_and_generate(
input={"text": "What is the notice period for contractors?"},
retrieveAndGenerateConfiguration={
"type": "KNOWLEDGE_BASE",
"knowledgeBaseConfiguration": {
"knowledgeBaseId": "KBID12345",
"modelArn": "arn:aws:bedrock:eu-central-1:111122223333:inference-profile/<eu-profile-id>",
"retrievalConfiguration": {"vectorSearchConfiguration": {"numberOfResults": 8}},
},
},
)
print(resp["output"]["text"])
for c in resp["citations"]:
print([r["location"] for r in c["retrievedReferences"]])
Implementation checklist for AWS:
- Organisation SCP: deny
bedrock:*outside approved Regions; allow-list model IDs/inference profiles per OU. - VPC interface endpoints (PrivateLink) for
bedrock-runtimeandbedrock-agent-runtime; no NAT path needed for inference. - Model invocation logging to S3/CloudWatch with KMS — enable deliberately and set retention, since it captures prompts.
- Guardrail versions managed in Terraform/CDK; applications reference a version, never
DRAFT. - Cost: tag inference profiles (application inference profiles) per team; Cost Explorer by tag.
STACKIT: AI Model Serving and AI Model Experiments
- AI Model Serving: shared open-weight models (gpt-oss, Qwen3.x, Gemma 4, Llama 3.3, embeddings) via an OpenAI-compatible API in EU01, billed per token; model list, limits and prices are in Private and Sovereign AI.
- AI Model Experiments: managed MLflow tracking server (release stage not confirmed; check STACKIT docs) with artifacts in STACKIT Object Storage; supports MLflow LLM tracing and evaluation.
- Compute Engine GPU + SKE: dedicated A100/L40S/H100 when shared models do not fit.
- Terraform:
stackit_modelserving_token,stackit_modelexperiments_instance,stackit_modelexperiments_tokeninstackitcloud/stackit.
resource "time_rotating" "rotate" { rotation_days = 80 }
resource "stackit_modelserving_token" "app" {
project_id = var.project_id
name = "rag-app"
rotate_when_changed = { rotation = time_rotating.rotate.id }
}
resource "stackit_modelexperiments_instance" "mlflow" {
project_id = var.project_id
region = "eu01"
name = "rag-experiments"
deleted_experiment_retention = "30d"
}
resource "stackit_modelexperiments_token" "ci" {
project_id = var.project_id
region = "eu01"
name = "ci-evals"
instance_id = stackit_modelexperiments_instance.mlflow.instance_id
ttl_duration = "80d"
}
import os
from openai import OpenAI
client = OpenAI(
base_url="https://api.openai-compat.model-serving.eu01.onstackit.cloud/v1",
api_key=os.environ["STACKIT_MODEL_SERVING_TOKEN"],
)
emb = client.embeddings.create(model="Qwen/Qwen3-VL-Embedding-8B", input=["Vertragslaufzeit", "notice period"])
chat = client.chat.completions.create(
model="google/gemma-4-31B-it",
messages=[{"role": "user", "content": "Draft a polite payment reminder in German."}],
)
print(len(emb.data[0].embedding), chat.choices[0].message.content)
OCI Generative AI
- Endpoint: OpenAI-compatible base URL
https://inference.generativeai.<region>.oci.oraclecloud.com/openai/v1with/responses,/chat/completions,/conversations,/files; Oracle recommends the Responses API as the primary entry point for most new agentic workloads. - Auth: Generative AI API keys for development; IAM-based auth (
oci_openaiauth classes with anhttpxclient) for production. The project OCID header (project=in the sample) is unconfirmed against Oracle's auth docs; check before use. - Hosting modes: on-demand, or dedicated AI clusters (pretrained models carry a minimum commitment; imported models bill with a one-hour minimum).
- EU residency: per-region model availability changes; check Oracle's model-regions page. Unconfirmed: gpt-oss on demand in Germany Central, private endpoints in EU Sovereign Central since 2026-02, and Gemini in Frankfurt being hosted by Google.
- Data Science service for notebooks, jobs and model deployment of custom models.
import os
from openai import OpenAI
client = OpenAI(
base_url="https://inference.generativeai.eu-frankfurt-1.oci.oraclecloud.com/openai/v1",
api_key=os.environ["OCI_GENAI_API_KEY"],
project=os.environ["OCI_GENAI_PROJECT_OCID"],
)
r = client.responses.create(model="openai.gpt-oss-120b", input="List three NIS2 obligations for IT providers.")
print(r.output_text)
Cloudflare: Workers AI, AI Gateway, Vectorize, AI Search
| Product (as of 2026-10) | Use |
|---|---|
| Workers AI | Serverless inference on Cloudflare's GPUs via the AI binding or REST |
| AI Gateway | Proxy for Workers AI and external providers: caching, rate limiting, logging, retries/fallback, DLP (block/flag), Guardrails (Llama Guard), unified billing. Shares the AI binding with Workers AI (unified since 2026-08 per Cloudflare blog; default gateway auto-creation unconfirmed) |
| Vectorize | Vector DB: up to 1,536 dimensions, 20 M vectors per index, topK 50 with metadata / 100 without |
| AI Search (formerly AutoRAG) | Managed index + retrieval pipeline over R2/websites, hybrid search, MCP endpoint; GA 2026-10-01, usage billing from 2026-11-01. Bindings ai_search (single instance) and ai_search_namespaces |
{
"name": "rag-edge",
"main": "src/index.ts",
"compatibility_date": "2026-10-01",
"ai": { "binding": "AI" },
"vectorize": [{ "binding": "VEC", "index_name": "docs-bge-m3" }]
}
export interface Env { AI: Ai; VEC: VectorizeIndex }
export default {
async fetch(req: Request, env: Env): Promise<Response> {
const { question } = await req.json<{ question: string }>();
const emb = await env.AI.run("@cf/baai/bge-m3", { text: [question] });
const hits = await env.VEC.query(emb.data[0], { topK: 8, returnMetadata: "all" });
const context = hits.matches.map((m) => m.metadata?.text).join("\n---\n");
const answer = await env.AI.run(
"@cf/meta/llama-3.1-8b-instruct-fp8",
{ messages: [
{ role: "system", content: `Answer only from this context:\n${context}` },
{ role: "user", content: question },
] },
{ gateway: { id: "default" } }, // logs, caching, DLP, rate limits via AI Gateway
);
return Response.json(answer);
},
};
Cloudflare is the right fit for edge-latency features, public-facing RAG over web content, and a SaaS-side gateway in front of several providers. For regulated personal data, check processing locations per model and use the EU data localisation options of the customer's Cloudflare plan.
Canonical: Charmed Kubeflow and Charmed MLflow
- Charmed Kubeflow
1.11/stableand1.10/stablechannels on Charmhub (as of 2026-10); deploys on Canonical Kubernetes or any conformant cluster with Juju. - Charmed MLflow
2.22/stable(MinIO artifact store, MySQL metadata) — integrates with Kubeflow; for MLflow 3.x features (LLM tracing, prompt registry) use upstream MLflow 3.x or STACKIT AI Model Experiments. - Fits customers standardised on Ubuntu Pro with long-term support needs, on-premise or on OpenStack.
juju add-model kubeflow
juju deploy kubeflow --trust --channel=1.11/stable
juju deploy mlflow --trust --channel=2.22/stable
juju status --watch 5s
Decision matrix
| Criterion | AWS Bedrock | STACKIT | OCI GenAI | Cloudflare | Canonical (self-run) |
|---|---|---|---|---|---|
| Data residency | EU Regions + EU inference profiles; US-headquartered provider | Germany, German operator (Schwarz Group) | EU and EU Sovereign regions; US-headquartered provider | Global edge; regional options per plan | Wherever the customer runs it |
| Model choice | Widest: Anthropic, OpenAI, Meta, Mistral, Amazon Nova, Qwen, Gemma, etc. | Curated open-weight set | Cohere, Meta, OpenAI gpt-oss, xAI, Google | Open models on Workers AI; any provider via AI Gateway | Any open model |
| Cost model | Per token; provisioned throughput | Per token; GPU VMs per hour | Per token; dedicated clusters | Per neuron/token; gateway free tier | Hardware + ops |
| Ops burden | Low | Low (shared) / medium (GPU) | Low–medium | Very low | High |
| Managed RAG | Knowledge Bases | Build on SKE/Postgres | Vector stores, agents | AI Search, Vectorize | Build |
| Agents | AgentCore | Build | OCI agents | Agents SDK | Build |
| Best for | Broad model needs, existing AWS estate | Sovereignty-first EU workloads | Oracle-centric estates, DB-adjacent AI | Edge, web-facing, multi-provider gateway | Full control, air-gap, Ubuntu shops |
Recommendation heuristics: sovereignty-sensitive data → STACKIT first (self-hosted on SKE if shared models are not enough); frontier proprietary models required → Bedrock in EU Regions with EU profiles; already on Oracle → OCI; public web workloads → Cloudflare. Always keep the gateway seam.
Portability patterns
- One protocol. Use the OpenAI SDK everywhere; Bedrock (
bedrock-runtime/openai/v1), STACKIT, OCI, Cloudflare and vLLM all expose compatible endpoints. - Aliases, not model names, in application config.
- Gateway config owns credentials, fallbacks, budgets and logging policy.
- Contract tests per alias: tool calling, JSON-schema output, streaming, max context — run in CI against every backend before switching.
- Keep embeddings sticky. Changing the embedding model means re-indexing; pin embedding backends per index.
model_list:
- model_name: chat-default
litellm_params:
model: openai/openai/gpt-oss-120b
api_base: https://api.openai-compat.model-serving.eu01.onstackit.cloud/v1
api_key: os.environ/STACKIT_MODEL_SERVING_TOKEN
- model_name: chat-default-fallback
litellm_params:
model: bedrock/converse/eu.<inference-profile-id> # from `aws bedrock list-inference-profiles`
aws_region_name: eu-central-1
- model_name: chat-batch
litellm_params:
model: hosted_vllm/gpt-oss-120b
api_base: http://vllm.llm.svc:8000/v1
router_settings:
fallbacks: [{ "chat-default": ["chat-default-fallback"] }]
num_retries: 2
litellm_settings:
turn_off_message_logging: true
Install LiteLLM from a hash-pinned lockfile or a digest-pinned image: releases 1.82.7 and 1.82.8 on PyPI were compromised in March 2026 (incident).
AI tool integration
Ready-made tools (coding assistants such as Claude Code and GitHub Copilot, workspace copilots such as Microsoft 365 Copilot or Gemini for Workspace, chat front-ends) deliver value fastest when rolled out as a managed platform, not as individual sign-ups.
Rollout technique
| Control | Implementation |
|---|---|
| Identity | SSO (SAML/OIDC) with SCIM provisioning; licences bound to IdP groups; enforce org login (e.g. Claude Code forceLoginOrgUUID, Copilot via enterprise-managed users) |
| Data controls | Enterprise plans with no training on customer data; retention settings; content exclusion for secrets and regulated repos; deny reads of .env and secret paths |
| Model routing | Point tools at the corporate gateway or a cloud provider (e.g. Claude Code on Bedrock) when residency requires it |
| MCP servers | Central allowlist of approved MCP servers by URL or command; internal registry for company servers; deny everything else |
| Telemetry | OpenTelemetry export to the customer's collector: sessions, tokens, cost, accepted edits, per team |
| Policy | Acceptable-use policy, code review stays mandatory, AI-generated code labelled in PR templates, security scanning in CI unchanged |
Claude Code — managed settings are deployed by MDM or as files in /Library/Application Support/ClaudeCode/ (macOS), /etc/claude-code/ (Linux/WSL) or C:\Program Files\ClaudeCode\ (Windows), or as server-managed settings from the claude.ai console; they override all user and project settings.
{
"permissions": {
"deny": ["Read(./.env)", "Read(./secrets/**)"],
"disableBypassPermissionsMode": "disable"
},
"allowManagedPermissionRulesOnly": true,
"allowManagedMcpServersOnly": true,
"allowedMcpServers": [
{ "serverUrl": "https://mcp.internal.example.com/*" }
],
"env": {
"CLAUDE_CODE_ENABLE_TELEMETRY": "1",
"OTEL_METRICS_EXPORTER": "otlp",
"OTEL_LOGS_EXPORTER": "otlp",
"OTEL_EXPORTER_OTLP_PROTOCOL": "grpc",
"OTEL_EXPORTER_OTLP_ENDPOINT": "https://otel.internal.example.com:4317"
}
}
Managed settings apply above user and project settings, and env is merged per variable with managed values winning, so the collector endpoint cannot be redirected from a developer's settings file. Verify with /status or claude doctor, and check that claude_code.session.count arrives at the collector.
GitHub Copilot — since 2026-08 enterprise managed settings support allowedMcpServers / deniedMcpServers, matching servers by remote URL, local command or name (per the GitHub changelog). Third-party write-ups report that malformed policy fails closed; verify in GitHub's docs before relying on it. The file path copilot/managed-settings.json comes from the changelog summaries; confirm it in the enterprise managed-settings docs. Treat the allowlist as governance: GitHub's docs note configuration files can be edited locally. JetBrains support followed on 2026-08-18.
Workspace copilots — the work is mostly data hygiene: fix oversharing in SharePoint/Drive permissions before enabling the copilot, apply sensitivity labels, and run the vendor's readiness reports. Measure with the admin usage dashboards plus a short user survey after 30 days.
Measuring value
- Adoption: weekly active users / licences, sessions per user.
- Throughput proxies: PR cycle time, review turnaround, lines accepted (as a signal, not a target).
- Cost: tokens and spend per team from telemetry or gateway.
- Quality: change failure rate and escaped defects unchanged or better.
Best practices
- Decide residency per data class first; it eliminates most platform options quickly.
- Gateway from day one, even with a single provider.
- Use EU-bound inference profiles on Bedrock and block global profiles where residency matters.
- Everything — guardrails, knowledge bases, tokens, gateway config, managed settings — in Git and deployed by pipeline.
- Rotate platform tokens automatically (
time_rotating+ Terraform, Secrets Manager rotation). - Roll out AI tools with telemetry on from the first pilot user, so value can be shown with data.
Pitfalls to design around
- Prompt logging captures personal data: enable it deliberately, encrypt, and set retention aligned with the DPIA.
- Provider-specific features leak into code (Bedrock KB citations, AI Search APIs): wrap them behind an internal retrieval interface.
- Shared-model catalogues change: deprecations on STACKIT, OCI retirements, Bedrock model lifecycle — monitor and keep aliases.
- MCP servers are code execution: allowlist by URL/command and review them like dependencies.
- Supply chain of the gateway is part of the attack surface: pin and scan.
Definition of done
- Decision record with the filled matrix and residency analysis signed off
- Landing zone (accounts/projects, IAM, private endpoints, budgets, logging) in Terraform; pipeline applies it
- Gateway deployed with aliases, fallbacks, per-team keys and budgets; contract tests green on every backend
- Guardrails / DLP configured and versioned; tested with a red-team prompt set
- RAG or agent workload evaluated against a golden set; results tracked (MLflow or platform evals)
- Dashboards: tokens, spend, latency, error rate per team and model
- Tokens and keys rotate automatically; no long-lived secrets in application config
- AI tools: SSO + SCIM, managed settings deployed, MCP allowlist enforced, telemetry flowing, AUP published
- 30-day adoption and value report delivered for the tool pilot
Sources
- AWS: Converse with guardrails, Chat Completions on Bedrock (bedrock-runtime / bedrock-mantle), EU cross-Region inference guide, Knowledge Bases vector stores, S3 Vectors with Knowledge Bases, AgentCore Policy GA, AgentCore Policy and Evaluations, AgentCore Memory FGAC
- STACKIT: AI Model Serving shared models, Use the models, AI Model Experiments, AI Model Experiments product page, Terraform provider
- OCI: OpenAI-compatible endpoints, Responses API, IAM auth, Model regions, Dedicated cluster shapes, Private endpoints in EU Sovereign Central
- Cloudflare: Workers AI bindings, AI Gateway, AI Gateway features, Unifying Workers AI and AI Gateway, Vectorize limits, AI Search, AI Search release notes, AutoRAG renamed to AI Search
- Canonical: Charmhub kubeflow, Charmhub mlflow, Announcing Charmed Kubeflow 1.10
- LiteLLM: docs, PyPI compromise timeline
- Claude Code: Settings, Managed settings, Monitoring usage (OpenTelemetry)
- GitHub Copilot: MCP allowlists in enterprise managed settings, Configure enterprise MCP allowlist, Enterprise managed settings, JetBrains support