Ugrás a fő tartalomhoz

Cloud AI Platforms and AI Tool Integration

Basic Advanced

Service: Cloud AI Platform Services · AI Tool Integration · Typical shape: 1–2 week platform selection and landing zone, then iterative delivery per workload; tool rollouts as a 4–8 week pilot-to-org programme

Managed AI platforms let a customer use LLMs, RAG and agents without running a GPU fleet. Our job is to pick the platform that matches data residency, model and cost constraints, implement it with IaC and CI/CD, and keep the application portable so the next move (another region, another cloud, or self-hosted per Private and Sovereign AI) is a configuration change.

What we deliver​

  • Platform decision record: decision matrix filled for the customer, with residency and contract notes.
  • Landing zone for AI: accounts/projects, IAM, private networking, logging, budgets — in Terraform.
  • Model access layer: an OpenAI-compatible gateway (LiteLLM or the platform's own gateway) with per-team keys, quotas and spend reporting.
  • Workload implementation: RAG (managed or custom), guardrails, agents, evaluation pipeline.
  • AI tool rollout: coding assistants and workspace copilots with SSO, data controls, MCP allowlists, telemetry and usage policy.

Reference architecture​

The gateway is the portability seam: applications speak the OpenAI Chat Completions (or Responses) protocol to one URL; routing, credentials, fallbacks and budgets live in gateway config.

Platform implementation notes​

AWS: Amazon Bedrock and SageMaker AI​

CapabilityService (as of 2026-10)Notes
Model invocationBedrock Converse / ConverseStreamOne request shape across model providers; tool use; guardrailConfig
OpenAI-compatiblehttps://bedrock-runtime.{region}.amazonaws.com/openai/v1 (recommended); https://bedrock-mantle.{region}.api.aws/v1 as compatibility endpoint for models or features not on bedrock-runtimeSigV4 or Bedrock API key (bedrock-mantle: API key or AWS credentials)
ResidencyIn-Region inference, or EU geographic cross-Region inference profiles (eu. prefix) that keep routing inside EU RegionsGlobal profiles route worldwide — exclude them via SCP when residency matters
RAGBedrock Knowledge Bases with S3 Vectors, OpenSearch Serverless, Aurora PostgreSQL and partner vector storesRetrieveAndGenerate or Retrieve + own prompt
SafetyBedrock Guardrails (content filters, denied topics, PII, contextual grounding)Attach per call or per agent
AgentsAmazon Bedrock AgentCore: Runtime, Gateway, Identity, Memory, Policy (Cedar, GA 2026-03), Evaluations, ObservabilityFramework-agnostic (Strands, LangGraph, CrewAI)
Custom models / trainingAmazon SageMaker AI (training jobs, HyperPod, real-time and async endpoints)Use for fine-tunes and open models needing dedicated endpoints
bedrock_converse.py
import os
import boto3

brt = boto3.client("bedrock-runtime", region_name="eu-central-1")

resp = brt.converse(
modelId=os.environ["BEDROCK_MODEL_ID"], # EU inference profile ID from `aws bedrock list-inference-profiles`
system=[{"text": "Answer from the provided context only."}],
messages=[{"role": "user", "content": [{"text": "Summarise our travel policy."}]}],
inferenceConfig={"maxTokens": 512, "temperature": 0.2},
guardrailConfig={
"guardrailIdentifier": os.environ["GUARDRAIL_ID"],
"guardrailVersion": "1",
"trace": "enabled",
},
)
print(resp["output"]["message"]["content"][0]["text"])
print(resp["usage"]) # inputTokens / outputTokens for chargeback
bedrock_kb.py
import boto3

agent_rt = boto3.client("bedrock-agent-runtime", region_name="eu-central-1")
resp = agent_rt.retrieve_and_generate(
input={"text": "What is the notice period for contractors?"},
retrieveAndGenerateConfiguration={
"type": "KNOWLEDGE_BASE",
"knowledgeBaseConfiguration": {
"knowledgeBaseId": "KBID12345",
"modelArn": "arn:aws:bedrock:eu-central-1:111122223333:inference-profile/<eu-profile-id>",
"retrievalConfiguration": {"vectorSearchConfiguration": {"numberOfResults": 8}},
},
},
)
print(resp["output"]["text"])
for c in resp["citations"]:
print([r["location"] for r in c["retrievedReferences"]])

Implementation checklist for AWS:

  1. Organisation SCP: deny bedrock:* outside approved Regions; allow-list model IDs/inference profiles per OU.
  2. VPC interface endpoints (PrivateLink) for bedrock-runtime and bedrock-agent-runtime; no NAT path needed for inference.
  3. Model invocation logging to S3/CloudWatch with KMS — enable deliberately and set retention, since it captures prompts.
  4. Guardrail versions managed in Terraform/CDK; applications reference a version, never DRAFT.
  5. Cost: tag inference profiles (application inference profiles) per team; Cost Explorer by tag.

STACKIT: AI Model Serving and AI Model Experiments​

  • AI Model Serving: shared open-weight models (gpt-oss, Qwen3.x, Gemma 4, Llama 3.3, embeddings) via an OpenAI-compatible API in EU01, billed per token; model list, limits and prices are in Private and Sovereign AI.
  • AI Model Experiments: managed MLflow tracking server (release stage not confirmed; check STACKIT docs) with artifacts in STACKIT Object Storage; supports MLflow LLM tracing and evaluation.
  • Compute Engine GPU + SKE: dedicated A100/L40S/H100 when shared models do not fit.
  • Terraform: stackit_modelserving_token, stackit_modelexperiments_instance, stackit_modelexperiments_token in stackitcloud/stackit.
infra/stackit/mlops.tf
resource "time_rotating" "rotate" { rotation_days = 80 }

resource "stackit_modelserving_token" "app" {
project_id = var.project_id
name = "rag-app"
rotate_when_changed = { rotation = time_rotating.rotate.id }
}

resource "stackit_modelexperiments_instance" "mlflow" {
project_id = var.project_id
region = "eu01"
name = "rag-experiments"
deleted_experiment_retention = "30d"
}

resource "stackit_modelexperiments_token" "ci" {
project_id = var.project_id
region = "eu01"
name = "ci-evals"
instance_id = stackit_modelexperiments_instance.mlflow.instance_id
ttl_duration = "80d"
}
stackit_openai.py
import os
from openai import OpenAI

client = OpenAI(
base_url="https://api.openai-compat.model-serving.eu01.onstackit.cloud/v1",
api_key=os.environ["STACKIT_MODEL_SERVING_TOKEN"],
)
emb = client.embeddings.create(model="Qwen/Qwen3-VL-Embedding-8B", input=["Vertragslaufzeit", "notice period"])
chat = client.chat.completions.create(
model="google/gemma-4-31B-it",
messages=[{"role": "user", "content": "Draft a polite payment reminder in German."}],
)
print(len(emb.data[0].embedding), chat.choices[0].message.content)

OCI Generative AI​

  • Endpoint: OpenAI-compatible base URL https://inference.generativeai.<region>.oci.oraclecloud.com/openai/v1 with /responses, /chat/completions, /conversations, /files; Oracle recommends the Responses API as the primary entry point for most new agentic workloads.
  • Auth: Generative AI API keys for development; IAM-based auth (oci_openai auth classes with an httpx client) for production. The project OCID header (project= in the sample) is unconfirmed against Oracle's auth docs; check before use.
  • Hosting modes: on-demand, or dedicated AI clusters (pretrained models carry a minimum commitment; imported models bill with a one-hour minimum).
  • EU residency: per-region model availability changes; check Oracle's model-regions page. Unconfirmed: gpt-oss on demand in Germany Central, private endpoints in EU Sovereign Central since 2026-02, and Gemini in Frankfurt being hosted by Google.
  • Data Science service for notebooks, jobs and model deployment of custom models.
oci_responses.py
import os
from openai import OpenAI

client = OpenAI(
base_url="https://inference.generativeai.eu-frankfurt-1.oci.oraclecloud.com/openai/v1",
api_key=os.environ["OCI_GENAI_API_KEY"],
project=os.environ["OCI_GENAI_PROJECT_OCID"],
)
r = client.responses.create(model="openai.gpt-oss-120b", input="List three NIS2 obligations for IT providers.")
print(r.output_text)
Product (as of 2026-10)Use
Workers AIServerless inference on Cloudflare's GPUs via the AI binding or REST
AI GatewayProxy for Workers AI and external providers: caching, rate limiting, logging, retries/fallback, DLP (block/flag), Guardrails (Llama Guard), unified billing. Shares the AI binding with Workers AI (unified since 2026-08 per Cloudflare blog; default gateway auto-creation unconfirmed)
VectorizeVector DB: up to 1,536 dimensions, 20 M vectors per index, topK 50 with metadata / 100 without
AI Search (formerly AutoRAG)Managed index + retrieval pipeline over R2/websites, hybrid search, MCP endpoint; GA 2026-10-01, usage billing from 2026-11-01. Bindings ai_search (single instance) and ai_search_namespaces
wrangler.jsonc
{
"name": "rag-edge",
"main": "src/index.ts",
"compatibility_date": "2026-10-01",
"ai": { "binding": "AI" },
"vectorize": [{ "binding": "VEC", "index_name": "docs-bge-m3" }]
}
src/index.ts
export interface Env { AI: Ai; VEC: VectorizeIndex }

export default {
async fetch(req: Request, env: Env): Promise<Response> {
const { question } = await req.json<{ question: string }>();
const emb = await env.AI.run("@cf/baai/bge-m3", { text: [question] });
const hits = await env.VEC.query(emb.data[0], { topK: 8, returnMetadata: "all" });
const context = hits.matches.map((m) => m.metadata?.text).join("\n---\n");

const answer = await env.AI.run(
"@cf/meta/llama-3.1-8b-instruct-fp8",
{ messages: [
{ role: "system", content: `Answer only from this context:\n${context}` },
{ role: "user", content: question },
] },
{ gateway: { id: "default" } }, // logs, caching, DLP, rate limits via AI Gateway
);
return Response.json(answer);
},
};

Cloudflare is the right fit for edge-latency features, public-facing RAG over web content, and a SaaS-side gateway in front of several providers. For regulated personal data, check processing locations per model and use the EU data localisation options of the customer's Cloudflare plan.

Canonical: Charmed Kubeflow and Charmed MLflow​

  • Charmed Kubeflow 1.11/stable and 1.10/stable channels on Charmhub (as of 2026-10); deploys on Canonical Kubernetes or any conformant cluster with Juju.
  • Charmed MLflow 2.22/stable (MinIO artifact store, MySQL metadata) — integrates with Kubeflow; for MLflow 3.x features (LLM tracing, prompt registry) use upstream MLflow 3.x or STACKIT AI Model Experiments.
  • Fits customers standardised on Ubuntu Pro with long-term support needs, on-premise or on OpenStack.
juju add-model kubeflow
juju deploy kubeflow --trust --channel=1.11/stable
juju deploy mlflow --trust --channel=2.22/stable
juju status --watch 5s

Decision matrix​

CriterionAWS BedrockSTACKITOCI GenAICloudflareCanonical (self-run)
Data residencyEU Regions + EU inference profiles; US-headquartered providerGermany, German operator (Schwarz Group)EU and EU Sovereign regions; US-headquartered providerGlobal edge; regional options per planWherever the customer runs it
Model choiceWidest: Anthropic, OpenAI, Meta, Mistral, Amazon Nova, Qwen, Gemma, etc.Curated open-weight setCohere, Meta, OpenAI gpt-oss, xAI, GoogleOpen models on Workers AI; any provider via AI GatewayAny open model
Cost modelPer token; provisioned throughputPer token; GPU VMs per hourPer token; dedicated clustersPer neuron/token; gateway free tierHardware + ops
Ops burdenLowLow (shared) / medium (GPU)Low–mediumVery lowHigh
Managed RAGKnowledge BasesBuild on SKE/PostgresVector stores, agentsAI Search, VectorizeBuild
AgentsAgentCoreBuildOCI agentsAgents SDKBuild
Best forBroad model needs, existing AWS estateSovereignty-first EU workloadsOracle-centric estates, DB-adjacent AIEdge, web-facing, multi-provider gatewayFull control, air-gap, Ubuntu shops

Recommendation heuristics: sovereignty-sensitive data → STACKIT first (self-hosted on SKE if shared models are not enough); frontier proprietary models required → Bedrock in EU Regions with EU profiles; already on Oracle → OCI; public web workloads → Cloudflare. Always keep the gateway seam.

Portability patterns​

  1. One protocol. Use the OpenAI SDK everywhere; Bedrock (bedrock-runtime/openai/v1), STACKIT, OCI, Cloudflare and vLLM all expose compatible endpoints.
  2. Aliases, not model names, in application config.
  3. Gateway config owns credentials, fallbacks, budgets and logging policy.
  4. Contract tests per alias: tool calling, JSON-schema output, streaming, max context — run in CI against every backend before switching.
  5. Keep embeddings sticky. Changing the embedding model means re-indexing; pin embedding backends per index.
litellm/config.yaml
model_list:
- model_name: chat-default
litellm_params:
model: openai/openai/gpt-oss-120b
api_base: https://api.openai-compat.model-serving.eu01.onstackit.cloud/v1
api_key: os.environ/STACKIT_MODEL_SERVING_TOKEN
- model_name: chat-default-fallback
litellm_params:
model: bedrock/converse/eu.<inference-profile-id> # from `aws bedrock list-inference-profiles`
aws_region_name: eu-central-1
- model_name: chat-batch
litellm_params:
model: hosted_vllm/gpt-oss-120b
api_base: http://vllm.llm.svc:8000/v1

router_settings:
fallbacks: [{ "chat-default": ["chat-default-fallback"] }]
num_retries: 2

litellm_settings:
turn_off_message_logging: true

Install LiteLLM from a hash-pinned lockfile or a digest-pinned image: releases 1.82.7 and 1.82.8 on PyPI were compromised in March 2026 (incident).

AI tool integration​

Ready-made tools (coding assistants such as Claude Code and GitHub Copilot, workspace copilots such as Microsoft 365 Copilot or Gemini for Workspace, chat front-ends) deliver value fastest when rolled out as a managed platform, not as individual sign-ups.

Rollout technique​

ControlImplementation
IdentitySSO (SAML/OIDC) with SCIM provisioning; licences bound to IdP groups; enforce org login (e.g. Claude Code forceLoginOrgUUID, Copilot via enterprise-managed users)
Data controlsEnterprise plans with no training on customer data; retention settings; content exclusion for secrets and regulated repos; deny reads of .env and secret paths
Model routingPoint tools at the corporate gateway or a cloud provider (e.g. Claude Code on Bedrock) when residency requires it
MCP serversCentral allowlist of approved MCP servers by URL or command; internal registry for company servers; deny everything else
TelemetryOpenTelemetry export to the customer's collector: sessions, tokens, cost, accepted edits, per team
PolicyAcceptable-use policy, code review stays mandatory, AI-generated code labelled in PR templates, security scanning in CI unchanged

Claude Code — managed settings are deployed by MDM or as files in /Library/Application Support/ClaudeCode/ (macOS), /etc/claude-code/ (Linux/WSL) or C:\Program Files\ClaudeCode\ (Windows), or as server-managed settings from the claude.ai console; they override all user and project settings.

/etc/claude-code/managed-settings.json
{
"permissions": {
"deny": ["Read(./.env)", "Read(./secrets/**)"],
"disableBypassPermissionsMode": "disable"
},
"allowManagedPermissionRulesOnly": true,
"allowManagedMcpServersOnly": true,
"allowedMcpServers": [
{ "serverUrl": "https://mcp.internal.example.com/*" }
],
"env": {
"CLAUDE_CODE_ENABLE_TELEMETRY": "1",
"OTEL_METRICS_EXPORTER": "otlp",
"OTEL_LOGS_EXPORTER": "otlp",
"OTEL_EXPORTER_OTLP_PROTOCOL": "grpc",
"OTEL_EXPORTER_OTLP_ENDPOINT": "https://otel.internal.example.com:4317"
}
}

Managed settings apply above user and project settings, and env is merged per variable with managed values winning, so the collector endpoint cannot be redirected from a developer's settings file. Verify with /status or claude doctor, and check that claude_code.session.count arrives at the collector.

GitHub Copilot — since 2026-08 enterprise managed settings support allowedMcpServers / deniedMcpServers, matching servers by remote URL, local command or name (per the GitHub changelog). Third-party write-ups report that malformed policy fails closed; verify in GitHub's docs before relying on it. The file path copilot/managed-settings.json comes from the changelog summaries; confirm it in the enterprise managed-settings docs. Treat the allowlist as governance: GitHub's docs note configuration files can be edited locally. JetBrains support followed on 2026-08-18.

Workspace copilots — the work is mostly data hygiene: fix oversharing in SharePoint/Drive permissions before enabling the copilot, apply sensitivity labels, and run the vendor's readiness reports. Measure with the admin usage dashboards plus a short user survey after 30 days.

Measuring value​

  • Adoption: weekly active users / licences, sessions per user.
  • Throughput proxies: PR cycle time, review turnaround, lines accepted (as a signal, not a target).
  • Cost: tokens and spend per team from telemetry or gateway.
  • Quality: change failure rate and escaped defects unchanged or better.

Best practices​

  • Decide residency per data class first; it eliminates most platform options quickly.
  • Gateway from day one, even with a single provider.
  • Use EU-bound inference profiles on Bedrock and block global profiles where residency matters.
  • Everything — guardrails, knowledge bases, tokens, gateway config, managed settings — in Git and deployed by pipeline.
  • Rotate platform tokens automatically (time_rotating + Terraform, Secrets Manager rotation).
  • Roll out AI tools with telemetry on from the first pilot user, so value can be shown with data.

Pitfalls to design around​

  • Prompt logging captures personal data: enable it deliberately, encrypt, and set retention aligned with the DPIA.
  • Provider-specific features leak into code (Bedrock KB citations, AI Search APIs): wrap them behind an internal retrieval interface.
  • Shared-model catalogues change: deprecations on STACKIT, OCI retirements, Bedrock model lifecycle — monitor and keep aliases.
  • MCP servers are code execution: allowlist by URL/command and review them like dependencies.
  • Supply chain of the gateway is part of the attack surface: pin and scan.

Definition of done​

  • Decision record with the filled matrix and residency analysis signed off
  • Landing zone (accounts/projects, IAM, private endpoints, budgets, logging) in Terraform; pipeline applies it
  • Gateway deployed with aliases, fallbacks, per-team keys and budgets; contract tests green on every backend
  • Guardrails / DLP configured and versioned; tested with a red-team prompt set
  • RAG or agent workload evaluated against a golden set; results tracked (MLflow or platform evals)
  • Dashboards: tokens, spend, latency, error rate per team and model
  • Tokens and keys rotate automatically; no long-lived secrets in application config
  • AI tools: SSO + SCIM, managed settings deployed, MCP allowlist enforced, telemetry flowing, AUP published
  • 30-day adoption and value report delivered for the tool pilot

Sources​