What Google Cloud Shipped & The Enterprise Problem It Solves
In our architectural evaluation of large-scale AI infrastructure, the most persistent bottleneck has not been raw compute power, but rather the orchestration rigidity surrounding that compute. Historically, managing Tensor Processing Unit (TPU) capacity on Google Kubernetes Engine (GKE) required platform teams to provision separate, fixed TPU slice topology nodepools.
This tight coupling between physical hardware and logical topology created three severe enterprise problems:
- The "Delete-and-Create" Cycle: Changing workload requirements meant manually deleting and recreating static nodepools. This operational friction drastically increased startup times and prevented multi-host nodepools from hosting multiple, heterogeneous AI workloads.
- Massive Failure Domains: A single hardware fault within a static slice could crash the entire topology. Recovery was agonizingly slow because it required reprovisioning the entire static nodepool.
- Stranded Capacity: Because nodepools were fixed, smaller workloads could not be densely packed, leading to idle, expensive TPU capacity.
Google Cloud has solved this by shipping GKE Dynamic Slicing. This capability fundamentally decouples physical hardware provisioning from logical slice formation. It relies on a baseline conversion of all available TPU capacity into uniform nodepools composed of 16-VM (4x4x4) TPU cubes.
Dynamic slicing operates in two primary modes:
- Dynamic Super-Slicing: Utilizing TPU Optical Circuit Switching (OCS) technology, GKE dynamically alters the network configuration at the time of workload scheduling. It stitches multiple 4x4x4 cubes together to form massive topologies (e.g., a 4x4x16 super-slice comprising 64 VMs and 256 chips). This happens instantly, without provisioning new VMs.
- Dynamic Sub-Slicing: Conversely, a single 16-VM TPU cube can be partitioned on demand into smaller, independent sub-slices (e.g., 2x2x2). The scheduler activates the high-speed Inter-Chip Interconnect (ICI) network to form these sub-slices, allowing multiple distinct jobs to run concurrently and fully isolated inside the same physical cube.
The operational metrics are definitive: job startup times are slashed by up to 5x, and workload recovery is accelerated by 4.5x by isolating large slice failures to a single TPU cube and swapping unhealthy cubes in one to two minutes.
Real-World Use Cases: Where This Moves the Needle in the Field
To understand how this translates from infrastructure primitives to business value, we must examine how modern AI platforms are architected in production.
1. High-Throughput Enterprise Workloads (Inference Tail-Latency Isolation)
- The Everyday Problem: During burst traffic events (e.g., a major retail sale or a sudden spike in financial transaction analysis), inference workloads compete for resources. In a static environment, scaling up means waiting for new nodepools to provision, leading to unacceptable tail-latency and timeouts.
- How It Works in Practice: Using Dynamic Sub-Slicing, platform teams maintain a warm pool of uniform 16-VM TPU cubes. When burst traffic hits, the GKE scheduler instantly partitions these cubes into smaller 2x2x2 or 2x2x4 inference slices.
- The Tangible Impact: Inference capacity scales almost instantly. Tail-latency is strictly isolated because the sub-slices are physically partitioned at the ICI network level, ensuring that a noisy neighbor in one sub-slice cannot impact the throughput of another.
2. Zero-Trust Governance & IAM (Multi-Tenant Packing with Strict Boundaries)
- The Everyday Problem: Enterprise security mandates strict isolation between different business units (e.g., HR data processing vs. public customer support bots). Historically, this required entirely separate physical clusters or nodepools, leading to massive hardware waste.
- How It Works in Practice: Dynamic Sub-Slicing allows multiple workloads to run inside the same physical TPU cube while maintaining complete logical isolation. By combining this with GKE Workload Identity and VPC Service Controls, each sub-slice operates under a distinct, least-privilege Service Account.
- The Tangible Impact: Security teams achieve the strict Zero-Trust boundaries they require, while FinOps teams achieve the high utilization rates of multi-tenant packing.
3. Agentic Orchestration with ADK 2.0
- The Everyday Problem: Building reliable AI agents that can orchestrate complex tasks—like routing a user query to a lightweight model for triage, and then dispatching a heavy reasoning task to a custom model on a TPU—requires a robust framework. Prototyping is easy; production reliability is hard.
- How It Works in Practice: Engineering teams deploy the Agent Development Kit (ADK) 2.0 on Cloud Run. ADK 2.0 introduces graph workflows, allowing developers to weave deterministic code with adaptive AI reasoning. The Cloud Run agent acts as the brain, evaluating requests using Vertex AI Gemini models (like Gemini 3.8 Flash or Gemini 4 Carbon), and then dynamically submitting Kubernetes Jobs to GKE, requesting specific TPU topologies on demand.
- The Tangible Impact: This architecture separates the lightweight, event-driven orchestration layer (Cloud Run) from the heavy, stateful compute layer (GKE TPUs), optimizing both cost and performance.
Reference Architecture on Google Cloud
When we inspect production topologies, the most resilient architectures decouple the agentic orchestration layer from the heavy compute layer.
In this blueprint, Cloud Run hosts the ADK 2.0 agents. Cloud Run provides a serverless execution environment that scales to zero, handling incoming HTTP requests or asynchronous events via Eventarc. The ADK agent utilizes Vertex AI (specifically the Gemini 2.5 Pro or Gemini 3.8 Flash models) for complex reasoning, intent classification, and graph workflow execution.
When the agent determines that a heavy training or custom inference task is required, it dynamically generates a Kubernetes Job manifest specifying the required TPU topology (e.g., 2x2x4). This manifest is submitted to GKE, where the Dynamic Slicing scheduler instantly stitches or partitions the pre-provisioned TPU cubes via Optical Circuit Switching (OCS) to fulfill the request.
flowchart LR
%% Client & Entry
Client([Client / Eventarc]) -->|HTTPS / Event| CR[Cloud Run<br/>ADK 2.0 Agent]
%% Agentic Reasoning Layer
subgraph Agentic_Layer["Agentic Orchestration & Reasoning"]
CR <-->|REST / gRPC| VAI[Vertex AI<br/>Gemini 3.8 Flash / 4 Carbon]
CR <-->|State / Memory| CS[Cloud SQL PG17<br/>pgvector]
end
%% Heavy Compute Layer
subgraph Heavy_Compute["GKE Dynamic Slicing on TPUs"]
CR -->|Submit K8s Job<br/>Topology Request| GKE_API[GKE Control Plane]
GKE_API -->|Schedule| TPU_Pool[Uniform TPU Cubes<br/>16-VM 4x4x4]
TPU_Pool -.->|OCS Stitching| SuperSlice[Super-Slice<br/>4x4x16]
TPU_Pool -.->|ICI Partitioning| SubSlice1[Sub-Slice A<br/>2x2x2]
TPU_Pool -.->|ICI Partitioning| SubSlice2[Sub-Slice B<br/>2x2x2]
end
%% Security & Governance
subgraph Governance["Security & Guardrails"]
IAM[Cloud IAM<br/>Workload Identity] -.-> CR
IAM -.-> GKE_API
VPC_SC[VPC Service Controls] -.-> Agentic_Layer
VPC_SC -.-> Heavy_Compute
end
Step-by-Step Implementation
To implement this architecture, we must configure both the GKE Dynamic Slicing layer and the Cloud Run ADK agent layer.
1. Requesting a Dynamic TPU Slice in GKE
With Dynamic Slicing enabled on your GKE cluster, you no longer provision static nodepools for specific topologies. Instead, you maintain a pool of uniform 4x4x4 cubes. When submitting a workload, you specify the required topology directly in the Kubernetes manifest using the google.com/tpu-topology node selector.
Here is an example of a Kubernetes Job requesting a 2x2x4 sub-slice. The GKE scheduler will automatically find a healthy 16-VM cube and partition it via the ICI network to fulfill this request.
apiVersion: batch/v1
kind: Job
metadata:
name: dynamic-tpu-inference-job
spec:
template:
spec:
nodeSelector:
cloud.google.com/gke-tpu-accelerator: tpu-v5-lite-podslice
cloud.google.com/gke-tpu-topology: 2x2x4 # Dynamic Sub-Slicing Request
containers:
- name: tpu-workload
image: us-docker.pkg.dev/my-project/ai-repo/custom-inference:latest
resources:
limits:
google.com/tpu: 16 # 2 * 2 * 4 = 16 chips
restartPolicy: Never
2. Orchestrating with ADK 2.0 on Cloud Run
To orchestrate this dynamically, we deploy an ADK 2.0 agent on Cloud Run. This Python agent uses Vertex AI Gemini to analyze an incoming request, determines the required compute topology, and then uses the Kubernetes API to submit the Job manifest shown above.
First, ensure you have the ADK installed:
pip install google-adk kubernetes
Next, we define the ADK agent. Notice how we use the 2026-current gemini-3.8-flash model for fast, cost-effective reasoning.
import os
import yaml
from kubernetes import client, config
from google.adk import Agent
from google.adk.tools import Tool
# 1. Define a Custom Tool to submit the GKE Job
def submit_dynamic_tpu_job(topology: str, image_uri: str) -> str:
"""
Submits a Kubernetes Job to GKE requesting a specific dynamic TPU slice topology.
"""
# In production, authenticate via Workload Identity
config.load_incluster_config()
batch_v1 = client.BatchV1Api()
# Calculate chips based on topology (e.g., "2x2x4" -> 16)
dims = [int(d) for d in topology.split('x')]
total_chips = dims[0] * dims[1] * dims[2]
job_manifest = f"""
apiVersion: batch/v1
kind: Job
metadata:
generateName: dynamic-tpu-job-
spec:
template:
spec:
nodeSelector:
cloud.google.com/gke-tpu-accelerator: tpu-v5-lite-podslice
cloud.google.com/gke-tpu-topology: {topology}
containers:
- name: tpu-workload
image: {image_uri}
resources:
limits:
google.com/tpu: {total_chips}
restartPolicy: Never
"""
job_dict = yaml.safe_load(job_manifest)
try:
response = batch_v1.create_namespaced_job(body=job_dict, namespace="ai-workloads")
return f"Successfully submitted job {response.metadata.name} requesting topology {topology}."
except Exception as e:
return f"Failed to submit job: {str(e)}"
tpu_scheduler_tool = Tool(
name="submit_dynamic_tpu_job",
description="Submits a job to GKE requesting a dynamic TPU slice.",
func=submit_dynamic_tpu_job
)
# 2. Initialize the ADK 2.0 Agent
orchestrator_agent = Agent(
name="tpu-orchestrator",
model="gemini-3.8-flash", # Utilizing current 2026 models
instruction="""
You are an autonomous infrastructure orchestrator.
Analyze the user's workload description. Determine the optimal TPU topology
(e.g., 2x2x2 for small inference, 4x4x16 for large training).
Then, use the submit_dynamic_tpu_job tool to schedule the workload.
""",
tools=[tpu_scheduler_tool],
)
# Example Invocation (In production, this would be wrapped in a Flask/FastAPI app on Cloud Run)
if __name__ == "__main__":
user_request = "We need to run a mid-sized batch inference job using the image us-docker.pkg.dev/proj/repo/infer:v2. A 16-chip slice should be sufficient."
response = orchestrator_agent.run(user_request)
print(response)
To deploy this agent to Cloud Run, containerize the application and use the gcloud CLI:
gcloud run deploy tpu-orchestrator-agent \
--image us-docker.pkg.dev/my-project/ai-repo/tpu-orchestrator:latest \
--service-account adk-orchestrator-sa@my-project.iam.gserviceaccount.com \
--region us-central1 \
--allow-unauthenticated \
--set-env-vars="GOOGLE_CLOUD_PROJECT=my-project"
Production Readiness: FinOps, Quotas & Security Guardrails
Architecturally, the shift from static nodepools to dynamic slicing combined with serverless agentic orchestration fundamentally alters the unit economics of an AI platform.
📊 Production FinOps & TCO Simulation
To quantify this, we utilize our deterministic FinOps simulation. We compare a legacy static architecture (where orchestration agents run on overprovisioned, always-on GKE Autopilot clusters using standard Gemini API calls) against our modern blueprint (where ADK 2.0 agents run on scale-to-zero Cloud Run, utilizing Gemini Context Caching for repetitive reasoning tasks).
Note: TPU hardware costs are excluded from this specific table to focus strictly on the orchestration and LLM reasoning unit economics. However, GKE Dynamic Slicing inherently reduces TPU idle time and hardware waste by enabling multi-tenant sub-slicing.
📊 Production FinOps & TCO Simulation: Enterprise AI Orchestration: Legacy Static vs. Dynamic Serverless Agents (Based on the latest Google SKU information)
Production Workload Assumptions (us-central1 / asia-southeast1):
- 730 hours per month (standard billing month).
- Option A represents a legacy static architecture: overprovisioned GKE Autopilot clusters running idle orchestration workloads, combined with unoptimized, uncached Gemini 2.5 Pro API calls.
- Option B represents a modern dynamic architecture: ADK 2.0 Agents running on serverless Cloud Run (scaling to zero), combined with Gemini 2.5 Pro Context Caching for high-volume repetitive reasoning.
- Note: TPU hardware costs are excluded from this specific table to focus strictly on the orchestration and LLM reasoning unit economics. However, GKE Dynamic Slicing inherently reduces TPU idle time and hardware waste by enabling multi-tenant sub-slicing and eliminating static nodepool recreation.
| Architecture Option |
Google SKU Unit Price & Monthly Formula |
Estimated Monthly Cost |
| Legacy Static Orchestration (GKE + Standard Gemini) |
Static Agent Orchestration (GKE Autopilot vCPU): $0.0445/vCPU-hour × 14,600 = $649.70
Static Agent Orchestration (GKE Autopilot Memory): $0.00492/GiB-hour × 58,400 = $287.33
Gemini 2.5 Pro Input (Uncached, Standard): $1.25/1M input tokens × 500 = $625.00
Gemini 2.5 Pro Output: $10/1M output tokens × 100 = $1,000.00 |
$2,562.03 / mo |
| Dynamic Serverless Orchestration (Cloud Run + Cached Gemini) |
Dynamic Serverless Agents (Cloud Run vCPU): $2.4e-05/vCPU-second × 8,640,000 = $207.36
Dynamic Serverless Agents (Cloud Run Memory): $2.5e-06/GiB-second × 17,280,000 = $43.20
Gemini 2.5 Pro Input (Cached): $0.3125/1M cached input tokens × 400 = $125.00
Gemini 2.5 Pro Input (Uncached): $1.25/1M input tokens × 100 = $125.00
Gemini 2.5 Pro Output: $10/1M output tokens × 100 = $1,000.00 |
$1,500.56 / mo |
| Net FinOps Impact (Monthly Savings) |
Based on the latest Google SKU information |
41.4% TCO Reduction ($1,061.47 / mo) |
Official Google Cloud SKU Pricing Sources (2026.09): cloud.google.com, cloud.google.com, cloud.google.com
Quotas & Capacity Planning
When implementing GKE Dynamic Slicing, quota management shifts from static nodepool limits to dynamic TPU cube availability.
- TPU Quotas: Ensure your Google Cloud project has sufficient TPU v5e or v4 quotas allocated in your target region. Because dynamic slicing relies on a unified pool of 16-VM cubes, your quota must account for the maximum concurrent super-slices you intend to stitch together via OCS.
- Cloud Run Concurrency: For the ADK 2.0 orchestration layer, configure the Cloud Run maximum concurrent requests per instance (up to 1000) to optimize instance utilization and prevent cold starts during burst traffic events.
Security Guardrails & Zero-Trust
Dynamic infrastructure requires dynamic security.
- VPC Service Controls (VPC SC): Enforce strict perimeter security around both the Cloud Run agentic layer and the GKE TPU clusters. This ensures that data processed by the Gemini models and the custom models on TPUs cannot be exfiltrated outside the defined perimeter.
- GKE Workload Identity: Never use static service account keys. The Cloud Run ADK agent must run under a dedicated Service Account with the exact IAM permissions required to submit Jobs to the
ai-workloads namespace in GKE, and nothing more. Similarly, the pods running within the dynamic TPU sub-slices must use Workload Identity to securely access Cloud Storage buckets containing training data or model weights.
By decoupling logical topologies from physical hardware, GKE Dynamic Slicing fundamentally changes how we architect AI platforms. It eliminates the operational drag of static nodepools, maximizes TPU goodput, and, when combined with serverless agentic orchestration via ADK 2.0, delivers a highly resilient, cost-optimized foundation for enterprise AI.
