Google Cloud Blueprint: New AI agent observability and governance across BigQuery and Looker — How Does It Work in Production?

TL;DR: As enterprises transition AI agents from experimental prototypes to production, deploying the model is only the first step; data and platform teams must also observe reasoning paths, govern data access, and control execution costs. Google Cloud has shipped a unified AI agent observability and governance framework across BigQuery and Looker, enabling first-party and custom third-party agents to stream structured telemetry natively into customer-owned BigQuery tables. This architecture provides zero-trust governance, granular FinOps controls, and deep semantic drift analysis without requiring separate export pipelines.

What Google Cloud Shipped & The Enterprise Problem It Solves

In our architectural evaluation of enterprise AI deployments, a recurring pattern emerges: the transition from proof-of-concept to production introduces severe operational friction. When an autonomous agent is deployed, it ceases to be a simple text-in/text-out function. It becomes a complex, multi-step orchestration engine that retrieves data, executes tools, and synthesizes responses. Platform engineering and data governance teams are consequently faced with a critical "black box" problem. They need to trace multi-step reasoning while protecting sensitive data, control token consumption and query spend, and evaluate answer accuracy before prompt or schema changes reach users.

To address these operational imperatives, Google Cloud has introduced expanded AI agent observability and evaluation capabilities across the Data Cloud. This release establishes a unified telemetry framework that standardizes how observability data is captured, stored, and analyzed, regardless of whether the agent is a first-party (1P) managed service or a custom third-party (3P) application.

The Unified Telemetry Framework

The core innovation lies in the separation of operational telemetry from conversation content, allowing administrators to govern each layer independently.

For operational telemetry, the framework integrates natively with Cloud Monitoring and Cloud Trace. At the project or organization level, administrators can enable aggregate metrics (such as request volume, latency, and token usage) and step-by-step trace spans for each request. Crucially, to protect sensitive data, these operational metrics and sanitized trace spans automatically exclude customer core content, such as user prompts, model responses, internal thoughts, and generated SQL.

For deep transcript auditing and semantic analysis, the framework introduces governed conversation logging in BigQuery. When enabled, a Google-managed service agent streams user prompts, responses, and turn-level telemetry directly into a customer-owned BigQuery table (typically named agent_events). This ingestion utilizes the BigQuery Storage Write API, ensuring high-throughput, low-latency streaming. Because the data lands in a customer-owned dataset, it immediately inherits the environment's existing Identity and Access Management (IAM) permissions, Customer-Managed Encryption Keys (CMEK), and VPC Service Controls.

This capability extends across two primary deployment models:

1. First-Party Data Agents: Out-of-the-box Conversational Analytics agents in BigQuery and Looker emit telemetry automatically. Queries routed to BigQuery data agents in Gemini Enterprise through agent-to-agent (A2A) workflows send logs straight into the designated project.
2. Custom Third-Party Agents: For agents built with the Agent Development Kit (ADK) or LangGraph, developers can instrument their code with a single-line SDK plugin (BigQueryAgentAnalyticsPlugin). This streams full tool-invocation execution trees, time-series telemetry, and multimodal payloads directly into BigQuery.

Once the telemetry resides in BigQuery, teams can leverage prebuilt dashboards in BigQuery Studio or Looker System Activity to track engagement, token usage, and latency. Furthermore, because the logs are natively queryable, data teams can run BigQuery AI functions (such as AI.GENERATE to summarize error patterns and AI.EMBED with VECTOR_SEARCH to cluster recurring issues) for post-hoc semantic drift clustering and LLM-as-a-judge sentiment analysis directly within the data warehouse.

Real-World Use Cases: Where This Moves the Needle in the Field

When inspecting production topologies, the theoretical benefits of observability must translate into tangible engineering outcomes. Here is how this architecture applies to concrete enterprise scenarios.

1. High-Throughput Enterprise Workloads

• The Everyday Problem: Under burst traffic conditions (e.g., a retail customer service agent during a flash sale), agents experience tail-latency spikes. Platform teams struggle to identify whether the bottleneck is the LLM inference, the database retrieval, or the external API tool execution, leading to SLA breaches and quota exhaustion.
• How It Works in Practice: By instrumenting the custom ADK agent with the BigQueryAgentAnalyticsPlugin, every request generates a full trace tree. The telemetry separates the latency of the gemini-2.5-pro model invocation from the latency of the Cloud SQL vector search. Engineers can query the agent_events table in BigQuery to isolate specific tool executions that exceed the P99 latency threshold.
• The Tangible Impact: Deterministic scaling. Platform teams can pinpoint the exact component causing the bottleneck, allowing them to optimize the specific database query or implement caching strategies, thereby restoring SLA adherence without blindly over-provisioning compute resources.

2. Zero-Trust Governance & IAM

• The Everyday Problem: Security teams block the deployment of AI agents because they cannot audit what data the agent is accessing or ensure that the agent isn't leaking Personally Identifiable Information (PII) into its logs.
• How It Works in Practice: The architecture relies on BigQuery's built-in privacy guardrails. Before each turn is logged, a fail-closed check is enforced. If a knowledge source utilizes row-level security, column-level policy tags, or dynamic data masking, the logging is skipped to prevent data leakage. Concurrently, fail-open execution ensures the user's chat session remains uninterrupted even if a logging error occurs. The entire agent_events dataset is wrapped in VPC Service Controls.
• The Tangible Impact: Unblocking production deployments. Security teams gain a fully auditable, immutable ledger of agent behavior that strictly adheres to data residency and privacy regulations, satisfying compliance requirements without hindering developer velocity.

3. Production FinOps & Unit Economics

• The Everyday Problem: Agentic workflows are inherently unpredictable in their token consumption. A single complex user query might trigger a loop of internal reasoning and tool calls, resulting in runaway token spend that destroys the unit economics of the application.
• How It Works in Practice: Telemetry streamed into BigQuery includes precise token counts (input, output, and cached) for every model invocation. This data is visualized in Looker System Activity dashboards, providing real-time visibility into token consumption per session, per user, or per tool.
• The Tangible Impact: Granular cost attribution. FinOps teams can establish baseline usage, identify inefficient prompts or looping behaviors, and implement active controls or caching mechanisms to optimize the cost-per-request, ensuring the AI initiative remains financially viable.

Reference Architecture on Google Cloud

Architecturally, deploying a robust, observable AI agent requires a decoupled topology that separates the ingress, execution, intelligence, knowledge, and observability layers. The following reference architecture illustrates a custom 3P agent deployment utilizing the Agent Development Kit (ADK), Cloud Run, Vertex AI, Cloud SQL, and BigQuery.

flowchart LR
    %% Define Styles
    classDef user fill:#f9f9f9,stroke:#333,stroke-width:2px
    classDef compute fill:#e8f0fe,stroke:#1a73e8,stroke-width:2px
    classDef ai fill:#fce8e6,stroke:#d93025,stroke-width:2px
    classDef data fill:#e6f4ea,stroke:#1e8e3e,stroke-width:2px
    classDef observe fill:#fef7e0,stroke:#f9ab00,stroke-width:2px
    classDef security fill:#f3e8fd,stroke:#9334e6,stroke-width:2px,stroke-dasharray: 5 5

    %% Nodes
    User((Client / User)):::user
    
    subgraph Vpc_Sc["VPC Service Controls Perimeter"]
        direction TB
        
        CR[Cloud Run<br/>ADK Custom Agent]:::compute
        
        subgraph Intelligence_Layer["Vertex AI"]
            Gemini[Gemini 2.5 Pro<br/>Inference API]:::ai
        end
        
        subgraph Knowledge_Layer["Cloud SQL"]
            PG17[(Cloud SQL PG17 HA<br/>Vector Search)]:::data
        end
        
        subgraph Observability_Layer["Google Data Cloud"]
            BQ[(BigQuery<br/>agent_events Table)]:::data
            Looker[Looker<br/>System Activity]:::observe
            Trace[Cloud Trace &<br/>Monitoring]:::observe
        end
    end

    %% Connections
    User -- HTTPS Request --> CR
    CR -- "Prompt & Context" --> Gemini
    Gemini -- "Generated Response" --> CR
    CR -- "Tool Execution (pgvector)" --> PG17
    PG17 -- "Retrieved Context" --> CR
    
    %% Telemetry Connections
    CR -- "Storage Write API<br/>(Execution Trees)" --> BQ
    CR -- "Operational Metrics<br/>(Sanitized Spans)" --> Trace
    BQ -- "Semantic Analysis &<br/>Dashboards" --> Looker
    
    %% Annotations
    class VPC_SC security

Architectural Component Breakdown

1. Execution Layer (Cloud Run): The custom agent, built using the Agent Development Kit (ADK), is containerized and deployed on Cloud Run. This provides a serverless, auto-scaling environment capable of handling high-throughput concurrent requests. The ADK orchestrates the reasoning loop, deciding when to call tools and when to query the LLM.
2. Intelligence Layer (Vertex AI): The agent utilizes gemini-2.5-pro for complex reasoning and tool calling. The model evaluates the user prompt, determines the necessary actions, and synthesizes the final response.
3. Knowledge Layer (Cloud SQL PG17 HA): For Retrieval-Augmented Generation (RAG) and tool execution, the agent relies on Cloud SQL for PostgreSQL High Availability. Utilizing the pgvector extension, it performs similarity searches to retrieve relevant context. The HA configuration ensures that a standby instance is synchronously replicated, providing automatic failover in the event of a zonal outage, which is critical for maintaining agent uptime.
4. Observability Layer (BigQuery & Looker): As the agent operates, the BigQueryAgentAnalyticsPlugin intercepts the execution flow. It streams structured telemetry—including full trace trees, tool arguments, and token counts—into the agent_events table in BigQuery using the high-throughput Storage Write API (gRPC). Simultaneously, sanitized operational metrics are sent to Cloud Trace and Cloud Monitoring. Looker connects to BigQuery to visualize this data, providing actionable insights into performance and cost.

Step-by-Step Implementation

To implement this observability framework for a custom ADK agent, engineers must configure the Google Cloud environment, set up the necessary IAM permissions, and instrument the application code.

1. Environment Preparation & IAM Configuration

First, enable the required APIs in your Google Cloud project. This includes Vertex AI, BigQuery, Cloud Trace, and the Storage Write API.

# Enable required Google Cloud APIs
gcloud services enable \
    aiplatform.googleapis.com \
    bigquery.googleapis.com \
    bigquerystorage.googleapis.com \
    cloudtrace.googleapis.com \
    run.googleapis.com

Next, ensure the service account used by the Cloud Run application has the necessary permissions to write to BigQuery and emit traces.

PROJECT_ID="your-production-project"
SERVICE_ACCOUNT="agent-runner@${PROJECT_ID}.iam.gserviceaccount.com"

# Grant BigQuery Data Editor for telemetry ingestion
gcloud projects add-iam-policy-binding ${PROJECT_ID} \
    --member="serviceAccount:${SERVICE_ACCOUNT}" \
    --role="roles/bigquery.dataEditor"

# Grant Cloud Trace Agent for operational spans
gcloud projects add-iam-policy-binding ${PROJECT_ID} \
    --member="serviceAccount:${SERVICE_ACCOUNT}" \
    --role="roles/cloudtrace.agent"

2. Instrumenting the ADK Agent (Python)

In your Python application, you will initialize the ADK and configure the BigQueryAgentAnalyticsPlugin. This plugin automatically handles the schema creation and streaming ingestion via the Storage Write API.

import os
from google.cloud import bigquery
from adk.agent import Agent
from adk.plugins.bigquery_analytics import BigQueryAgentAnalyticsPlugin
from vertexai.generative_models import GenerativeModel

PROJECT_ID = os.getenv("GOOGLE_CLOUD_PROJECT")
DATASET_ID = "agent_observability_prod"
TABLE_ID = "agent_events"

# Initialize the BigQuery client
bq_client = bigquery.Client(project=PROJECT_ID)

# Configure the BigQuery Analytics Plugin
# This plugin intercepts ADK execution and streams telemetry
analytics_plugin = BigQueryAgentAnalyticsPlugin(
    client=bq_client,
    dataset_id=DATASET_ID,
    table_id=TABLE_ID,
    enable_trace=True, # Emits sanitized spans to Cloud Trace
    log_full_prompts=True # Opt-in to log full conversation content
)

# Initialize the 2026 active model
model = GenerativeModel("gemini-2.5-pro")

# Initialize the ADK Agent with the observability plugin
agent = Agent(
    model=model,
    tools=[...], # Your custom tools (e.g., Cloud SQL vector search)
    plugins=[analytics_plugin]
)

# Execute a request
response = agent.run("Analyze the Q3 financial report and summarize the key risk factors.")
print(response.text)

3. In-Warehouse Drift Analysis with BigQuery AI

Once the telemetry is landing in the agent_events table, data teams can perform advanced analysis without moving the data. For example, you can use the AI.GENERATE function to automatically summarize error patterns from failed tool executions.

-- BigQuery SQL: Summarize recurring tool execution errors
SELECT
  tool_name,
  COUNT(*) as error_count,
  ML.GENERATE_TEXT(
    MODEL `your_project.your_dataset.gemini_2_5_flash_model`,
    (
      SELECT CONCAT(
        'Summarize the root cause of these agent tool errors: ',
        STRING_AGG(error_message, ' | ' LIMIT 50)
      )
    ),
    STRUCT(
      0.2 AS temperature,
      256 AS max_output_tokens
    )
  ).text AS error_summary
FROM
  `your_project.agent_observability_prod.agent_events`
WHERE
  execution_status = 'FAILED'
GROUP BY
  tool_name
ORDER BY
  error_count DESC;

Production Readiness: FinOps, Quotas & Security Guardrails

Deploying agentic systems at scale requires rigorous adherence to the Well-Architected Framework: Cost optimization pillar. This involves aligning spending with business value, optimizing resource usage, and continuously monitoring consumption.

📊 Production FinOps & TCO Simulation

To provide a deterministic view of unit economics, we have executed a FinOps simulation comparing two architectural approaches for handling 10 million monthly agent interactions. This simulation utilizes official Google Cloud SKU pricing and avoids unverified estimations.

📊 Production FinOps & TCO Simulation: Production Agent Observability & Execution TCO (10M Interactions/Month) (Based on the latest Google SKU information)

Production Workload Assumptions (us-central1 / asia-southeast1):

• 10 million monthly agent interactions
• Option A uses 1P BigQuery Conversational Analytics with Gemini 2.5 Flash (1000 input tokens, 500 output tokens per interaction)
• Option B uses 3P ADK Custom Agent on Cloud Run with Gemini 2.5 Pro (1000 input tokens, 500 output tokens per interaction)
• Option B Cloud Run instances process 10M requests, averaging 2 seconds per request, using 2 vCPU and 4 GiB memory
• Option B Cloud SQL HA uses 8 vCPU Enterprise Plus for 730 hours/month
• BigQuery telemetry storage adds 500 GiB logical storage per month for both options
• Option A uses 100 BigQuery Enterprise slots for 730 hours/month for analytics

Architecture Option	Google SKU Unit Price & Monthly Formula	Estimated Monthly Cost
Option A: 1P BigQuery Conversational Analytics (Gemini 2.5 Flash)	Gemini 2.5 Flash Input (10M * 1000 tokens): $0.15/1M input tokens × 10 = $1.50 Gemini 2.5 Flash Output (10M * 500 tokens): $0.6/1M output tokens × 5 = $3.00 BigQuery Logical Storage (Telemetry): $0.02/GiB-month × 500 = $10.00 BigQuery Enterprise Slots (Analytics): $0.06/slot-hour × 73,000 = $4,380.00	$4,394.50 / mo
Option B: 3P Custom ADK Agent (Gemini 2.5 Pro + Cloud Run + Cloud SQL)	Gemini 2.5 Pro Input (10M * 1000 tokens): $1.25/1M input tokens × 10 = $12.50 Gemini 2.5 Pro Output (10M * 500 tokens): $10/1M output tokens × 5 = $50.00 Cloud Run vCPU (10M * 2s * 2 vCPU): $2.4e-05/vCPU-second × 40,000,000 = $960.00 Cloud Run Memory (10M * 2s * 4 GiB): $2.5e-06/GiB-second × 80,000,000 = $200.00 Cloud SQL Ent Plus vCPU (8 vCPU * 730h): $0.0826/vCPU-hour × 5,840 = $482.38 BigQuery Logical Storage (Telemetry): $0.02/GiB-month × 500 = $10.00	$1,714.88 / mo
Net FinOps Impact (Monthly Savings)	Based on the latest Google SKU information	61.0% TCO Reduction ($2,679.62 / mo)

Official Google Cloud SKU Pricing Sources (2026.09): cloud.google.com, cloud.google.com, cloud.google.com, cloud.google.com

Quotas & Scalability Guardrails

When designing for high-throughput observability, specific quotas must be monitored and managed:

1. BigQuery Storage Write API Throughput: The BigQuery overview details that the Storage Write API is designed for massive scale, but default project quotas apply to concurrent connections and throughput (MB/s). For workloads exceeding thousands of requests per second, proactive quota increase requests via the Google Cloud Console are mandatory to prevent RESOURCE_EXHAUSTED errors during telemetry ingestion.
2. Vertex AI Token Limits: Agentic reasoning loops can consume tokens exponentially faster than standard chat applications. Platform teams must monitor the GenerateContent API quotas (Requests Per Minute and Tokens Per Minute) for gemini-2.5-pro and gemini-2.5-flash. Implementing context caching (vertex_gemini_25_pro_cache_storage_1m_hr) is a critical architectural pattern to mitigate quota exhaustion and reduce latency for repetitive tool executions.

Security & Compliance Guardrails

To ensure the observability framework does not become a vector for data exfiltration or compliance violations, the following security guardrails must be enforced:

1. VPC Service Controls (VPC-SC): The entire architecture—including Cloud Run, Vertex AI, Cloud SQL, and BigQuery—must be enclosed within a VPC-SC perimeter. This prevents data exfiltration by ensuring that the agent_events table cannot be queried from outside the authorized network, even if IAM credentials are compromised.
2. Customer-Managed Encryption Keys (CMEK): The BigQuery dataset storing the conversation logs must be encrypted using CMEK managed in Cloud Key Management Service (KMS). This provides cryptographic control over the telemetry data, allowing security teams to instantly revoke access by disabling the key.
3. Dynamic Data Masking & Row-Level Security: Leverage BigQuery's native policy tags to implement dynamic data masking on columns containing sensitive information (e.g., user IDs or financial data) within the agent_events table. This ensures that while platform engineers can analyze aggregate performance metrics, they cannot view the raw PII contained within the conversation transcripts.