Agent Governance for Financial Services: What Compliance Teams Actually Need

If you work in financial services and want to deploy AI agents in production, your compliance team will ask questions that most agent frameworks can't answer. "Which model made that decision?" "Can you prove that agent never accessed customer PII?" "Show me the audit trail for that transaction." "What happens if a model gets compromised?" Standard frameworks silence at these questions because they weren't built with compliance in mind.

This post covers what regulators actually require, why standard agent frameworks fail, and how to build compliance into your agent infrastructure from day one.

What regulators actually care about

The SEC, OCC, and FINRA have issued guidance on algorithmic decision-making and model risk management. None of it uses the word "agent." But all of it applies.

SEC Guidance on Market Manipulation and Algorithmic Trading The SEC expects firms to have robust controls on algorithmic systems. That means knowing what the algorithm did, whether it operated within approved parameters, and whether it caused harm. For AI agents, this translates to: audit trails, approval workflows, and the ability to replay any execution and explain every decision.

OCC SR 11-7: Guidance on Third-Party Relationships Though originally about vendor risk, SR 11-7 now applies to AI models: you need to understand model performance, validate that the model does what you think it does, and monitor it for drift or degradation. For agents, this means model whitelisting, version tracking, and continuous monitoring.

FINRA Rule 4110: Anti-Money Laundering Compliance Agents that access customer data or make decisions about accounts must operate under AML controls. Transaction monitoring agents, customer screening agents, and reconciliation agents all trigger this requirement. You need to know which customer data was accessed, by which agent, at what time, and for what reason.

GLBA and CCPA: Data Privacy Agents that touch PII (names, account numbers, SSNs, transaction history) fall under privacy regulations. You must restrict which agents can access PII, log every access, and ensure agents don't leak data. This requires fine-grained access controls at the tool level.

Model Risk Management (Industry Standard) The industry consensus (shared by the big three banks' risk functions) is that any model making decisions in production must be: validated before deployment, monitored in production for performance degradation or drift, and audit-trailed so you can explain any output. For agents using LLMs, this is especially critical because LLMs are probabilistic and can produce different outputs from the same input.

The common thread: Auditability, governance, and transparency. Your agent system must be able to answer every question regulators ask.

Why Standard Frameworks Fall Short

Let's be direct: LangGraph, CrewAI, AutoGen, and similar tools are built for prototyping and research. They don't assume production constraints. When you try to run them in a regulated environment, you hit gaps immediately.

No Audit Trails Most frameworks don't record what happened. You get logs if you configure logging, but there's no immutable audit trail of agent actions. If you need to answer "Prove that agent X never called tool Y," you're out of luck.

No Model Governance Standard frameworks let you swap models dynamically. You can change the model an agent uses at runtime with a config change. Regulators want model changes to go through change control. You should only be able to use whitelisted, validated models. Dynamic model switching is the opposite of that.

No Access Controls on Tools If an agent has a tool, it can use it. There's no way to say "Agent A can call the transactions API, but only for the current customer's data, not all customers." This is a massive gap for anything touching PII or sensitive operations.

No Cost Attribution You can't answer "How much did that transaction cost?" If an agent calls an LLM three times and then calls an external API twice, you don't know which costs went where. For compliance reporting and budget enforcement, this is a problem.

No Declarative Governance Governance (who can do what) is embedded in code or not enforced at all. There's no declarative, version-controlled, reviewable source of truth for policies. This makes audits painful because you can't show a simple document proving your governance rules.

No Dead-Letter Handling If an agent fails, standard frameworks either retry forever or give up. For financial operations, you need failed transactions to go to a queue where an operator can investigate and replay them. Silent failures are compliance disasters.

What Compliance Teams Actually Ask

When a compliance officer sits down with an engineer to evaluate an agent system, these are the real questions:

"Can you prove which model made which decision?" Agents should log every model call: timestamp, input prompt, output, model name, version. You need to be able to reproduce any decision by replaying the execution with the exact same model and input. If the model version matters (and it does), you need to track and archive every version.

"Can you restrict which agents access what data?" Not all agents should be able to read the full customer database. A reconciliation agent should only access transaction records. A KYC agent should only access customer identity documents. A payment processing agent shouldn't access trade secrets. You need tool-level permissions that the runtime enforces, not recommendations in code.

"Show me the audit trail for this transaction." Every transaction touched by an agent should have an immutable record: which agent handled it, what it did, what data it accessed, what decisions it made, when it happened. This record should be tamper-evident (ideally immutable in storage).

"What happens if a model gets compromised or behaves unexpectedly?" You need to detect when a model starts producing nonsensical or harmful outputs. This might happen because of a prompt injection attack, a model update that broke behavior, or fine-tuning gone wrong. You need monitoring, alerting, and the ability to quickly rotate the model to a previous version or alternate provider.

"Can you enforce approval workflows?" For high-risk transactions (large transfers, policy changes, data access), compliance might require human approval before an agent acts. You need to pause agent execution, route to a human, wait for approval, then resume. This is different from logging everything after the fact.

"How do you limit blast radius if an agent goes rogue?" If an agent starts accessing data it shouldn't or calling tools in loops, you need rate limits, token budgets, and circuit breakers to stop it before it causes massive damage. You need alerts that fire before damage is done, not after.

"How are you handling LLM-specific risks?" LLMs can be jailbroken. They can hallucinate. They can be prompt-injected. Compliance teams want to know: How are you preventing prompt injection? How are you validating that model outputs are reasonable? How are you detecting and responding to jailbreak attempts? These are unique to LLM-based agents.

How Orloj Addresses These

Orloj was built from the ground up with compliance requirements in mind. Here's how it maps to what compliance teams need:

Audit Logging Every agent execution in Orloj generates a complete, immutable audit trail. You can query: "Show me every time agent X called tool Y in the last week." "Replay execution ID abc123." "Show me the cost breakdown for this agent." The audit trail is structured data (JSON), queryable, and never modified after creation.

Model Governance via ModelEndpoint Models aren't hardcoded. They're declared as ModelEndpoint resources. You whitelist which models your agents are allowed to use. Changing a model requires changing the manifest and applying it through version control. The runtime only routes to approved endpoints. You can't silently upgrade or swap models.

Tool-Level Permissions ToolPermission resources let you declare fine-grained access control. An agent has permission to call a tool, but only for certain data. "Call the transactions API" is broad. "Call the transactions API but only for the current customer's ID" is specific. Orloj enforces these constraints at the execution layer. Unauthorized calls fail immediately.

Cost Attribution Orloj tracks which agent incurred which costs. Every model call, every tool invocation, every token consumed is attributed to an agent. You can sum costs per agent, per day, per team. Reports are queryable and structured.

Declarative Governance as YAML All policies, roles, and permissions are declared in version-controlled YAML manifests. Your governance rules are code. Changes go through code review. You can show an auditor the exact manifest that was in effect on a given date. Version control is your paper trail.

Dead-Letter Queues Failed agent tasks don't disappear or retry forever. They go to a dead-letter queue (DLQ) where operators can investigate. Once the underlying issue is fixed, operators replay the task from the DLQ. For financial operations, this prevents silent failures from becoming compliance violations.

Token Budgets and Rate Limiting AgentPolicy resources let you set token budgets and rate limits. An agent can spend at most 100,000 tokens per day. An agent can call an external API at most 1000 times per hour. Once limits are hit, the agent is blocked. This contains blast radius and prevents runaway execution.

Structured Execution History Every step of an agent's execution is logged: timestamp, agent, tool, input, output, decision. This creates a searchable, auditable record. Compliance teams can investigate incidents by replaying the execution history.

Model Monitoring and Drift Detection Orloj integrates with observability tools to monitor model outputs. You can set up alerts: "If this model starts returning errors in >5% of calls, page the on-call." "If this model's latency jumps 50%, investigate." "If this model's output distribution shifts significantly, flag for review." Drift detection is part of the operational model.

Sample: Financial Services Agent System in YAML

Here's a simplified but realistic example: a reconciliation agent system for post-trade processing.

---
# Declare the models this system is allowed to use
models:
  gpt4_latest:
    provider: openai
    model: gpt-4
    version: "2026-05-20"
    cost_per_1m_input_tokens: 30
    cost_per_1m_output_tokens: 60
    environment: production

# Declare the tools (external systems)
tools:
  settlements_db:
    type: http
    endpoint: "https://internal-db.example.com/settlements"
    requires_auth: true
  trades_db:
    type: http
    endpoint: "https://internal-db.example.com/trades"
    requires_auth: true
  audit_log:
    type: http
    endpoint: "https://audit.example.com/log"
    requires_auth: true
  escalation_queue:
    type: queue
    endpoint: "sqs://escalations"

# Declare roles (groups of permissions)
roles:
  reconciliation_agent:
    description: "Can access trade and settlement data, log to audit trail"
    permissions:
      - tool: trades_db
        actions: ["read"]
        constraints:
          - filter: "settlement_date == today()"
      - tool: settlements_db
        actions: ["read", "write"]
        constraints:
          - filter: "settlement_date == today()"
      - tool: audit_log
        actions: ["write"]

  escalation_agent:
    description: "Can only read from escalation queue and send to audit log"
    permissions:
      - tool: escalation_queue
        actions: ["read"]
      - tool: audit_log
        actions: ["write"]

# Declare agents
agents:
  reconciliation_worker:
    model_endpoint: gpt4_latest
    role: reconciliation_agent
    system_prompt: |
      You are a reconciliation agent. Your job is to compare trade data with settlement data.
      If there is a mismatch, investigate and propose a resolution.
      Always be conservative: when in doubt, escalate to a human.
    tools:
      - trades_db
      - settlements_db
      - audit_log
      - escalation_queue

# Declare the multi-agent system
agent_system:
  name: post_trade_reconciliation
  description: "Daily reconciliation of trades and settlements"
  graph:
    nodes:
      - id: reconciliation_worker
        type: agent
      - id: escalation_agent
        type: agent
    edges:
      - from: reconciliation_worker
        to: escalation_agent
        label: "if unresolved_discrepancy"

# Declare policies
policies:
  reconciliation_worker:
    token_budget:
      per_execution: 50000
      per_day: 500000
    rate_limits:
      trades_db: 100_calls_per_minute
      settlements_db: 100_calls_per_minute
    fail_closed: true
    require_approval: false

  escalation_agent:
    token_budget:
      per_execution: 10000
      per_day: 50000
    rate_limits:
      escalation_queue: 1000_writes_per_day
    fail_closed: true
    require_approval: false

# Declare observability / monitoring
observability:
  structured_logging: true
  audit_trail: true
  trace_all_executions: true
  cost_attribution: true
  alerts:
    - name: "high_error_rate"
      condition: "error_rate > 0.05"
      action: "page_on_call"
    - name: "budget_exceeded"
      condition: "token_usage_today > 400000"
      action: "block_agent"
    - name: "model_drift"
      condition: "output_distribution_kl_divergence > 0.5"
      action: "page_data_science"

This manifest declares:

• Which models are allowed (whitelisted, versioned)
• Which tools agents can access (fine-grained)
• What data each agent can read or write (constraints)
• Who approves model changes (version control / code review)
• Token budgets and rate limits (blast radius containment)
• Audit logging and cost attribution (observability)
• Alerts for anomalies (drift detection)

The runtime enforces every constraint. If reconciliation_worker tries to write to trades_db, it fails immediately (only permitted to read). If it exceeds its token budget, it's blocked. If it goes into a loop calling the same tool 1000 times in a second, the rate limit stops it. All of this is automated, not reactive.

FAQ: Compliance Questions About Agent Systems

Q: Do AI agents fall under SR 11-7 (Model Risk Management)?

A: Yes. Any agent using an LLM or other ML model for decision-making should be treated as a model for SR 11-7 purposes. You need to: (1) validate the model before deployment, (2) monitor it in production for drift or performance degradation, (3) document its limitations and error rates, and (4) maintain an audit trail of decisions. The fact that it's an agent doesn't change the regulatory requirement. It just makes it more important to get governance right.

Q: How do we audit multi-agent systems where Agent A calls Agent B calls Agent C?

A: Trace the execution lineage. Every execution should have a unique ID. When Agent A calls Agent B, Agent B inherits that execution ID. When Agent B calls Agent C, Agent C also inherits it. The audit trail shows: execution X started at 2026-05-26 10:00:00, Agent A was invoked, which invoked Agent B, which invoked Agent C. Each agent's actions are recorded separately, but they're correlated by execution ID. This creates an unbroken chain of accountability.

Q: Can agents access customer PII?

A: Yes, but not without explicit permission and audit logging. If an agent needs to access customer SSN or account numbers, it should: (1) have explicit permission to access PII (declared in ToolPermission), (2) have a clear business reason (documented in the agent's system prompt and manifest), and (3) have every access logged with who accessed what when. The tool itself should enforce data masking (e.g., return only the last 4 digits of an SSN). And access patterns should be monitored: if an agent suddenly starts accessing 100x more customer records than normal, alert.

Q: What if an agent makes a bad decision? Can we reverse it?

A: Yes, but you need two things: (1) audit trail (so you know what happened), and (2) idempotent operations (so you can undo/redo safely). If an agent decides to reject a transaction, the decision should be logged. If you later determine it was wrong, you can reverse it by replaying a corrected version of the decision. This only works if the underlying operations are idempotent (e.g., "approve transaction ID X" is idempotent; running it once or twice produces the same result).

Q: How do we handle prompt injection attacks?

A: Multiple layers: (1) Don't let customer input go directly into agent prompts. Sanitize and validate input. (2) Use system prompts that are explicit about constraints ("You can only call these tools"; "You cannot access customer data unless explicitly provided"). (3) Monitor agent outputs for anomalies. If an agent starts calling tools in unexpected patterns, alert. (4) Rate-limit and budget all agents so malicious prompts can't cause runaway damage. (5) Rotate models and update system prompts regularly.

Q: What's the difference between agent governance and regular application governance?

A: Agents are harder to govern because their behavior is partially non-deterministic (LLMs are stochastic) and harder to predict (you can't read an LLM's "code" to understand what it will do). Regular applications have deterministic code paths; agents have probabilistic outputs. This means: (1) you need tighter constraints (explicit role-based access control, not trust the code), (2) you need more observability (trace every decision, not just errors), and (3) you need monitoring for drift (because the agent might behave differently tomorrow even though you didn't change the code). Standard application governance isn't enough.

Q: Who approves agents going to production?

A: That depends on your risk classification. Low-risk agents (informational only, no decisions) might only need a code review. Medium-risk agents (recommendations, internal analysis) might need risk sign-off. High-risk agents (decision-making, PII access, financial transactions) should require explicit approval from risk/compliance, plus validation that audit logging is working. This should be part of your change control process, not an afterthought.

Q: Can we use public cloud LLM APIs (OpenAI, Anthropic) or do we need on-prem?

A: Both can work, but on-prem or private cloud is safer for sensitive data. If you use a public API, you need: (1) data masking (no sending customer SSNs to OpenAI), (2) logging/audit on your side (the API provider doesn't audit for you), and (3) contract review (does your vendor agreement allow this use case?). Many banks prefer private endpoints or on-prem models for anything touching customer data. Orloj supports both patterns.

Q: How do we handle model drift or degradation in production?

A: Monitor outputs and performance continuously. Set thresholds for alerts: error rate >5%, latency >2x baseline, output distribution shift detected. When an alert fires, you have options: (1) roll back to a previous model version, (2) switch to an alternate model provider, (3) pause the agent and investigate, (4) reduce the agent's permissions while you debug. The key is automation: you can't wait for a daily report to detect drift. Monitoring and alerting need to be real-time.

Building Compliance into Your Agent Architecture

The pattern is simple:

1. Define your governance model in YAML. Who can do what? Which models are approved? What data can each agent access? Write it down, version-control it, review it.

2. Use an infrastructure layer that enforces governance. Don't hope agents follow rules; build the runtime to enforce them. Orloj does this with AgentPolicy, ToolPermission, and ModelEndpoint resources.

3. Instrument everything. Audit trails, structured logging, cost attribution. Make it easy to answer "what happened?" at 3am on Sunday.

4. Monitor in real-time. Detect drift, anomalies, and attacks as they happen. Build alerts for the things that would keep a compliance officer up at night.

5. Document and practice incident response. When something goes wrong (and it will), you need a playbook. Have you practiced rolling back a model? Disabling an agent? Extracting an audit trail for regulators? Do that before you need it.

---

Important Disclaimer

This post describes patterns and frameworks for thinking about agent governance. It is not legal advice and does not constitute regulatory compliance. Financial services regulations are complex, firm-specific, and constantly evolving. Before deploying any agent system, work with your compliance team, legal counsel, and risk management to ensure your implementation meets applicable rules and your firm's risk tolerance.

The examples and best practices here are informed by industry standards and regulatory guidance, but each firm's situation is unique. Use this as a starting point for conversations with your governance teams, not as a checklist you can work through alone.