Reliability

Runtime policy enforcement

Dynamically enabling/disabling tools by context.

Policy that changes mid-flight

The previous lessons assumed the agent's policy (which tools, which limits, which constraints) is fixed at startup. In production, that's not realistic. Policy needs to change at runtime: an incident is in progress and you want to disable a tool; a customer's tier changed and they get more capabilities; an audit is happening and you need every action approved.

This lesson covers runtime policy enforcement: how to change what the agent can do without restarting, without losing in-flight requests, and without creating windows where the policy is briefly inconsistent.

What needs to be runtime-mutable

Practical examples that come up:

Toggle tools on or off

A vendor's API is misbehaving. Disable the tool that calls it for an hour. Don't redeploy the agent.

Update allow lists

A customer just upgraded their plan. The agent serving them should now have access to a tool it didn't before. The change should be visible on their next request, not after the next deploy.

Change rate limits

A traffic spike is causing cost concerns. Tighten the per-session call budget for non-paying users from 100 to 30. Bring it back when the spike passes.

Switch to read-only mode

Maintenance window or incident response. The agent can read but cannot write or send. Resume normal operations afterward.

These changes need to apply quickly (seconds, not minutes), reliably (every agent picks up the change), and with rollback (reverting is easy).

Policy as data, not code

The fundamental shift: policy lives in a config store, not in source code. The agent reads policy on every relevant decision (or with a short TTL).

class Policy:
    def __init__(self, store):
        self.store = store
        self.cache = {}
        self.cache_ttl = 5  # seconds
 
    def get(self, key):
        cached = self.cache.get(key)
        if cached and cached["expires"] > time.time():
            return cached["value"]
        value = self.store.get(key)
        self.cache[key] = {"value": value, "expires": time.time() + self.cache_ttl}
        return value
 
 
def is_tool_enabled(policy, agent, tool):
    return tool in policy.get(f"agents.{agent}.allowed_tools")

The policy store is whatever your config infrastructure is (Redis, an admin DB, a feature flag service like LaunchDarkly, etc). The TTL trades freshness for load on the store; 5-30 seconds is typical.

Where to apply policy

Three layers, in increasing risk:

At tool-presentation time

The tool list shown to the model is filtered by current policy. The model never sees disabled tools, never tries to call them. Cleanest, but the policy decision is "made" at every turn.

def visible_tools(agent, all_tools, policy):
    enabled = policy.get(f"agents.{agent}.allowed_tools")
    return [t for t in all_tools if t.name in enabled]

At tool-execution time

Even if a tool slipped through (older model, jailbreak, inconsistent cache), the executor checks the policy before running.

def execute_tool(agent, tool, args, policy):
    if tool not in policy.get(f"agents.{agent}.allowed_tools"):
        return {"status": "denied", "reason": "tool currently disabled"}
    return TOOL_REGISTRY[tool](**args)

At the orchestrator level

Cross-cutting policies (rate limits, total budgets, mode flags like read-only) live at the orchestrator. Applied before any tool decision is even made.

Belt-and-suspenders: do all three. The cost is tiny; the benefit is that no policy change is more than one layer deep from being effective.

The cache invalidation problem

The TTL is a tradeoff. Too long and policy changes take minutes to propagate; too short and the policy store gets hammered.

Three patterns:

Pull (TTL-based)

What we just described. Each agent caches with a short TTL. Simple, no infrastructure beyond the store. Policy changes have a delay equal to the TTL.

Push (notifications)

The store notifies subscribed agents when policy changes. Agents invalidate their cache immediately. Faster propagation, more infrastructure.

Generation counters

Each policy change bumps a global generation number. Agents check the generation cheaply (e.g., one Redis GET); when it changes, they re-fetch the full policy. Compromise between pull and push.

For most production systems, generation counters are the right pattern. They're as fast as push for the common case (most reads see the same generation) and as simple as pull for the implementation.

Atomic policy changes

If your policy change is "increase the user's plan tier and give them tool X access," those are two writes. Between them, the agent might see "tier upgraded but no new tools" or "new tools but no tier change." The agent's behavior in the gap is undefined.

Make policy changes atomic where possible:

async def upgrade_user(user_id, new_tier):
    async with policy_store.transaction():
        await policy_store.set(f"users.{user_id}.tier", new_tier)
        await policy_store.set(f"users.{user_id}.allowed_tools", TOOLS_FOR_TIER[new_tier])

Either both writes happen or neither. The agent never sees an inconsistent state.

For policy stores that don't support transactions, use a single composite value: a JSON blob containing all the related fields, written atomically as one key.

Rolling policy changes

Some changes you don't want to flip globally; you want to roll out:

• 1% of traffic.
• 10%.
• Full rollout.

This is exactly what feature flag services solve. Your policy isn't a single boolean per setting; it's a function of the user, the request, and the current rollout stage.

def is_tool_enabled_for_user(user_id, tool):
    if not flag_service.is_enabled("tool_x_global", default=False):
        return False
    if user_id in flag_service.get_list("tool_x_allowlist"):
        return True
    return flag_service.is_enabled_for_user("tool_x_rollout", user_id)

For agent systems, rolling policy changes catches regressions early without affecting all users. Use it especially for new tools, new models, or new prompts.

Audit trails for policy

Every policy change should be logged. Who changed it, when, from what to what, why. This is operationally critical:

• Post-incident analysis ("what was the policy state when this happened?").
• Change attribution ("this regression started after the policy change").
• Compliance requirements ("show me every change to write-tool access").

def set_policy(actor, key, new_value, reason):
    old = store.get(key)
    store.set(key, new_value)
    audit_log.append({
        "actor": actor,
        "key": key,
        "from": old,
        "to": new_value,
        "reason": reason,
        "timestamp": time.time(),
    })

Treat the audit log like git history for your policy. It will save your team during incidents.

In-flight requests

A policy change happens during a long-running agent request. Two policies:

Apply to next decision

The current step finishes with old policy; the next decision uses new policy. Cleaner, more predictable.

Apply immediately

Whenever the agent next checks the policy, it gets the new value. Faster propagation but can produce mid-loop inconsistency (one tool call gets old policy, the next gets new).

For most production systems, "apply to next decision" is the right default. The TTL on the cache provides the natural boundary.

Common patterns

Kill switch

A boolean per tool: tool_x_disabled. Flipping it instantly stops new uses of the tool. The simplest and most useful runtime-policy feature.

Per-tenant limits

Each tenant has its own rate limits. Updating a tenant's limit takes effect within seconds. This is the standard "raise this customer's quota" workflow.

Read-only mode

A global flag that disables all write tools. Used during database maintenance, security incidents, etc.

Approval mode toggle

Force every action through human approval (Track 2 Module 5 lesson 3). Useful during a heightened-risk window.

Wrapping up reliability

This module covered the four core reliability patterns:

• Lesson 1 (failure taxonomy): knowing what kind of failure you have.
• Lesson 2 (retries and fallbacks): handling transient and dependency failures.
• Lesson 3 (output validation): catching successful-but-wrong outputs.
• Lesson 4 (runtime policy): changing the rules without redeploying.

Together they form a defense-in-depth pattern: most failures get retried; persistent failures hit fallbacks; surviving outputs get validated; and the whole envelope is shaped by runtime policy that operators control.

The next module is evaluation: how to know whether your reliability efforts are working, and how to test agent systems where every output is non-deterministic.

Key takeaway

Runtime policy treats agent constraints as data, not code: tools to enable, limits to apply, modes to switch. Policy lives in a config store and gets re-fetched with a short TTL or via push notifications. Apply at three layers (presentation, execution, orchestration). Make multi-key changes atomic. Audit every change. Production agents need this kind of operational lever; without it, every policy adjustment is a deploy.