AI Agent Credential Management: Production Guide | MewCP | MewCP
AI Agent Credential Management Is Harder Than It Looks
by Rohit Gite, Founder CTO @MewCP··17 min read
Encrypting tokens at rest is the part everyone already does. The parts that take agents down are refresh races, tokens that outlive a revocation, and secrets riding along in a prompt.
Ana and Ben both use your agent. Both connected their Gmail. Both ask it to summarize this morning's unread mail. Day 19 covered the rule that keeps their two mailboxes apart: the credential resolves from the caller's identity, on every request, and never lives in the process. AI agent credential management is everything underneath that one sentence, and it is where most agent products spend their first security incident.
The rule is short. The implementation is a storage design, a lifecycle, a distributed locking problem and a leak surface. Each of those has a failure mode that only appears under concurrency or under a second user, which is exactly why they survive code review and then show up in production.
Five things have to be true at once for Ana and Ben to coexist:
Their secrets are stored so that a database dump is not a breach, and Ana's row cannot be decrypted into Ben's context.
Access tokens are refreshed before they expire, exactly once per user, no matter how many workers notice at the same moment.
A revocation takes effect in seconds, including for work already queued.
Missing permissions produce a consent flow, not a 403 the model tries to reason its way around.
No secret ever reaches the model, the logs, the traces or an exception message.
That last one is where we should start, because almost every agent codebase begins by violating it.
Why AI agent credential management breaks at user two
Here is the shortcut. It works, it ships, and it is the reason this post exists.
SYSTEM = f"""
You are a mail assistant.
Use this token when calling the Gmail API: {access_token}
"""
The usual objection is that it is insecure. The more useful objection is that it is not a credential system at all. Look at what you gave up in three lines:
Per-user isolation. There is one string in one template. Ana's run and Ben's run both render the same prompt unless every call site remembers to rebuild it, and the one that forgets fails silently with a 200 OK and the wrong mailbox.
Rotation. You cannot rotate a value that has already been copied into conversation history, a prompt cache, a trace span and an eval fixture. Rotation only works when there is exactly one authoritative copy.
Scoping. A token in a prompt is available to every tool the model can reach, including the ones the user never authorized for this task. The blast radius of that token is now the whole tool surface rather than one call.
Revocation. Ana clicks disconnect. You delete the row. The token is still sitting in a checkpointed agent state, a queued job payload and last night's trace export, and every one of those is a live copy. When something then sends mail nobody asked for, you cannot even tell which run did it, because the credential was never attached to a request.
And then the security part, which is worse than the usual framing. Anything you put in the context window is reachable by anything else in the context window. A model that reads a web page, a PDF, a calendar invite or an email body is reading attacker-controllable text, and that text sits in the same window as your token. OWASP tracks this as two separate entries in its Top 10 for LLM Applications, prompt injection and system prompt leakage, and the second one exists specifically because teams treat the system prompt as a safe container for credentials and architecture details. It is not a container. It is the most copied artifact in your entire system.
So the working rule for the rest of this post: the context window is an output surface, not a storage layer. Intent goes in. Secrets never do. The model decides what should happen, the transport layer decides what credential proves it, and those two facts meet for the first time inside an HTTP client.
async def gmail_list_unread(ctx: RequestContext, limit: int = 10) -> str: """List unread subjects for whoever made this request.""" token = await credentials.resolve(ctx, "google", {GMAIL_READONLY}) async with httpx.AsyncClient(timeout=15.0) as http: response = await http.get( "https://gmail.googleapis.com/gmail/v1/users/me/messages", params={"q": "is:unread", "maxResults": limit}, headers={"Authorization": f"Bearer {token.reveal()}"}, ) ...
The model called a function with a number in it. The token appeared one line before it was used and never entered a string the model can see. Everything below is about making credentials.resolve trustworthy.
Storage: envelope encryption bound to a user
Encrypting the token column with one application-level key is better than plaintext and worse than it looks. That key lives in your config, which means every process can decrypt every row, and rotating it means reading and rewriting the entire table through your application.
Envelope encryption fixes both. A root key in a KMS wraps a fresh data key for every row. The data key encrypts one credential. The root key never leaves the KMS and your application never holds long-lived key material.
The part worth getting right is the encryption context. In AWS KMS it is a set of non-secret key value pairs that KMS uses as additional authenticated data. The same context must be supplied to decrypt or the operation fails, and KMS writes it in plaintext to CloudTrail so you can see which key was used for what. Non-secret and logged, so it holds identifiers and never holds tokens.
Two independent bindings fall out of that, and they are the reason to bother. KMS refuses to unwrap the data key unless the caller supplies the same context, so a service that can only assert user_id=ana cannot unwrap Ben's key. AES-GCM refuses to decrypt unless the same bytes are supplied as additional authenticated data, so a ciphertext copied from Ana's row into Ben's row fails authentication instead of quietly working. A row that gets moved fails twice, at two different layers, for two different reasons.
Three practical notes on the same code.
Keep issued_at, expires_at, scopes and key_version as ordinary plaintext columns. You need to find credentials expiring in the next hour, audit which scopes are live and drive a rewrap job, and none of that should require decrypting anything. Never index on a column that holds ciphertext, and never bury something you plan to filter on inside the blob.
You cannot reliably scrub plaintext_key afterwards, because Python bytes are immutable and the interpreter may already have copied them. Hold it in a bytearray and zero it if that matters to your threat model, but treat the narrow scope as the real control.
Rotation becomes a background job rather than a migration. Read a page of rows, unwrap each data key under its recorded key_version context, rewrap under the new one, write back. The ciphertext never changes, so the job is cheap and interruptible. When you extend this from users to tenants, the tenant identifier goes into the encryption context and, for anything regulated, into a separate root key per tenant. That is the next post's problem.
The token lifecycle, and the three places it goes wrong
A token is not a value you store. It is a small state machine with a clock attached, and the clock is owned by somebody else.
Refresh before expiry, not on 401
The common pattern is to call the API, catch a 401, refresh, retry. It works in a demo and it is the wrong trigger in production.
401 is ambiguous: expired token, revoked grant, wrong audience and a provider incident all look identical from the outside, so refreshing on all four means hammering a token endpoint during an outage. It also doubles latency at every expiry boundary, and it breaks long calls, because a streaming response or a 90 second export that starts with 20 seconds of validity left fails partway through and the retry restarts work you already paid for. Expiry also clusters. Everyone who authorized during your Monday launch expires within the same minute on Tuesday.
Refresh proactively, at a fraction of the token's lifetime, with jitter:
import randomREFRESH_AT = 0.75 # refresh once three quarters of the lifetime is gonedef needs_refresh(cred: StoredCredential, now: float | None = None) -> bool: now = now if now is not None else time.time() lifetime = max(cred.expires_at - cred.issued_at, 1.0) deadline = cred.issued_at + lifetime * REFRESH_AT # Up to 60s of negative jitter so a fleet that authorized together # does not refresh together. return now >= deadline - random.uniform(0, 60)
For Google's roughly one hour access tokens that gives you a fifteen minute window to fail, retry and alert before anything user visible breaks. Keep the 401 handler, but treat it as a signal that something is wrong rather than as the normal path.
One refresh per user, across every worker
Ana's agent fires six tool calls concurrently. All six read a credential inside the refresh window. All six call the token endpoint.
With refresh token rotation, which OAuth 2.1 requires for public clients, the first response invalidates the refresh token the other five are holding. RFC 9700, the OAuth 2.0 Security Best Current Practice published in January 2025, is explicit about what a good authorization server does next: reuse of an already rotated refresh token is treated as evidence of theft, and the server revokes the whole token family. Your concurrency bug is indistinguishable from an attacker replaying a stolen token, so the provider does the correct thing and logs Ana out entirely.
An in-process asyncio.Lock is not enough, because the six calls may be on six pods. You need a lock the whole fleet agrees on, and if you already run Postgres you already have one.
import hashlibLOCK_NAMESPACE = 0x63726564 # "cred", any stable int4 constantdef lock_key(user_id: str, provider: str) -> int: digest = hashlib.blake2b(f"{user_id}:{provider}".encode(), digest_size=4).digest() return int.from_bytes(digest, "big", signed=True)class Credentials: def __init__(self, pool, store, providers) -> None: self.pool, self.store, self.providers = pool, store, providers async def _refresh_once(self, user_id: str, provider: str) -> StoredCredential: async with self.pool.acquire() as conn: async with conn.transaction(): held = await conn.fetchval( "SELECT pg_try_advisory_xact_lock($1, $2)", LOCK_NAMESPACE, lock_key(user_id, provider), ) if not held: # Another worker is refreshing. Wait for its write and # re-read. Never queue a second refresh behind the first. return await self._await_fresh(user_id, provider) cred = await self.store.get(user_id, provider, conn=conn) if not needs_refresh(cred): return cred # somebody already won while we waited try: fresh = await self.providers[provider].refresh(cred) except InvalidGrant: # Terminal. The grant is gone at the provider. Retrying # cannot help and hides the real state from the user. await self.store.mark_disconnected(user_id, provider, conn=conn) raise ReconnectRequired(provider) await self.store.put(fresh, conn=conn) return fresh
pg_try_advisory_xact_lock returns immediately instead of blocking, and the lock releases when the transaction ends, including on a crash. That last property is what makes it safe, where a Redis lock with a TTL forces you to reason about what happens when the TTL expires mid refresh.
There is a real trade-off in that code: it holds a pooled connection open across an outbound HTTP call, which is normally a thing to avoid. It is worth it here because the alternative revokes a real user's session, but bound it with a short timeout on the refresh request, a lock_timeout on the transaction, and a metric on refresh duration.
invalid_grant is terminal. The grant no longer exists at the provider and retrying cannot change that, so mark the connection dead and surface a reconnect prompt. A retry loop on invalid_grant produces a background job that fails forever and an alert nobody reads.
Revocation has to reach the work already in flight
Deleting the row is not revocation. The grant still exists on the provider's side, and the refresh token still works for anyone who has a copy.
async def disconnect(user_id: str, provider: str) -> None: cred = await store.get(user_id, provider) if cred is None: return await providers[provider].revoke(cred) # provider's revocation endpoint first await store.delete(user_id, provider) # then local state await jobs.cancel_for(user_id, provider) # then anything queued or scheduled
The third line is the one teams skip. A scheduled digest, a paused approval, a long-running research run: each holds a user_id and will resolve a credential later, which is exactly why jobs should carry an identity rather than a token. If a queue payload contains a serialized access token, revocation cannot reach it, your queue retention becomes your credential lifetime, and every dead letter queue becomes a secret store.
Providers also expire things without telling you. Google invalidates a refresh token when the user revokes access, when it has gone six months unused, and when the account exceeds one hundred live refresh tokens for a single OAuth client ID, at which point the oldest is silently dropped.
Bind three identities per request, not one
Most systems track one identity in a tool call. There are three, and conflating them is how a background job ends up acting as an admin.
The end user is who authorized the grant. This is the identity the provider will attribute the action to.
The agent is what is running: which agent, which version, which run. You need it for attribution and for policy, because "the nightly summarizer may read mail" and "the interactive assistant may send mail" are different rules about the same user's token.
The service is what your own infrastructure calls as when a tool touches shared resources, which is a different credential entirely and must never be substituted for a missing user credential.
Carry them together, immutably, from the authenticated request that started the run:
Two rules make it work. Resolve credentials from ctx, never from module state or a thread local you set earlier. And re-resolve at the moment of use rather than at the start of a plan. An agent that decides at 09:00 to send an email at 09:40 must resolve the token at 09:40, because between those two moments Ana may have revoked, the scopes may have narrowed, and the token has certainly expired.
Scope escalation is a flow, not an exception
Ana connected with read-only access. The agent now wants to archive a thread. The wrong answer is a 403 that reaches the model, which will apologize, guess, and try a different tool.
If your tools sit behind an MCP server, the protocol already specifies this exchange. The server answers 403 with a WWW-Authenticate challenge naming the scopes the operation needs:
Two details in the spec matter more than they look. Servers should include every scope the operation needs in a single challenge, because returning one missing scope at a time drags the user through a consent screen per permission. And on the client side, the step-up request must ask for the union of the scopes already granted and the scopes being challenged, otherwise re-authorizing to gain one permission silently drops another.
Inside the agent, model it as a typed, recoverable outcome:
class ScopeRequired(Exception): def __init__(self, provider: str, missing: frozenset[str], granted: frozenset[str]): self.provider, self.missing, self.granted = provider, missing, granted# At the tool boundarytry: return await tool(ctx, **args)except ScopeRequired as exc: return ToolResult( is_error=True, content=( f"This action needs additional permission for {exc.provider}: " f"{', '.join(sorted(exc.missing))}. Tell the user that a one time " "approval is required, then stop and wait." ), meta={ "authorize_url": step_up_url(ctx, exc.provider, exc.granted | exc.missing), }, )
The message goes to the model so it can explain the pause in the user's own words. The URL goes in metadata for your application to render as a real button, and it stays out of the model's text on purpose. A model relaying a clickable authorization link is a phishing surface, and it only takes one injected instruction to swap the destination.
A typed secret is the control, redaction is the backstop
Regex redaction on the way into your logger is worth having, and it is a backstop, not a control. It only fires on the shapes you thought of, and the leak that hurts is the one that does not look like a bearer token.
The actual control is a type that refuses to render itself.
class Secret: """A string that will not print, format or serialize itself.""" __slots__ = ("_value", "label") def __init__(self, value: str, label: str) -> None: self._value = value self.label = label # "google:access_token", never the value def reveal(self) -> str: return self._value def __repr__(self) -> str: return f"<Secret {self.label}>" __str__ = __repr__ def __format__(self, spec: str) -> str: return repr(self) def __iter__(self): raise TypeError("Secret is not iterable")
Now f"Bearer {token}" renders Bearer <Secret google:access_token> and the request fails immediately in development instead of succeeding in production with a token in a trace. json.dumps on a structure containing a Secret raises TypeError, which is the failure you want: a red test rather than a credential in your observability vendor. Pydantic's SecretStr does the same job if you are already in a Pydantic codebase, though you should check exactly how your serializer treats it before trusting it in a response model.
The real payoff is that .reveal() is greppable. One CI check that fails if .reveal() appears anywhere outside your HTTP transport module turns a diffuse security property into a lint rule. It catches the well meaning debug line, the new engineer's print, and the exception handler that helpfully includes the request kwargs.
Keep the backstop anyway, on the way out to every sink:
Run it on anything you hand to the model, the logger or the tracer, and be specific about one case: provider error bodies. Several APIs echo request headers back in their error payloads, and returning that body to the model so it can self-correct is how a token lands in conversation history that then gets written to long-term memory with completely different access controls.
Two credential planes when tools live behind MCP
If your tools run inside your own process, everything above is the whole story. If they sit behind an MCP server over HTTP, there are two credential planes and they must never touch.
Inbound is the token the MCP client presents. The server acts as an OAuth 2.1 resource server and must validate that the token was issued specifically for it as the audience, must only accept tokens valid for its own resources, and must not accept or transit any others. Clients must include the RFC 8707 resource parameter naming the server's canonical URI on both the authorization request and the token request, and servers must implement RFC 9728 protected resource metadata so clients can discover the authorization server.
Outbound is the per-user Gmail token from the store above, resolved from the identity the inbound token established.
Between them, the spec draws a hard line: if the MCP server calls an upstream API, it acts as an OAuth client to that API, the upstream token is a separate token from the upstream authorization server, and the server must not pass through the token it received from the MCP client.
Passthrough is tempting because it makes the diagram simpler, and it destroys every property in this post. The token's audience no longer matches its consumer, so audience validation stops meaning anything. Its lifetime belongs to a different authorization server, so your refresh logic is guessing. Its scopes were minted for the wrong resource. And you have built a confused deputy, because any service that can obtain a token from the same shared authorization server can now drive your server's upstream privileges.
The inbound plane tells you who is calling. Acting as that person against Gmail is still entirely your job, and it runs through the resolver.
The AI agent credential management checklist
No credential is read at import time, and no provider client is constructed at module scope
Secrets are wrapped in a type that will not print, format or serialize, and .reveal() is confined to the transport layer by a CI check
Nothing resembling a token ever enters a prompt, a system message, tool output, conversation history or agent memory
Credentials are stored with envelope encryption, a fresh data key per row, and an encryption context that names the user, provider and key version
The encryption context is also passed as AES-GCM additional authenticated data, so a copied row fails at two layers
issued_at, expires_at, scopes and key_version stay in plaintext columns so you can query, audit and rewrap without decrypting
Key rotation is a background rewrap job driven by key_version, not a table migration
Refresh fires at a fraction of the token lifetime with jitter, not on a 401
Refresh is single flight per user and provider across the whole fleet, using a lock that releases on crash
invalid_grant is terminal: mark the connection dead, prompt a reconnect, never retry
Queue payloads and checkpoints carry a user ID, never a token
Disconnect calls the provider's revocation endpoint, then deletes local state, then cancels queued work for that user and provider
Every tool call carries an immutable request context with user, tenant, agent and run, and credentials resolve from it at the moment of use
Missing scopes raise a typed error that becomes a consent flow, requesting the union of granted and challenged scopes, with the authorization URL in metadata rather than in model text
Provider error bodies are redacted before they reach the model, the logger or the tracer
If tools sit behind an MCP server, inbound token audiences are validated and no inbound token is ever forwarded upstream
You have logged in as two users, run the same tool concurrently, and confirmed in the provider's own audit log that two distinct identities made the calls
Where this leaves you
Credential work feels like a security chore and behaves like an infrastructure design. The security half, encryption at rest and secrets out of source control, is the part most teams finish in a week. The half that actually breaks agents is operational: a refresh that races, a token that outlives a revocation because it was serialized into a queue, a secret that rode along in a prompt into a trace store with different access rules.
The design that survives all of that is small. One store keyed by user and provider. One resolver that checks scope, refreshes under a lock, and returns a type that refuses to print. One request context that travels with the run. The model gets intent and never gets a secret, so a prompt injection ends with an attacker reading instructions instead of reading a token.
Next in the series: what happens when those users are grouped into tenants, and isolation stops being one row per user and starts being a property of your whole platform.
