Explainability has become one of those words that gets nodded at in a vendor demo and then quietly abandoned the moment someone with a clipboard asks a real question. In the demo, explainability means a nice paragraph the model generates about why it made a recommendation. In an audit, explainability means something much narrower and much less forgiving: can you reconstruct, after the fact, exactly what inputs, permissions, and context produced this specific output, for this specific user, at this specific time.
Those are different problems wearing the same word. A demo-grade explanation is persuasive prose generated alongside the answer, and it's often generated by the same model that produced the answer in the first place, which means it's explaining itself to itself. An audit-grade explanation has to survive someone who doesn't trust the system and wants a trail independent of the system's own narration. Picture a team that ships an AI agent capable of approving refunds under a dollar threshold, and six months later a regulator asks why one particular refund went through. 'The model determined it was eligible' is not an answer. Which policy document was in context, which prior decisions it was conditioned on, what identity and scope it was operating under at that moment — that's the answer, and it has to exist as a record, not as a story the model tells on request.
The access-control angle
This is where explainability stops being a model-quality question and becomes an access-control question, which is the frame we keep coming back to across our practice whether the actor is a person, a service account, or an AI agent. An auditor doesn't ultimately care how eloquently a system explains itself; they care what it was allowed to touch, under what identity, and whether that grant was logged and bounded. If an AI agent's permissions are a vague blob of 'has access to the customer database' rather than a scoped, time-stamped, revocable grant, no amount of generated explanation closes that gap, because the explanation is describing a decision made inside a black box of standing privilege that nobody can reconstruct the boundaries of.
A common failure mode is building the explainability layer and the access layer as two separate projects, owned by two separate teams, that never reconcile. The AI team ships a feature that produces human-readable justifications. The security team manages identity and entitlements for the service accounts the model runs under. Neither side's artifacts line up with the other's, so when an audit asks for both at once, the organization produces a plausible-sounding narrative and a permissions log that don't actually corroborate each other. That's not a documentation gap, it's a design gap: the explanation and the access record were never meant to be read together.
What audit-grade explainability actually requires is less glamorous than a model that writes good prose about itself. It requires that every AI-driven action be attributable to a scoped identity, that the inputs and permissions in effect at decision time be captured as a durable record rather than reconstructed from logs after the fact, and that the explanation a human eventually reads be traceable back to that record rather than regenerated from memory. None of that is an AI capability. It's an access-control discipline applied to a new kind of actor, which happens to be the same discipline that governs what a contractor's laptop or a service account can reach.
So when someone asks whether a system is explainable, the honest first question isn't 'can it tell you why' — it's 'who was it, what could it touch, and can you prove that after the fact.' An AI agent is still just a user with credentials, and the thing that makes its decisions auditable is the same thing that makes any user's actions auditable: a record of identity, scope, and time, kept independently of the actor being asked to explain itself.