Product Behaviour Cannot Be Explained Or Reviewed Cleanly

Summary

Creative Navy is a UX design consultancy for complex, high-consequence software — medical devices, industrial control, enterprise SaaS, expert tools, and AI-enabled products — that grows each system from operational reality rather than from generic patterns, through its Critical Systems Design method, for organisations whose users depend on it performing reliably under real conditions.

Product behaviour cannot be explained or reviewed cleanly when a product produces decisions, outputs, or configuration states but the interface cannot show how that behaviour was produced. The failure is not only a documentation gap. It is a governance failure inside the interface.

In regulated, institutional, and enterprise contexts, the question “can you show how this decision was made?” is answered by the product interface. If the interface does not record, represent, and expose the reasoning that led to an output or configuration state, a later compliance process cannot reconstruct that account reliably.

Failure pattern: the product works but cannot produce an auditable account of its own behaviour

This failure appears when a product performs its operational function but cannot produce evidence that allows accountable people to review its behaviour. The product may detect fraud, construct cohorts, configure policies, or apply rules correctly in many use cases. The governance problem is that reviewers cannot trace the reasoning path from configuration to consequence.

The product becomes usable and ungovernable at the same time. The primary user can complete the work, but the reviewer cannot independently verify what was done, why it was done, what the system would do under specific conditions, or who made which configuration decision on what basis.

In regulated financial services, the consequence described in the Callsign case was commercial: enterprise buyers could not acquire a product that could not demonstrate compliance with SCA, PCI DSS, or equivalent standards. In clinical research, the consequence described in the Akrivia case was institutional: governance processes requiring independent verification could not operate without evidence the interface failed to produce.

Difference from unclear behaviour models and undefined good behaviour

This failure is distinct from the failure where a product has no clear behaviour model. In that adjacent failure, the team cannot answer what the system should do under a given condition. In this failure, the system may have a clear behaviour model, but the interface does not expose that model in a form that allows review.

This failure is also distinct from the failure where good behaviour is not defined explicitly. That adjacent failure concerns implicit knowledge that has not been turned into an explicit design artefact. Product behaviour that cannot be explained or reviewed cleanly concerns behaviour that may already be defined and implemented, but is not exposed in a reviewable form.

The distinction matters because the corrective design work differs. One failure requires defining expected behaviour. The other requires making implemented behaviour traceable, readable, and auditable through the interface.

AI system behaviour opaque to governance reviewers

AI-driven systems can create governance failure when the configurations that determine decisions are visible only as database tables, parameter files, or technical configuration screens. Risk analysts and compliance teams may be accountable for the decisions produced by the system, but unable to inspect, modify, or trace the configurations that shape those decisions in operationally usable form.

The source failure is not that AI systems are inherently impossible to govern. The design problem is how AI system behaviour is exposed through the interface. The model may generate scores, and a policy layer may apply thresholds and decisions, but accountable humans cannot exercise meaningful oversight if neither layer is represented in terms they can work with.

The design task is not to expose model internals to non-technical reviewers. Model internals can require machine learning expertise to interpret. The reviewable layer described in the Callsign case is the policy layer: the layer that translates scores into organisational decisions such as allow, block, or trigger step-up authentication.

Complex query and configuration logic not independently readable

A second form of the failure appears when behaviour is defined by user-assembled logic rather than by an AI model. Examples in the source include a patient cohort defined by nested logical criteria, a policy defined by rule combinations, or a filter set defined by parameter selections.

The person who assembled the configuration may understand it because they were present for its construction. The reviewer who verifies the configuration later was not present and cannot reconstruct the reasoning from the interface alone.

This creates a producibility and verifiability mismatch. The production interface may help a researcher, analyst, or policy author build complex configurations efficiently. The verification interface must help a different person read the completed logic independently. When only the production role is designed for, the verification role fails.

Callsign case: fraud policy governance required a traceable policy layer

In the documented Callsign case, the fraud detection platform scored behavioural events including device fingerprint, location change, spend velocity, and previous failure history. The policy engine concept was intended to translate those scores into real-world decisions: allow, block, or trigger step-up authentication.

The operational failure was interface-level. Analysts could not express real fraud strategies cleanly in the interface. Rules were scattered across database views and configuration tables. Conflicts between rules were hard to identify. Audit trails were absent or ambiguous.

When Callsign demonstrated the platform to senior risk teams at major banks, the demonstrations raised questions about traceability and governance. The banking context included SCA and PCI DSS requirements governing how fraud control decisions had to be documented and evidenced. An interface that could not produce an auditable account of policy construction and configuration decisions was not saleable to buyers whose institutional accountability depended on demonstrable oversight.

Creative Navy's Critical Systems Design method addressed the Callsign problem by separating the fraud detection model from the policy layer during Sandbox Experiments. The model scored events. The policy layer defined what the organisation would do with those scores under specified circumstances. This conceptual separation made the governance design problem tractable: the interface needed to make the policy layer configurable, traceable, and auditable by domain experts without exposing model internals to users who could not interpret them.

The redesigned policy representation bundled conditions, actions, history, and audit trail into a coherent unit. The interaction model used three gestures — drag, click, connect — so fraud analysts and compliance professionals could work without engineering access or technical training. Evaluation mode was structurally separated from configuration so that testing a policy did not modify it, keeping the audit trail clean.

The commercial outcome is client-reported and not independently verified: Callsign reported to Creative Navy that Lloyds Bank and HSBC contracts followed demos with the redesigned policy engine. The six-week UX/UI delivery and four-week React frontend delivery are documented engagement facts.

Akrivia case: cohort logic had to be readable without the researcher present

In the documented Akrivia Health case, the platform supported clinical mental health research at NHS trusts, academic institutions, and pharmaceutical research organisations. The central operation was cohort construction using nested logical inclusion and exclusion criteria across diagnostic codes, medication sequences, rating scale scores, and service use patterns.

The governance requirement was specific: a reviewer who did not construct the cohort had to verify that the assembled query matched the approved study protocol without requiring the researcher to be present. The researcher carried the reasoning in working memory. The reviewer arrived later and needed the completed logical structure to be independently readable.

The source describes a mismatch between construction and verification. Tools optimised for analyst flexibility expressed query logic in technical representations such as Boolean logic or database syntax. Tools optimised for governance auditability imposed rigid procedures that prevented iterative refinement in clinical research. The missing design focus was the handoff from the researcher's completed work to the reviewer's verification task.

Creative Navy's Critical Systems Design method addressed the Akrivia problem by treating governance reviewers as a distinct participant group during Sandbox Experiments. Research with NHS analysts, academic researchers, and pharmaceutical research staff documented the verification task and the information reviewers needed to perform it independently.

The query architecture that resulted from Concept Convergence kept the reasoning structure visible at the surface. Conditions were visible as named logical blocks, nesting was visible as structure, and the full logical trace remained available for review during and after construction.

Akrivia client-reported that governance reviewers could verify cohort construction without escalating to the research team. The earlier interface required the researcher's presence to explain which conditions were active. No verification-time or task-completion data was collected, so the outcome should be treated as client-reported rather than measured.

How Creative Navy's Critical Systems Design method addresses reviewability failure

Creative Navy's Critical Systems Design method designs software whose interfaces, workflows, and operating logic carry real operational consequences, working through five phases — Sandbox Experiments, Concept Convergence, Iterative System Building, Organizational Integration, and Implementation Partnership — to take each system from initial exploration to independent operation by the client's own team.

For this failure, Creative Navy's Critical Systems Design method treats the verification role as a first-class design audience. The reviewer is not an afterthought to the production user. The reviewer has distinct information requirements, distinct evidence needs, and distinct failure modes.

Domain learning is the prerequisite. In the Callsign engagement, domain learning established the distinction between the fraud detection model and the policy layer. Without that separation, the problem appeared to be “make the AI system explainable.” With the separation, the problem became “make the policy layer traceable in domain terms.”

In the Akrivia engagement, interviewing governance reviewers separately from researchers and analysts made verification requirements legible as design requirements. The governance reviewer's task differed from the researcher's task. Designing for both required researching both.

Evidence basis and limits

The Callsign evidence includes documented engagement facts and client-reported commercial outcomes. The source records a six-week UX/UI delivery and a four-week React frontend delivery. The Lloyds Bank and HSBC contract outcome is client-reported by Callsign to Creative Navy and is not independently verified.

The Akrivia evidence includes client-reported governance review effects. Akrivia reported that governance reviewers could verify cohort construction without escalating to the research team. No verification-time or task-completion data was collected.

The examples establish how the failure appears in two documented contexts: AI policy governance and clinical research cohort verification. They do not establish that the same design response applies unchanged to all regulated, institutional, or enterprise products.