Human oversight is a design challenge: How to build clinical AI that people actually trust

A radiologist has a few seconds per image in a busy reading session. The AI has flagged a nodule with an "87% confidence" score. What does she do with that number? Accept it? Re-read the whole scan? Ignore it, because the last three flags were false positives?

Regulation has a term for what she is supposed to do: human oversight. The EU AI Act requires it for high-risk AI systems, including many AI-enabled medical devices that need third-party conformity assessment under EU medical-device legislation. Most of the debate focuses on documentation, data governance and conformity assessment. Far less attention goes to the question that decides whether oversight works in practice: can a clinician under time pressure see what the system is doing and step in?

The requirement is legal. Whether it works in practice is an interface question.

Why the new deadline buys no time

The AI Act originally set August 2027 for these obligations. The Digital Omnibus on AI, Regulation (EU) 2026/1744, which entered into force on 27 July 2026, moved the relevant high-risk requirements for AI systems covered by Article 6(1) and Annex I, including qualifying medical-device AI under the MDR and IVDR, to 2 August 2028.

For product teams, the shift changes little. A clinical software product launching in 2028 is being specified, designed and tested now. And oversight cannot be bolted on at the end. If the interface hides uncertainty or makes overriding the system awkward, no technical file will fix that.

The pressure is not only regulatory. Across digital health, AI is moving from pilots into daily clinical work. Ambient documentation now runs at enterprise scale: Mass General Brigham has made it available across its physician workforce, and more than 4,000 providers now use it routinely. The question has shifted from "does the model work?" to "do people trust it enough to use it, and not so much that they stop checking?"

Two oversight risks that design can make worse

• Automation bias. The system is right often enough that users stop questioning it. The confirm button becomes a reflex.

• Under-trust and alert fatigue. The system is wrong or noisy often enough that users discount its output, including the moments when intervention would matter.

Both are well-known human factors challenges that can turn into use-related risks under IEC 62366-1. And both depend on concrete design decisions: how confidence is displayed, how much effort an override costs, how often the system interrupts, and what happens after a user disagrees.

Five design principles for oversight that works

Many of these improvements start with design decisions rather than new technology. Product teams make most of them anyway, often without noticing.

1. Explain what the user can act on

Explainability means different things to different people. A data scientist may want a heatmap. The radiologist from our opening needs something simpler: what the system looked at, what it found, and what would have changed the result. The right depth of explanation follows from the decision she has to make next.

2. Give uncertainty a meaning

On its own, "87%" tells her very little. In many cases, a short explanation is more useful than a raw score. "Low confidence: image quality" points her to what she should check. Whether users read such labels as intended is something to find out in formative evaluation, while the interface can still change.

3. Make overriding as easy as accepting

Picture an interface where accepting the AI takes one click, while overriding it takes four clicks and a mandatory comment. Without anyone deciding it, the interface is voting for the AI.

Balancing that effort is one of the simplest ways to protect real oversight. Override patterns are also worth tracking as one input into post-market monitoring.

4. Interrupt only when it matters

Every alert that turns out to be unimportant spends a little of the trust the system will need later. Teams that rank notifications by urgency and let routine information stay quiet save that trust for the moments that count.

5. Design for a twelve-hour shift

Oversight rarely happens in calm conditions. It happens between interruptions, with gloves on, at shared workstations and during handovers. That is why it pays to observe users in their real environment before the first concept, and to test with representative users in realistic scenarios.

Lessons from automation that came before AI

The trust problem is older than clinical AI. Pharmacy automation, connected devices and clinical software have faced similar human factors challenges for years, and many of the design principles carry over.

Pharmacy automation at BD Rowa. When FLUID Design redesigned the Smart UI for BD Rowa's pharmacy robots, the brief centered on building confidence in automation. The team observed pharmacists at work in Germany, the UK and Italy. The redesign introduced one-tap access to robot functions, step-by-step troubleshooting, and a notification system that ranks issues by urgency. The Smart UI was a nominee for the UX Design Award 2026.

Therapy software at STORZ MEDICAL. The software for the MASTERPULS Icon shockwave therapy devices had grown through incremental add-ons and drifted away from real treatment workflows. FLUID rebuilt it around the actual therapy flow: indication, parameter selection, execution, documentation. The information hierarchy now scales with how critical a task is, so essential data appears where practitioners need it. FLUID also worked on the industrial design of the MASTERPULS ICON device line, helping align the physical device and its digital interface.

Neither product is an AI system. Both illustrate the same design principle: automated systems are easier to work with when users can understand their state, intervene without friction and focus on what matters.

Who owns oversight in your team?

In most digital health teams, oversight sits between disciplines. Data science builds the model, regulatory documents it, engineering implements it. Often, no single discipline owns the quality of the human-AI interaction at the point of decision. A few practical steps close that gap:

• Define oversight scenarios early in the use specification: when must the user check, override or escalate?

• Include AI-related use errors in the use-related risk analysis, such as accepting an incorrect recommendation without the intended verification.

• Run formative studies on how users interpret confidence levels and explanations, before the UI is frozen.

• Validate with the AI behaving realistically, including plausible but incorrect outputs.

Teams that combine clinical field research, interface design and usability testing can close this gap, especially where hardware, software and clinical workflow meet. One example is FLUID Design, a Munich-based design agency for high-stakes products, from medical and lab devices to industrial HMI and AI-powered interfaces.

Ranked No. 3 among German design studios in the Medical/Healthcare category of the iF Design Ranking, the team works on medical and digital health design for connected devices and clinical software.

Conclusion

The AI Act names human oversight as a requirement. Whether it exists in practice is decided on the screen: in a confidence label, an override button, an alert that does or does not fire. Digital health teams that treat these as design decisions, and test them with real users, will build AI that clinicians question when they should.

FAQ

What does human oversight mean for AI medical devices?

Human oversight means that the people assigned to it can understand the relevant capabilities and limitations of a high-risk AI system, interpret its output and, where appropriate, disregard, override or interrupt it. In practice, this depends on how the interface presents results, uncertainty and options to intervene.

Does the EU AI Act apply to AI-enabled medical devices?

In many cases, yes. AI that is a medical device, or a safety component of one, counts as high-risk when the device needs third-party conformity assessment under the MDR or IVDR. Under Regulation (EU) 2026/1744, the relevant high-risk requirements apply from 2 August 2028.

How does IEC 62366 relate to AI features?

IEC 62366-1 defines the usability engineering process for medical devices. AI-related use errors, such as accepting an incorrect recommendation without the intended verification, can be addressed through the use-related risk analysis, formative evaluation and, where safety-relevant, the scenarios selected for summative evaluation.

What is automation bias in clinical software?

Automation bias is the tendency to accept a system's output without enough scrutiny, especially under time pressure. Interface design can help reduce it, for example through meaningful uncertainty information and override flows without unnecessary friction.

When should design work on AI oversight start?

As early as the use specification. Oversight scenarios, confidence displays and override flows need formative testing before the interface is frozen. Retrofitting them late is costly and often incomplete.