Calibrating Reliance on Automated Advice
Summary: Tatasciore & Loft (2025) test two interventions for improving human reliance on automated AI advice: transparency (showing the reasoning) and trust calibration feedback (telling users when their trust is miscalibrated). Transparency works; the feedback intervention does not — a finding explained by the cognitive miser hypothesis.
Sources: Academia/Calibrating Reliance on Automated Advice Transparency and Trust Calibration Feedback.pdf
Last updated: 2026-05-06
The problem
Automated decision aids (AI systems, autopilots, recommendation engines) fail in two opposite directions:
- Misuse (over-reliance): humans accept incorrect automated advice — automation bias
- Disuse (under-reliance): humans reject correct automated advice
Both errors matter. The question is how to calibrate human trust to match the automation’s actual reliability. Two design interventions were tested. (source: Academia/Calibrating Reliance on Automated Advice Transparency and Trust Calibration Feedback.pdf)
The two interventions
Automation transparency — providing real-time information about why the system is giving the advice it gives. Operationalized as three levels (the SAT model: Chen et al. 2014):
- Purpose and intentions
- Rationale behind advice
- Projected outcomes if followed
Trust calibration feedback — explicit, trial-by-trial feedback about whether the human’s trust was appropriate. When trust was miscalibrated (i.e., the human accepted incorrect advice), the feedback read: “Your trust was inappropriate. Take your time and carefully check the display information.”
Findings
| Outcome | Higher transparency | Trust calibration feedback |
|---|---|---|
| Automation use accuracy | ✓ Improved | ✗ No benefit |
| Decision time | ✓ Improved (faster) | ✗ No benefit |
| Perceived workload | ✓ Reduced | ✗ No benefit |
| Perceived trust | ✓ Higher (calibrated) | ✗ No benefit |
| Usability | ✓ Higher | ✗ No benefit |
The predicted interaction effect (feedback amplifying transparency) also did not materialize.
Why metacognitive prompting fails: the cognitive miser
The failure of trust calibration feedback is explained by the cognitive miser hypothesis: humans prefer minimal cognitive effort in decision-making. When automation provides advice, it functions as a System 1 heuristic — it reduces the need to process information independently. Telling people to “slow down and check” does not override this tendency because:
- The instruction arrives after the miscalibrated response (post-error slowing occurs but doesn’t generalize)
- The mechanism of automation bias is not conscious preference but structural reliance on the AI as a cognitive shortcut
- A verbal prompt cannot reach the heuristic layer it is trying to correct
This is a direct instantiation of dual-process-cognition: the bias operates at System 1 (automatic, effort-minimizing); the feedback targets System 2 (deliberate, effortful); System 2 is not reliably activated by instruction alone.
Why transparency works
Transparency operates differently from feedback: it changes what System 1 has access to, rather than asking System 2 to override System 1. When the rationale for automated advice is visible, users can more easily notice when it contradicts other information — without effortful deliberation. Transparency restructures the decision environment; feedback merely requests a different response within the same environment.
This maps onto Meadows’s insight in systems-thinking: “Adding or restoring information can be a powerful intervention.” Transparency adds information to the decision context. Trust calibration feedback does not.
A note on automation bias
Even with high transparency, some studies (including prior work by the same authors) found a bias toward agreeing with highly transparent advice — as if providing a rationale amplifies perceived authority. Making the AI’s reasoning visible can inadvertently make it seem more credible, increasing misuse even as it reduces disuse. This asymmetry remains an open design problem.
Connections
trust — Sztompka defines trust as a bet about others’ future contingent actions. Automation trust is a bet about the system’s accuracy. This paper shows the bet can be miscalibrated in both directions, and that simply labeling the miscalibration does not fix it.
dual-process-cognition — automation bias is the cognitive miser hypothesis in action: System 1 uses AI advice as a heuristic replacement for vigilant information-seeking. Transparency succeeds because it changes the information environment; feedback fails because it asks System 2 to override a System 1 tendency without changing the conditions that produce it.
llm-and-mind — as LLMs are increasingly deployed as automated advisors, these findings matter directly. Users will over-rely on fluent, confident LLM outputs. Transparency (showing reasoning, uncertainty, knowledge limits) is likely to help. Metacognitive warnings probably will not.