Keywords
Mental Model Misalignment
Abstract
The widespread assumption that artificial intelligence improves decision-making by augmenting human judgment has obscured a troubling countertrend: the integration of AI into organizational decisions can systematically degrade the quality of those decisions, even when AI outputs are accurate and explanations are technically provided. This degradation occurs not because AI systems are unreliable, but because of a constellation of cognitive and organizational failures that emerge when human decision-makers interact with AI systems whose reasoning they cannot accurately model. This paper develops a comprehensive analysis of cognitive explainability failure in AI-assisted organizational decision-making, arguing that the fundamental challenge is not making AI explanations more technically accurate, but preventing AI assistance from undermining the cognitive foundations of good human judgment. Drawing on three foundational references and twelve supplementary citations spanning metacognitive science, human-AI collaboration, cognitive load theory, and organizational behavior, this study identifies four interconnected mechanisms through which AI assistance degrades decision quality: mental model misalignment, cognitive overload and explanation fatigue, automation bias and confidence contamination, and metacognitive outsourcing failure. The analysis reveals that the MIRROR benchmark's findings on systematic AI miscalibration, the Verification Tax's structural constraints on AI auditing, and the practical demand for explainability in human resource analytics collectively illuminate a phenomenon that has received insufficient scholarly attention: the conditions under which AI assistance erodes rather than enhances human decision capacity. The paper concludes with a research agenda for developing AI systems that support rather than supplant human epistemic autonomy.
References
1. Wang, J. Z. (2026). MIRROR: A Hierarchical Benchmark for Metacognitive Calibration in Large Language Models. arXiv preprint arXiv:2604.19809.
2. Lee, M. H., et al. (2025). Metacognitive sensitivity: The key to calibrating trust and optimal decision making with AI. PMC, National Institutes of Health.
3. Bei, J., Liu, Z., Huang, J., Wang, X., & Yang, P. (2025). Strategic Human Resource Analytics with Explainable Artificial Intelligence. In Proceedings of the 2025 6th International Conference on Computer Science and Management Technology.
4. Parasuraman, R., & Manzey, D. H. (2024). Automation bias in AI-decision support: Results from an empirical study. PMC, National Institutes of Health.
5. Buçinca, Z., et al. (2025). Between transparency and trust: Identifying key factors in AI system perception. ACM Transactions on Interactive Intelligent Systems.
6. Schelble, S. M., et al. (2025). Enhancing intuitive decision-making and reliance through human-AI collaboration: A review. Journal of AI, 12(4), 135.
7. Cheng, L., et al. (2025). Exploring the impact of explainable AI and cognitive capabilities on users' decisions. arXiv preprint arXiv:2505.01192.
8. Liao, Q. V., et al. (2025). Human-centered evaluation of explainable AI applications: A systematic review. Frontiers in Artificial Intelligence, 7, 1456486.
9. Swamy, G., et al. (2024). Toward human-centered explainable AI: A survey of literature and practical frameworks. arXiv preprint arXiv:2410.15952.
10. Glikson, E., & Woolley, A. W. (2025). Organizational trust in AI: A review and research agenda. Academy of Management Annals.
11. Edelman. (2025). The AI Trust Imperative: Navigating the Future with Confidence. Edelman Trust Barometer 2025.
12. ACM CHI Conference. (2025). The amplifying effect of explainability in AI-assisted decision-making in groups. CHI 2025 Proceedings.
13. Schulz, C., & Knierim, P. (2024). Cognitive challenges in human–artificial intelligence collaboration: Investigating the path toward productive delegation. Information Systems Research.
14. Bansal, G., et al. (2025). Beyond accuracy: The role of mental models in human-AI collaboration. ACM Conference on Fairness, Accountability, and Transparency.
15. Wang, J. Z. (2026). The Verification Tax: Fundamental Limits of AI Auditing in the Rare-Error Regime. arXiv preprint arXiv:2604.12951.
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
Copyright (c) 2026 Victoria Price (Author)