Humans-in-the-Contestability-Loop: Designing for Social Counselors to Understand and Challenge Algorithmic Decisions

Making algorithmic decisions explainable and contestable is essential to ensure the responsible use of high-risk AI systems. Yet the human-centered design and implementation of explainability and contestability remain underexplored in empirical research. In this paper, we follow a participatory design approach in three stages to develop an interface that supports social counselors in understanding and contesting a welfare fraud detection system. We analyze counselors' information needs, derive functional and usability requirements, and design and evaluate both low-fidelity and high-fidelity interfaces that implement these requirements. Our findings show that contesting administrative decisions is a complex socio-technical process involving social counselors, clients, and social agencies, requiring carefully tuned explanation and contestation elements. Social counselors prioritized procedural information and tools for interactive exploration of model predictions, while emphasizing that explanations must be concise, easy to understand, and sensitive to the emotional impact of information on clients. We find that human intervention remains crucial in contestation processes, that explanations can support counselors by surfacing contestation reasons and facilitating client-side communication, and that conversational explanations enable flexible inquiry. Our insights contribute to trustworthy AI implementations by outlining the potential and challenges of a human-centered explanation and contestation interface for individuals affected by high-risk AI systems.