Learning Constraints From Human Stop-Feedback in Reinforcement Learning

We investigate an approach for enabling a reinforcement learning
agent to learn about dangerous states or constraints from stop-
feedback preventing the agent from taking any further, potentially
dangerous, actions. Such feedback could be provided by human
supervisors overseeing the RL agent’s behavior while carrying
out some complex tasks. To enable the RL agent to learn from
the supervisor’s feedback, we propose a probabilistic model for
approximating how the supervisor’s feedback could have been
generated and consider a Bayesian approach for inferring dangerous
states. We evaluated our approach using an OpenAI Safety Gym
environment and demonstrated that our agent can effectively infer
the imposed safety constraints. Furthermore, we conducted a user
study to validate our human-inspired feedback model and to obtain
insights into the human provision of stop-feedback.