In some scenarios, a decision can be shared between the human and the AI agent. For instance, in the design of a machine learning algorithm, the human user wants to be in charge of the choice of the high-level interpretable features of the model, delegating the low-level design choices (e.g. the hyperparameters of the optimizer) to the AI agent. When the human chooses first, it is easy for the AI agent to make an optimal decision; however, in some scenarios, the AI agent has to choose first, in which case it has to anticipate the reaction of the user.

We investigated the case where the two agents are aiming to maximize a black-box function. We propose to characterize users along two dimensions: their inclination either to explore unfamiliar regions of the domain or to remain within familiar ones, and the degree of conservatism they exhibit when updating their beliefs. We demonstrated the importance of incorporating such a user model in the AI agent’s decision-making. Following a similar idea, we also explored a setting where the decision is divided between a retrieval and an adaptation process. In that case, the goal of the AI agent is to retrieve informative past solved cases, from a potentially large memory, in order to help the user make a choice for their new problem. I show that it is then essential to understand how the user will interpret the proposed cases, in order to more efficiently select them.