Discovering Individual and Social Preferences through Inverse Reinforcement Learning

提供服务和创造产品的组织通常根据问卷和/或与其用户群（例如患者，客户，公民）的其他明确形式的沟通做出决定。提供商和用户之间进行信息交换的目的是揭示用户的“奖励功能”，即用户实际想要从他们的交互中得到什么以及当前产品/服务阵容存在哪些问题。明确的信息交换形式对于组织来说可能是繁琐和昂贵的，并且对用户具有干扰性。此外，响应偏差是基于调查的方法的一个众所周知的问题，特别是在敏感主题方面，受访者可能由于社会或文化问题而不愿意参与。间接提问方法（项目计数和随机回答技术）提供了一些解决回答偏差的实用方法。然而，这些解决方案都是不实用的大规模和真实的时间settings.We假设，理想的组织应该尝试引出其用户群的奖励功能（即用户的首选状态）通过使用从用户活动产生的观察数据。受最近人工智能研究文献的启发，我们提出了一个三方面的计划，旨在通过a）试图通过收集行为数据来推断用户奖励功能，直接解决用户想要什么的问题（例如网站点击、流量行为、电影偏好）; B）创建简短、非侵入性的在线问卷，以消除任何不确定性;以及c）利用用户偏好，以改善服务和产品供应。拟议的研究旨在促进开发可以嵌入人工智能系统的方法，人工智能系统必须通过与人类互动来获取和理解偏好，以适应他们的行为，并允许更自然的体验，通过这项研究，我们有四个关键目标：（a）了解用户偏好，并开发通过数据和行为发现和学习奖励功能的方法;（B）开发交互式和对话式方法，用于引起用户的响应和交互，从而允许使用自动系统的更自然的用户体验;（c）探讨我们的方法的社会局限性（例如，个人奖励在多大程度上不是由个人偏好决定的，而是由社会强制决定的？）及（d）研究可采取哪些步骤，透过（a）及（B）项所述方法，吸引顾客选择，使提供新服务及产品的程序完全自动化。该奖学金提供了一个独特的机会，将人工智能技术和社会科学结合在一起，以解决一系列企业和组织在与客户和客户打交道时面临的问题，并试图引发通过行为和互动的偏好和需求。我们将与本项目的行业合作伙伴英国电信（BT）和埃塞克斯郡理事会（ECC）密切合作，调查在各自的背景下引发和理解偏好的问题和挑战，以告知和塑造工作方案。

Organisations that provide services and create products often base their decisions on questionnaires and/or other explicit forms of communication with their user base (e.g. patients, customers, citizens). The aim of this information exchange between providers and users is to uncover the users' "reward function", i.e. what users actually want from their interactions and what issues exist with the current product/service line-up. Explicit forms of information exchange can be cumbersome and expensive to design for organisations and are intrusive to the user. Furthermore, response bias is a well-known problem for survey based methods, particularly around sensitive topics, where respondents maybe unwilling to engage due to social or cultural concerns. Some practical solutions to response bias are provided by indirect questioning methods (item count and randomized response techniques). However, none of these solutions are practical for large scale and real time settings.We postulate that ideally an organisation should try to elicit the reward function of its user base (i.e. what states are preferred by users) by using observational data generated from user activity. Inspired by recent literature in AI research, we propose a three-facet programme that aims to directly attack the problem of what users want by a) trying to infer the user reward function through the collection of behavioural data (e.g. website clicks, traffic behaviour, movie preferences); b) creating short, non-intrusive online questionnaires that will remove any uncertainties; and c) exploiting user preferences in order to improve service and product provision.The proposed research aims to contribute to developing methods that can be embedded in artificial intelligence systems which must elicit and understand preferences by interacting with humans in order to adapt their behaviour and allow for a more natural experience and interaction.Through this research we have four key objectives: (a) understand user preferences and develop methods to uncover and learn the reward function through data and behaviours; (b) develop interactive and conversational methods for eliciting responses and interactions from users that allow for a more natural user experience with automatic systems; (c) explore the social limitations of our approach (for instance, to what extend are personal rewards not dictated by individual preferences, but rather by social coercion?); and (d) investigate what steps can be taken to fully automate the procedure of provisioning new services and products through eliciting preferences via the methods developed under (a) and (b).This Fellowship provides a unique opportunity to bring together artificial intelligence techniques and social science to tackle problems that are faced by a range of businesses and organisations in dealing with clients and customers and attempting to elicit preferences and needs through behaviours and interactions. We will be working closely with our industry partners in this project, British Telecom (BT) and the Essex County Council (ECC), to investigate the issues and challenges of eliciting and understanding preferences as being faced in their own contexts to inform and shape the programme of work.