Examining User Heterogeneity in Digital Experiments
Examining User Heterogeneity in Digital Experiments
复制标题
DOI：
10.1145/3578931
复制
发表时间：
2023-01
影响因子：
5.6
通讯作者：
S. Somanchi;A. Abbasi;Ken Kelley;David G. Dobolyi;T. Yuan
中科院分区：
文献类型：
--
作者：
S. Somanchi;A. Abbasi;Ken Kelley;David G. Dobolyi;T. Yuan
数字实验通常被用来测试治疗相对于现状控制设置的价值--例如，网站的新搜索相关性算法或移动应用的新结果布局。随着数字实验在组织和各种研究领域变得越来越普遍，它们的增长给实验平台带来了一系列新的挑战。一个挑战是，实验往往侧重于平均治疗效果(ATE)，而没有明确考虑主要亚组之间的差异：异质治疗效果(HTE)。这尤其有问题，因为随着更明显的好处已经实现，许多组织的ATE已经减少。然而，关于用户HTE的普遍性以及如何最好地检测它们的问题比比皆是。我们提出了一个在数字实验中检测和分析用户HTE的框架。我们的框架结合了一系列用户特征和双重机器学习。对17.6亿会话的27个真实世界实验和模拟数据的分析表明，相对于现有技术，我们的检测方法是有效的。我们还发现，在10%到20%的真实世界实验中，交易、人口统计、参与度、满意度和生命周期特征在统计上表现出显著的HTE，这突显了在分析实验结果时考虑用户异质性的重要性；否则，个性化功能和体验就无法发生，从而降低了有效性。就实验和用户会话的数量而言，我们不知道有任何研究以这种规模检查用户HTE。我们的发现对信息检索、用户建模、平台和数字体验环境具有重要意义，在这些环境中，在线实验经常被用来评估设计人工制品的有效性。
Digital experiments are routinely used to test the value of a treatment relative to a status-quo control setting—for instance, a new search relevance algorithm for a website or a new results layout for a mobile app. As digital experiments have become increasingly pervasive in organizations and a wide variety of research areas, their growth has prompted a new set of challenges for experimentation platforms. One challenge is that experiments often focus on the average treatment effect (ATE) without explicitly considering differences across major sub-groups: heterogeneous treatment effect (HTE). This is especially problematic, because ATEs have decreased in many organizations as the more obvious benefits have already been realized. However, questions abound regarding the pervasiveness of user HTEs and how best to detect them. We propose a framework for detecting and analyzing user HTEs in digital experiments. Our framework combines an array of user characteristics with double machine learning. Analysis of 27 real-world experiments spanning 1.76 billion sessions and simulated data demonstrates the effectiveness of our detection method relative to existing techniques. We also find that transaction, demographic, engagement, satisfaction, and lifecycle characteristics exhibit statistically significant HTEs in 10% to 20% of our real-world experiments, underscoring the importance of considering user heterogeneity when analyzing experiment results; otherwise, personalized features and experiences cannot happen, thus reducing effectiveness. In terms of the number of experiments and user sessions, we are not aware of any study that has examined user HTEs at this scale. Our findings have important implications for information retrieval, user modeling, platforms, and digital experience contexts, in which online experiments are often used to evaluate the effectiveness of design artifacts.