Collaborative Research: Leveraging Crowd-AI Teams for Scalable Novelty Ratings of Heterogeneous Design Representations

这项研究项目将通过开发一个框架，利用人和机器的专业知识来对设计文物的新颖性和质量进行评级，从而推动工程设计领域的发展。该项目将解决该领域的一个重大挑战，即如何准确评估在设计研究中产生的大量复杂、不同种类的想法。该项目将有助于理解如何有效地利用人力和计算专业知识来评估设计人工制品，从而改进产品设计。在专注于工程设计用例的同时，该项目将通过提供证据来帮助机器学习(ML)研究，这些证据表明在评估异类数据集时，应该向人类学习什么、何时以及如何学习。该项目将建立一个网站，提供开放源码数据和为职业早期专业人员举办讲习班，以加强这项工作对整个工程设计界的更广泛影响。该研究团队将与宾夕法尼亚州立大学的女性科学与工程研究计划(WISER)和多元文化工程计划(MEP)合作，鼓励代表不足的群体参与工程。该项目调查三种不同的方法，即(1)人类专家设计师和众包人类评分员；(2)基于ML的纯方法；以及(3)专家设计师辅助的ML方法。这些方法将用于更好地了解设计表现对基于人群的设计评级的可靠性的影响，建立和验证用于设计评估的多模式表示学习方法，并创建用于部署可扩展设计评估的群组-ML协作的方法。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

This research project will advance the field of engineering design by developing a framework for rating the novelty and quality of design artifacts using a combination of human and machine expertise. The project will address a significant challenge in the field, which is how to accurately evaluate the vast amounts of complex, heterogeneous ideas generated in design studies. The project will contribute to understanding how to effectively leverage human and computational expertise in evaluating design artifacts, leading to improved product design. While focused on an engineering design use case, the project will contribute to machine learning (ML) research by providing evidence on what, when, and how to learn from humans when evaluating heterogeneous datasets. The project will establish a website with open-source data and workshops for early-career professionals to enhance the broader impacts of the work on the engineering design community at large. The research team will partner with the Women in Science and Engineering Research (WISER) program and the Multicultural Engineering Program (MEP) at Penn State to encourage participation of underrepresented groups in engineering.The project investigates three distinct means of evaluating heterogeneous design artifacts (CAD drawings, text, sketches, prototypes) against established design metrics at scale, namely (1) human expert designers and crowd-sourced human raters; (2) pure ML-based methods; and (3) expert designer-assisted ML methods. These methods will be used to better understand the impact of design representation on the reliability of crowd-based design ratings, establish and validate multi-modal representation learning methods for design evaluation, and create methods for deploying crowd-ML collaborations for scalable design evaluation.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.