Collaborative Research: Leveraging Crowd-AI Teams for Scalable Novelty Ratings of Heterogeneous Design Representations

该研究项目将通过开发一个框架来评估设计工件的新颖性和质量，从而推动工程设计领域的发展，该框架将结合人类和机器的专业知识。该项目将解决该领域的一个重大挑战，即如何准确评估设计研究中产生的大量复杂、异构的设计思想。该项目将有助于我们理解如何有效地利用人类和计算机专业知识来评估设计工件，从而改进产品设计。虽然专注于工程设计用例，但该项目将通过提供在评估异构数据集时向人类学习什么、何时以及如何学习的证据，为机器学习（ML）研究做出贡献。该项目将建立一个网站，提供开源数据，并为初入职场的专业人士举办研讨会，以增强工程设计界的广泛影响。该研究团队将与宾夕法尼亚州立大学的女性科学与工程研究（WISER）项目和多元文化工程项目（MEP）合作，鼓励未被充分代表的群体参与工程。该项目研究了三种不同的方法来评估异质设计工件（CAD图纸、文本、草图、原型），以确定规模的设计指标，即(1)人类专家设计师和人群来源的人类评分者；(2)纯基于ml的方法；(3)专家设计辅助ML方法。这些方法将用于更好地理解设计表征对基于群体的设计评级可靠性的影响，建立和验证用于设计评估的多模态表征学习方法，并创建用于部署用于可扩展设计评估的群体-机器学习协作的方法。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

This research project will advance the field of engineering design by developing a framework for rating the novelty and quality of design artifacts using a combination of human and machine expertise. The project will address a significant challenge in the field, which is how to accurately evaluate the vast amounts of complex, heterogeneous design ideas generated in design studies. The project will contribute to our understanding of how to effectively leverage human and computational expertise in evaluating design artifacts, leading to improved product design. While focused on an engineering design use case, the project will contribute to machine learning (ML) research by providing evidence on what, when, and how to learn from humans when evaluating heterogeneous datasets. The project will establish a website with open-source data and workshops for early-career professionals to enhance the broader impacts of the work on the engineering design community at large. The research team will partner with the Women in Science and Engineering Research (WISER) program and the Multicultural Engineering Program (MEP) at Penn State to encourage participation of underrepresented groups in engineering.The project investigates three distinct means of evaluating heterogeneous design artifacts (CAD drawings, text, sketches, prototypes) against established design metrics at scale, namely (1) human expert designers and crowd-sourced human raters; (2) pure ML-based methods; and (3) expert designer-assisted ML methods. These methods will be used to better understand the impact of design representation on the reliability of crowd-based design ratings, establish and validate multi-modal representation learning methods for design evaluation, and create methods for deploying crowd-ML collaborations for scalable design evaluation.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.