Collaborative Research: Leveraging Crowd-AI Teams for Scalable Novelty Ratings of Heterogeneous Design Representations

协作研究：利用群体人工智能团队对异构设计表示进行可扩展的新颖性评级

• 批准号: 2231261
• 负责人: Scarlett Miller
• 金额: $ 29.97万
• 依托单位: Pennsylvania State Univ University Park
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2023
• 资助国家: 美国
• 起止时间: 2023-07-01 至 2026-06-30
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2231261&HistoricalAwards=false
• 关键词: Collaborative; Research; Leveraging; Crowd; AI

This research project will advance the field of engineering design by developing a framework for rating the novelty and quality of design artifacts using a combination of human and machine expertise. The project will address a significant challenge in the field, which is how to accurately evaluate the vast amounts of complex, heterogeneous ideas generated in design studies. The project will contribute to understanding how to effectively leverage human and computational expertise in evaluating design artifacts, leading to improved product design. While focused on an engineering design use case, the project will contribute to machine learning (ML) research by providing evidence on what, when, and how to learn from humans when evaluating heterogeneous datasets. The project will establish a website with open-source data and workshops for early-career professionals to enhance the broader impacts of the work on the engineering design community at large. The research team will partner with the Women in Science and Engineering Research (WISER) program and the Multicultural Engineering Program (MEP) at Penn State to encourage participation of underrepresented groups in engineering.The project investigates three distinct means of evaluating heterogeneous design artifacts (CAD drawings, text, sketches, prototypes) against established design metrics at scale, namely (1) human expert designers and crowd-sourced human raters; (2) pure ML-based methods; and (3) expert designer-assisted ML methods. These methods will be used to better understand the impact of design representation on the reliability of crowd-based design ratings, establish and validate multi-modal representation learning methods for design evaluation, and create methods for deploying crowd-ML collaborations for scalable design evaluation.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

该研究项目将通过开发一个框架来推进工程设计领域，该框架用于使用人类和机器专业知识相结合来评估设计工件的新奇和质量。该项目将解决该领域的一个重大挑战，即如何准确评估设计研究中产生的大量复杂，异构的想法。该项目将有助于理解如何有效地利用人类和计算专业知识来评估设计工件，从而改进产品设计。虽然专注于工程设计用例，但该项目将通过提供关于在评估异构数据集时从人类学习什么、何时以及如何学习的证据，为机器学习（ML）研究做出贡献。该项目将建立一个网站，提供开放源数据，并为职业生涯初期的专业人员举办讲习班，以加强这项工作对整个工程设计界的广泛影响。该研究团队将与宾夕法尼亚州立大学的科学与工程研究女性项目（WISER）和多元文化工程项目（MEP）合作，鼓励代表性不足的群体参与工程。（CAD图纸，文本，草图，原型）与已建立的大规模设计指标，即（1）人类专家设计师和众包人类评分员;（2）纯ML方法;（3）专家设计辅助ML方法。这些方法将用于更好地理解设计表征对基于人群的设计评级的可靠性的影响，建立和验证用于设计评估的多模态表征学习方法，并创造方法来部署人群-可扩展设计评估的ML合作。该奖项反映了NSF的法定使命，并被认为值得通过使用基金会的智力价值和更广泛的评估来支持。影响审查标准。