Formalizing and Automating Technology Maturity Assessment

你会用40年前的方法或工具来推进一个尖端的工程项目吗？你会使用另一个组织在不同领域开发的方法，而不了解该方法结果的普遍性吗？目前技术成熟度评估的最佳做法是技术准备程度（TRL）量表，该量表由NASA在20世纪70年代开发。今天，TRL评估被用于在制造业，能源和运输行业的组织以及全球主要的赠款机构和监管机构做出数百万美元的决策。 TRL在已建立的应用领域中没有得到充分验证，在技术密集型初创企业蓬勃发展的新领域中也很少进行研究。TRL依赖于专家评估，这已被证明是高度主观的。我最近取得的进展确定了目前这种评估最佳做法的局限性，为拟议的工作奠定了基础。随着越来越多地获得数据和自动分类技术，存在着一个重要的机会来提高技术成熟度评估的可靠性。 我的长期愿景是彻底改变技术成熟度评估过程，因此，我提出了一个研究计划，以正式化和自动化这一过程，有三个短期目标：1.建立一个正式的理解，在工程密集型组织的TRL评估的人与人之间的协议。设计一种实验方法来收集TRL评估，以正式建立各种行业背景下专家分歧的衡量标准。2.为TRL生成自动分类模型。收集和分析TRL评估的公开数据集，包括来自NASA的数据集，以及科学出版物，专利和风险投资。建立技术成熟度的概念模型，为机器学习技术成熟度分类器的开发提供信息。3.开发和验证一个新的技术成熟度评估分析工具，以支持决策。结合专家和数据驱动的知识来评估技术成熟度。将这一工具纳入一个可公开访问的在线门户网站，以便进行持续的数据收集，促进未来的研究和激发未来的合作。为技术成熟度评估提出的新工具将通过组织更明智的决策来促进创新，这些决策涉及：工程密集型开发、风险投资和公共资金的授予。最先进的机器学习技术的应用将影响系统工程决策的广泛领域，以重新思考关键流程，其中许多流程目前仅依赖专家。HQP，包括本科暑期研究人员和研究生，将获得技能，不仅在技术工程设计，而且数据分析和行为研究，目前在加拿大严重短缺。

Would you apply a 40-year old method or tool to advance a cutting-edge engineering project? Would you use a method developed at another organization, in a different field, without an understanding of the generalizability of the method's results? Current best practice in technology maturity assessment is the Technology Readiness Level (TRL) scale, which was developed by NASA in the 1970s. Today, TRL assessments are used to make multi-million-dollar decisions at organizations in the manufacturing, energy and transportation industries, and at major grant agencies and regulatory bodies worldwide. The TRL is not sufficiently validated in established fields of application, and is little studied in new fields where technology-intensive start-ups are blossoming. The TRLs rely on expert-assessments, which have been shown to be highly subjective. My recent progress has established the limitations of current best practices in such assessments, serving as a foundation for the proposed work. With increasing access to data and automated classification techniques, an important opportunity exists to improve the reliability of technology maturity assessments. My long-term vision is to revolutionize the technology maturity assessment process, and I therefore propose a research program to formalize and automate this process, with three short-term objectives: 1.Establish a formal understanding of inter-human agreement on TRL assessment in engineering-intensive organizations. Design an experimental method to collect TRL assessments, to formally establish a measure of expert disagreement in various industry contexts. 2.Generate an automated classification model for TRL. Collect and analyze publicly available data sets of TRL assessments, including one from NASA, as well as scientific publications, patents, and venture funding. Build a conceptual model of technology maturity to inform the development of a machine-learning technology-maturity classifier. 3.Develop and validate a novel Technology Maturity Assessment Analytics Tool for decision support. Incorporate expert and data-driven knowledge to assess technology maturity. Embody this tool in a publicly accessible online portal such that ongoing data collection can be conducted, stimulating future research and sparking future collaboration. The novel tool proposed for technology maturity assessment will boost innovation through better-informed decisions at organizations that involve: engineering-intensive development, venture capital investment, and the granting of public funds. The application of state-of-the-art machine learning techniques will influence the broad field of systems engineering decision-making to re-think critical processes, many of which currently rely solely on experts. HQP, including undergraduate summer researchers and graduate students, will gain skills not only in technical engineering design but also data analytics and behavioural studies, currently in critical shortage in Canada.