Formalizing and Automating Technology Maturity Assessment

你会用一种有40年历史的方法或工具来推进一个尖端工程项目吗？在不了解方法结果的概括性的情况下，您会使用另一个组织在不同领域开发的方法吗？目前技术成熟度评估的最佳做法是技术准备水平(TRL)量表，它是由NASA在20世纪70年代开发的。今天，TRL评估被用于在制造、能源和交通行业的组织以及世界各地的主要赠款机构和监管机构做出数百万美元的决定。TRL在现有的应用领域没有得到充分的验证，在技术密集型初创企业蓬勃发展的新领域也很少有人研究。TRL依赖于专家评估，这已被证明是高度主观的。我最近取得的进展确定了目前最佳做法在这类评估中的局限性，为拟议的工作奠定了基础。随着越来越多的人获得数据和自动分类技术，提高技术成熟度评估的可靠性是一个重要的机会。我的长期愿景是彻底改变技术成熟度评估过程，因此我提出了一个研究计划，以使这一过程形式化和自动化，并有三个短期目标：1.在工程密集型组织中建立人与人之间关于TRL评估的正式协议。设计一种收集TRL评估的实验方法，以正式建立不同行业背景下专家分歧的衡量标准。2.生成TRL的自动分类模型。收集和分析公开可用的TRL评估数据集，包括来自NASA的数据集，以及科学出版物、专利和风险投资。建立技术成熟度的概念模型，为机器学习技术成熟度分类器的开发提供信息。3.开发并验证用于决策支持的新型技术成熟度评估分析工具。结合专家和数据驱动的知识来评估技术成熟度。将这一工具体现在可公开访问的在线门户中，以便可以进行持续的数据收集，从而刺激未来的研究并引发未来的合作。建议用于技术成熟度评估的新工具将通过组织中更知情的决策来促进创新，这些决策涉及：工程密集型开发、风险资本投资和公共资金的发放。最先进的机器学习技术的应用将影响系统工程决策的广泛领域，使其重新思考关键过程，其中许多过程目前仅依赖于专家。HQP，包括本科生暑期研究人员和研究生，将不仅获得技术工程设计方面的技能，还将获得数据分析和行为研究方面的技能，这些技能目前在加拿大严重短缺。

Would you apply a 40-year old method or tool to advance a cutting-edge engineering project? Would you use a method developed at another organization, in a different field, without an understanding of the generalizability of the method's results? Current best practice in technology maturity assessment is the Technology Readiness Level (TRL) scale, which was developed by NASA in the 1970s. Today, TRL assessments are used to make multi-million-dollar decisions at organizations in the manufacturing, energy and transportation industries, and at major grant agencies and regulatory bodies worldwide. The TRL is not sufficiently validated in established fields of application, and is little studied in new fields where technology-intensive start-ups are blossoming. The TRLs rely on expert-assessments, which have been shown to be highly subjective. My recent progress has established the limitations of current best practices in such assessments, serving as a foundation for the proposed work. With increasing access to data and automated classification techniques, an important opportunity exists to improve the reliability of technology maturity assessments. My long-term vision is to revolutionize the technology maturity assessment process, and I therefore propose a research program to formalize and automate this process, with three short-term objectives: 1.Establish a formal understanding of inter-human agreement on TRL assessment in engineering-intensive organizations. Design an experimental method to collect TRL assessments, to formally establish a measure of expert disagreement in various industry contexts. 2.Generate an automated classification model for TRL. Collect and analyze publicly available data sets of TRL assessments, including one from NASA, as well as scientific publications, patents, and venture funding. Build a conceptual model of technology maturity to inform the development of a machine-learning technology-maturity classifier. 3.Develop and validate a novel Technology Maturity Assessment Analytics Tool for decision support. Incorporate expert and data-driven knowledge to assess technology maturity. Embody this tool in a publicly accessible online portal such that ongoing data collection can be conducted, stimulating future research and sparking future collaboration. The novel tool proposed for technology maturity assessment will boost innovation through better-informed decisions at organizations that involve: engineering-intensive development, venture capital investment, and the granting of public funds. The application of state-of-the-art machine learning techniques will influence the broad field of systems engineering decision-making to re-think critical processes, many of which currently rely solely on experts. HQP, including undergraduate summer researchers and graduate students, will gain skills not only in technical engineering design but also data analytics and behavioural studies, currently in critical shortage in Canada.