Hardware-aware Network Architecture Search under ML Training workloads

实现最佳的延迟/能量/精度需要神经网络（NN）工作负载及其部署的系统硬件的协同优化。这在使用电池的移动系统上尤为重要，因为要部署更复杂的工作负载，再训练适应可能会受到能量或计算限制的限制。神经架构搜索（NAS）解决方案能够以有效的方式探索神经网络图特征，使神经架构的设计细节适应手头的机器学习任务。本博士的目的是解决一个开源的，硬件感知的NAS方法的设计，不仅优化推理，而且优化设备上的训练。此外，硬件方面的考虑将扩展到较低级别的处理器规格（例如，NVM参数、DVFS设计）以及系统级特性。关键研究问题本研究的目的是探讨硬件感知网络架构搜索（NAS）领域的开放性问题。特别是，它将侧重于概念验证系统模型，该模型将意识到提高效率的既定机会。这些例子包括不同频率/电压工作点（DVFS）和具有多种电源工作模式的混合存储器（片外/片内，非易失性/易失性）。一个关键的挑战将是将这种硬件定义的搜索空间与神经架构的搜索空间相结合，以一种易于处理的方式对神经架构和硬件设计空间进行联合优化。这种NAS方法的设计将是一个有用的工具，但同样重要的是该工具能够产生的结果-特别是新颖的神经架构和新颖的硬件设计，特别是在共同运行时产生高效率的系统。预计通过分析工作负载感知优化系统，设计和微架构创新将为这种异构硬件设计带来卓越的功率、性能和面积指标。在博士期间，学生将研究和解决包括NAS方案和优化设备上培训工作量的系统模型的计算和内存要求。在解决关键研究问题时，学生将在一流的环境中与剑桥大学CaMLSys团队的同事一起工作。学生将在工业合作伙伴IMEC工作至少3个月，在不同的环境中与致力于纳米电子和数字技术的世界领先研发和创新的科学家一起工作。

Achieving the optimal delay/energy/accuracy requires the co-optimization of Neural Network (NN) workload together with the system hardware it is deployed in. This is particularly important on mobile systems running on batteries, where to deploy more complex workloads, retraining-adaptation might be limited due to energy or computation constraints.Neural Architecture Search (NAS) solutions enable the exploration of the NN graph characteristics in an efficient way, adapting the design specifics of the neural architecture to the machine learning task at hand. The aim of this PhD tackles the design of an open-source, hardware aware NAS methodology to optimize not only inference but also on-device training. Furthermore, hardware considerations will be expanded to lower-level processor specifications (for instance, NVM parameters, DVFS design) alongside system-level characteristics.Key Research QuestionsThe objective of the research is to investigate the open problems in the domain of hardware-aware network architecture search (NAS). In particular, it will focus on a proof of concept system model that will be aware of established opportunities for efficiency. Examples of these include different frequency/voltage operation points (DVFS) and hybrid memories (off-chip/on-chip, non-volatile/volatile) with multiple power operation modes. A key challenge will be integrating this hardware-defined search-space with that of search-space of the neural architecture to perform joint-optimization of both neural architecture and hardware design spaces in a tractable manner. The design of this NAS method will be a useful tool, but just as important will be the results this tool is able to produce - specifically novel neural architectures and novel hardware designs that especially when run jointly produce high efficiency systems. It is expected that by analysing the workload-aware optimized systems, design and micro-architectural innovations will result in the exceptional power, performance and area metrics for such heterogeneous hardware designs.Work expected of the ICASE studentDuring the Ph.D, the student will research and address the computing and memory requirements of a system model that includes the NAS scheme and optimises an on-device training workload.While addressing the key research questions, the student will work in a top-class environment, with colleagues from the CaMLSys group at the University of Cambridge. There will be a minimum of a 3 month placement with the industrial partner IMEC where the student will work in a diverse environment with scientists devoted to world leading R&D and innovation in nanoelectronics and digital technologies.