Hardware-aware Network Architecture Search under ML Training workloads

实现最佳延迟/能量/精度需要神经网络（NN）工作负载与其部署的系统硬件一起进行协同优化。这在依靠电池运行的移动的系统上尤其重要，因为在这些系统中，要部署更复杂的工作负载，再训练-自适应可能会受到能量或计算限制的限制。神经架构搜索（NAS）解决方案能够以有效的方式探索NN图特征，使神经架构的设计细节适应手头的机器学习任务。这个博士学位的目的是解决开源，硬件感知NAS方法的设计，不仅优化推理，而且优化设备上的训练。此外，硬件的考虑将扩大到较低级别的处理器规格（例如，NVM参数，DVFS设计）旁边的系统级characteristics.Key研究的目的是调查开放的问题，在该领域的硬件感知网络体系结构搜索（NAS）。特别是，它将侧重于概念系统模型的证明，该模型将意识到现有的效率机会。这些的示例包括不同的频率/电压操作点（DVFS）和具有多个功率操作模式的混合存储器（片外/片上、非易失性/易失性）。一个关键的挑战将是将这个硬件定义的搜索空间与神经架构的搜索空间相集成，以易于处理的方式对神经架构和硬件设计空间进行联合优化。这种NAS方法的设计将是一种有用的工具，但同样重要的是这种工具能够产生的结果-特别是新颖的神经架构和新颖的硬件设计，特别是当联合运行时，可以产生高效率的系统。预计通过分析工作负载感知优化系统，设计和微架构创新将为此类异构硬件设计带来卓越的功率，性能和面积指标。学生将研究和解决一个系统模型的计算和内存要求，该模型包括NAS方案，并优化了一个在解决关键研究问题的同时，学生将在一流的环境中与来自剑桥大学CaMLSys小组的同事一起工作。将有至少3个月的实习与工业合作伙伴IMEC，学生将在一个多元化的环境中与科学家致力于世界领先的研发和创新的纳米电子和数字技术。

Achieving the optimal delay/energy/accuracy requires the co-optimization of Neural Network (NN) workload together with the system hardware it is deployed in. This is particularly important on mobile systems running on batteries, where to deploy more complex workloads, retraining-adaptation might be limited due to energy or computation constraints.Neural Architecture Search (NAS) solutions enable the exploration of the NN graph characteristics in an efficient way, adapting the design specifics of the neural architecture to the machine learning task at hand. The aim of this PhD tackles the design of an open-source, hardware aware NAS methodology to optimize not only inference but also on-device training. Furthermore, hardware considerations will be expanded to lower-level processor specifications (for instance, NVM parameters, DVFS design) alongside system-level characteristics.Key Research QuestionsThe objective of the research is to investigate the open problems in the domain of hardware-aware network architecture search (NAS). In particular, it will focus on a proof of concept system model that will be aware of established opportunities for efficiency. Examples of these include different frequency/voltage operation points (DVFS) and hybrid memories (off-chip/on-chip, non-volatile/volatile) with multiple power operation modes. A key challenge will be integrating this hardware-defined search-space with that of search-space of the neural architecture to perform joint-optimization of both neural architecture and hardware design spaces in a tractable manner. The design of this NAS method will be a useful tool, but just as important will be the results this tool is able to produce - specifically novel neural architectures and novel hardware designs that especially when run jointly produce high efficiency systems. It is expected that by analysing the workload-aware optimized systems, design and micro-architectural innovations will result in the exceptional power, performance and area metrics for such heterogeneous hardware designs.Work expected of the ICASE studentDuring the Ph.D, the student will research and address the computing and memory requirements of a system model that includes the NAS scheme and optimises an on-device training workload.While addressing the key research questions, the student will work in a top-class environment, with colleagues from the CaMLSys group at the University of Cambridge. There will be a minimum of a 3 month placement with the industrial partner IMEC where the student will work in a diverse environment with scientists devoted to world leading R&D and innovation in nanoelectronics and digital technologies.