Automatic Deep Neural Network Design by a Novel Stochastic Search Process

机器学习，特别是深度学习方面的进步已经为不同的应用带来了更好的建模精度，如图像分类、对象检测、语音识别，甚至医疗应用。然而，这些模型的设计周期仍然依赖于手动的、耗时的、非常复杂的过程，这需要高级的专业知识。因此，使用这种耗时的人工过程来设计和开发专业和高效的神经网络模型是非常困难的，这减慢了它们在许多实际应用中的采用速度。特别是当需要考虑到计算能力和可用硬件内存的固有限制时，这种缺点变得更加严重。神经结构搜索（NAS）方法为解决这些挑战提供了一种解决方案。NAS通过启发式搜索解空间，自动设计模型的底层图结构，寻找最优结构。尽管NAS方法有很大的潜力，但它仍处于起步阶段，计算复杂，仅限于特定的应用。拟议的研究计划旨在减轻这些挑战。以下目标将被研究：i)研究随机分支预测策略，以降低搜索空间架构的维数和相应的计算复杂度。ii)描述基于新随机过程的深度学习模型计算图中的复杂信息。iii)在设计过程中结合硬件特征，并开发一种新的方法来为深度学习模型生成特定于硬件的优化计算图。vi)通过考虑环境属性的新方法设计和开发更具适应性的机器学习系统。该研究计划的成果将提高深度神经网络在自动驾驶、消费电子、医疗保健等不同行业的适用性。由于其自动化的特性，所提出的方法大大缩短了设计周期，这有助于开发基于该技术的应用程序，而最终用户的专业知识水平要低得多。这种新方法也将推动加拿大作为人工智能领域的先驱国家在不同行业采用深度学习模型。在拟议的研究项目中开发的知识和技术将通过与微软、英特尔和达尔文人工智能等工业伙伴的积极合作进行转移。这确保了为现实世界的机器学习应用程序高效开发和部署深度神经网络。在这个项目中训练的HQP将成为人工智能和机器学习领域的专家，使他们在工业界和学术界的人工智能领导角色中处于有利地位，目前专家短缺。

The promising advances made in machine learning and especially deep learning have led to better modeling accuracy for different applications such as image classification, object detection, speech recognition, and even medical applications. However, the design cycle of these models still rely on manual, time-consuming, and very complex process, which requires high-level expertise. Consequently, the design and development of specialized and efficient neural network models using such time-consuming manual processes are very difficult slowing down their adoption in many real-world applications. Such shortcoming becomes even more critical especially when it is necessary to take into account the inherent limitations in the computational power and the memory of the available hardware. Neural architecture search (NAS) methods offer a solution to address these challenges. NAS designs the underlying graph structure of models automatically by heuristically searching the solution space to find the most optimal structure. Despite their promising potentials, NAS approaches are still in their infancy, computationally complex and limited to specific applications. The proposed research program aims to mitigate these challenges. The following objectives will be investigated: i) Investigate stochastic branch prediction strategies to reduce the dimensionality of the search space architecture and accordingly the computational complexity. ii) Characterize the complex information inside the computational graph of a deep learning model based on the new stochastic process. iii) Incorporate hardware characteristics in the design process and develop a novel approach to generate hardware-specific optimized computational graphs for the deep learning models. vi) Design and develop more adaptive machine learning systems via the novel method which take environmental properties into account. The results of the proposed research program will increase the applicability of deep neural networks in different industries such as autonomous driving, consumer electronics and healthcare. Due to its automatic nature, the proposed approach offers dramatically reduced design cycles, which facilitates developing applications based on this technology with much less level of end-user expertise. This new approach will also boost the adoption of deep learning models in different industries in Canada as a pioneer country in the field of AI. The knowledge and technologies developed in the proposed research program will be transferred through active collaborations with industrial partners such as Microsoft, Intel and DarwinAI. This ensures the efficient development and deployment of deep neural networks for real-world Machine Learning applications. HQP trained in this program will become experts in the field of artificial intelligence and machine learning, putting them in a strong position for AI leadership roles in industry as well as in academia, where there is currently a shortage of experts.