Memory Impedance for Efficient Complex-valued Neural Networks

点击翻译按钮获取中文摘要

AI has a hardware problem because current computing systems consume far too much energy. This is not sustainable and is rapidly becoming a critical societal challenge. The soaring demand for computing power vastly outpaces improvements made through Moore's scaling or innovative architectural solutions - the computing demands now double every 2 months. As a direct consequence, the cost of training state-of-the-art sophisticated AI models increased from a few $ in 2012 to ~$10m in 2020 [Nature 604, 255-260 (2022). ]. The challenge is even more pronounced where energy resources are limited (e.g. IoT devices). There is a pressing need to address this issue at the fundamental level and develop efficient AI systems. Memristors (memory + resistor) are a strong candidate for future non-CMOS computing solutions, capable of yielding significant energy-efficiency improvements. Specifically, arrays of memristive devices enable parallel multiply-and-accumulate (MAC) operations while mitigating the need for costly data movement, unlike much less efficient conventional von Neumann systems. However, memristors, transistors, and other computational primitives operate only on real-valued signals (either in a digital or analogue form). This is a significant limitation because complex numbers representing both amplitude and phase are much more compact and are used as a standard in biomedical sciences, physics, robotics, communications, image & audio processing, radar, etc. Computational primitives capable of directly manipulating complex-valued signals would yield much better energy efficiency and provide higher computational power using fewer elemental building blocks.We propose to solve this problem by generalising the concept of memristance and developing fundamentally novel nanoscale electronic elements capable of directly processing complex-valued signals. Such nanoscale devices do not yet exist; however, if developed, they would enable extremely energy-efficient direct processing at the edge where signals are recorded. We propose to develop electrically programmable analogue memimpedors (memory + impedance) - new class of computational nanodevices with programmable impedance. Once working memimpedors are realised, we will explore and demonstrate their functionality by constructing memimpedor crossbars and the first hardware accelerator of complex-valued neural networks (CVNNs). CVNNs, in contrast to commonly used artificial neural networks (ANNs), use complex values for weights and activation functions, where both magnitude and phase are essential. CVNNs, although much less studied than conventional real-valued artificial neural networks, have demonstrated better performance for structures of complex-valued data [IEEE Trans. Neural Netw. Learn. Syst. 23, 541-551, (2012)] and can solve known problems in conventional ANNs (e.g. overfitting) [arXiv:2101.12249 (2021)]. This project has the potential to generate new research directions at the interface between materials science, microelectronics, AI, and be a game-changer for timely energy-efficient, highly functional AI, neuromorphic and signal processing applications. Development of memimpedors and efficient implementations of CVNN will offer new exciting avenues in deep learning and AI.