Approximate Hybrid Binary-Unary Computing with Applications in BERT Language Model and Image Processing
Approximate Hybrid Binary-Unary Computing with Applications in BERT Language Model and Image Processing
复制标题
近似混合二元-一元计算及其在 BERT 语言模型和图像处理中的应用
DOI：
10.1145/3543622.3573181
复制
发表时间：
2023
期刊：
影响因子：
--
通讯作者：
Bazargan, Kia
中科院分区：
文献类型：
--
作者：
Khataei, Alireza;Singh, Gaurav;Bazargan, Kia
We propose a novel method for approximate hardware implementation of univariate math functions with significantly fewer hardware resources compared to previous approaches. Examples of such functions include exp(x) and the activation function GELU(x), both used in transformer networks, gamma(x), which is used in image processing, and other functions such as tanh(x), cosh(x), sq(x), and sqrt(x). The method builds on previous works on hybrid binary-unary computing. The novelty in our approach is that we break a function into a number of sub-functions such that implementing each sub-function becomes cheap, and converting the output of the sub-functions to binary becomes almost trivial. Our method also uses self-similarity in functions to further reduce the cost. We compare our method to the conventional binary, previous stochastic computing, and hybrid binary-unary methods on several functions at 8-, 12-, and 16-bit resolutions. While preserving high accuracy, our method outperforms previous works in terms of hardware cost, e.g., tolerating less than 0.01 mean absolute error, our method reduces the (area x latency) cost on average by 5, 7, and 2 orders of magnitude, compared to the conventional binary, stochastic computing, and hybrid binary-unary methods, respectively. Ultimately, we demonstrate the potential benefits of our method for natural language processing and image processing applications. We deploy our method to implement major blocks in an encoding layer of BERT language model, and also the Roberts Cross edge detection algorithm. Both include non-linear functions.
We propose a novel method for approximate hardware implementation of univariate math functions with significantly fewer hardware resources compared to previous approaches. Examples of such functions include exp(x) and the activation function GELU(x), both used in transformer networks, gamma(x), which is used in image processing, and other functions such as tanh(x), cosh(x), sq(x), and sqrt(x). The method builds on previous works on hybrid binary-unary computing. The novelty in our approach is that we break a function into a number of sub-functions such that implementing each sub-function becomes cheap, and converting the output of the sub-functions to binary becomes almost trivial. Our method also uses self-similarity in functions to further reduce the cost. We compare our method to the conventional binary, previous stochastic computing, and hybrid binary-unary methods on several functions at 8-, 12-, and 16-bit resolutions. While preserving high accuracy, our method outperforms previous works in terms of hardware cost, e.g., tolerating less than 0.01 mean absolute error, our method reduces the (area x latency) cost on average by 5, 7, and 2 orders of magnitude, compared to the conventional binary, stochastic computing, and hybrid binary-unary methods, respectively. Ultimately, we demonstrate the potential benefits of our method for natural language processing and image processing applications. We deploy our method to implement major blocks in an encoding layer of BERT language model, and also the Roberts Cross edge detection algorithm. Both include non-linear functions.
登录
查看更多内容
混合二进制-一元硬件加速器
DOI：
--
发表时间：
2019
期刊：
Asia and South Pacific Design Automation Conference
影响因子：
--
作者：
S. R. Faraji;K. Bazargan
通讯作者：
K. Bazargan
基于函数导数的并行一元计算
DOI：
10.1145/3418464
发表时间：
2020
期刊：
ACM Transactions on Reconfigurable Technology and Systems (TRETS)
影响因子：
--
作者：
S. Mohajer;Zhiheng Wang;K. Bazargan;Yuyang Li
通讯作者：
Yuyang Li
适用于 FPGA 上 DSP 应用的混合二元-一元截断乘法
DOI：
--
发表时间：
2020
期刊：
2020 IEEE/ACM International Conference On Computer Aided Design (ICCAD)
影响因子：
--
作者：
S. R. Faraji;K. Bazargan
通讯作者：
K. Bazargan
HBUNN - 混合二元-一元神经网络：在 FPGA 上实现完整的 CNN
DOI：
10.1109/iccd46524.2019.00027
发表时间：
2019
期刊：
2019 IEEE 37th International Conference on Computer Design (ICCD)
影响因子：
--
作者：
Sayed Abdolrasouol Faraji;Gaurav Singh;K. Bazargan
通讯作者：
K. Bazargan
更新的方法
DOI：
--
发表时间：
2012
期刊：
影响因子：
--
作者：
Fukuzawa;K.;Dannoura;M. and Shibata;H
通讯作者：
H