A gradient-interleaved scheduler for energy-efficient backpropagation for training neural networks

Nanda Unnikrishnan; Keshab K. Parhi

A gradient-interleaved scheduler for energy-efficient backpropagation for training neural networks

Nanda Unnikrishnan, Keshab K. Parhi

Electrical and Computer Engineering

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

3 Scopus citations

Abstract

This paper addresses design of accelerators using systolic architectures for training of neural networks using a novel gradient interleaving approach. Training the neural network involves backpropagation of error and computation of gradients with respect to the activation functions and weights. It is shown that the gradient with respect to the activation function can be computed using a weight-stationary systolic array while the gradient with respect to the weights can be computed using an output-stationary systolic array. The novelty of the proposed approach lies in interleaving the computations of these two gradients to the same configurable systolic array. This results in reuse of the variables from one computation to the other and eliminates unnecessary memory accesses. The proposed approach leads to 1.4 − 2.2× savings in terms of number of cycles and 1.9× savings in terms of memory accesses. Thus, the proposed accelerator reduces latency and energy consumption.

Original language	English (US)
Title of host publication	2020 IEEE International Symposium on Circuits and Systems, ISCAS 2020 - Proceedings
Publisher	Institute of Electrical and Electronics Engineers Inc.
ISBN (Electronic)	9781728133201
State	Published - 2020
Event	52nd IEEE International Symposium on Circuits and Systems, ISCAS 2020 - Virtual, Online Duration: Oct 10 2020 → Oct 21 2020

Publication series

Name	Proceedings - IEEE International Symposium on Circuits and Systems
Volume	2020-October
ISSN (Print)	0271-4310

Conference

Conference	52nd IEEE International Symposium on Circuits and Systems, ISCAS 2020
City	Virtual, Online
Period	10/10/20 → 10/21/20

Bibliographical note

Funding Information:
This research was supported in part by the National Science Foundation under grant number CCF-1814759.

Publisher Copyright:
© 2021 IEEE

Keywords

Accelerator architectures
Deep learning
Gradient interleaving
Neural network
Processor scheduling
Systolic array

OpenUrl availability

Full text

Cite this

Unnikrishnan, N., & Parhi, K. K. (2020). A gradient-interleaved scheduler for energy-efficient backpropagation for training neural networks. In 2020 IEEE International Symposium on Circuits and Systems, ISCAS 2020 - Proceedings Article 9181242 (Proceedings - IEEE International Symposium on Circuits and Systems; Vol. 2020-October). Institute of Electrical and Electronics Engineers Inc..

A gradient-interleaved scheduler for energy-efficient backpropagation for training neural networks. / Unnikrishnan, Nanda; Parhi, Keshab K.
2020 IEEE International Symposium on Circuits and Systems, ISCAS 2020 - Proceedings. Institute of Electrical and Electronics Engineers Inc., 2020. 9181242 (Proceedings - IEEE International Symposium on Circuits and Systems; Vol. 2020-October).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Unnikrishnan, N & Parhi, KK 2020, A gradient-interleaved scheduler for energy-efficient backpropagation for training neural networks. in 2020 IEEE International Symposium on Circuits and Systems, ISCAS 2020 - Proceedings., 9181242, Proceedings - IEEE International Symposium on Circuits and Systems, vol. 2020-October, Institute of Electrical and Electronics Engineers Inc., 52nd IEEE International Symposium on Circuits and Systems, ISCAS 2020, Virtual, Online, 10/10/20.

@inproceedings{afb0f89a465844cb937fa01a8e3e260e,

title = "A gradient-interleaved scheduler for energy-efficient backpropagation for training neural networks",

abstract = "This paper addresses design of accelerators using systolic architectures for training of neural networks using a novel gradient interleaving approach. Training the neural network involves backpropagation of error and computation of gradients with respect to the activation functions and weights. It is shown that the gradient with respect to the activation function can be computed using a weight-stationary systolic array while the gradient with respect to the weights can be computed using an output-stationary systolic array. The novelty of the proposed approach lies in interleaving the computations of these two gradients to the same configurable systolic array. This results in reuse of the variables from one computation to the other and eliminates unnecessary memory accesses. The proposed approach leads to 1.4 − 2.2× savings in terms of number of cycles and 1.9× savings in terms of memory accesses. Thus, the proposed accelerator reduces latency and energy consumption.",

keywords = "Accelerator architectures, Deep learning, Gradient interleaving, Neural network, Processor scheduling, Systolic array",

author = "Nanda Unnikrishnan and Parhi, {Keshab K.}",

note = "Funding Information: This research was supported in part by the National Science Foundation under grant number CCF-1814759. Publisher Copyright: {\textcopyright} 2021 IEEE; 52nd IEEE International Symposium on Circuits and Systems, ISCAS 2020 ; Conference date: 10-10-2020 Through 21-10-2020",

year = "2020",

language = "English (US)",

series = "Proceedings - IEEE International Symposium on Circuits and Systems",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

booktitle = "2020 IEEE International Symposium on Circuits and Systems, ISCAS 2020 - Proceedings",

}

TY - GEN

T1 - A gradient-interleaved scheduler for energy-efficient backpropagation for training neural networks

AU - Unnikrishnan, Nanda

AU - Parhi, Keshab K.

PY - 2020

Y1 - 2020

N2 - This paper addresses design of accelerators using systolic architectures for training of neural networks using a novel gradient interleaving approach. Training the neural network involves backpropagation of error and computation of gradients with respect to the activation functions and weights. It is shown that the gradient with respect to the activation function can be computed using a weight-stationary systolic array while the gradient with respect to the weights can be computed using an output-stationary systolic array. The novelty of the proposed approach lies in interleaving the computations of these two gradients to the same configurable systolic array. This results in reuse of the variables from one computation to the other and eliminates unnecessary memory accesses. The proposed approach leads to 1.4 − 2.2× savings in terms of number of cycles and 1.9× savings in terms of memory accesses. Thus, the proposed accelerator reduces latency and energy consumption.

AB - This paper addresses design of accelerators using systolic architectures for training of neural networks using a novel gradient interleaving approach. Training the neural network involves backpropagation of error and computation of gradients with respect to the activation functions and weights. It is shown that the gradient with respect to the activation function can be computed using a weight-stationary systolic array while the gradient with respect to the weights can be computed using an output-stationary systolic array. The novelty of the proposed approach lies in interleaving the computations of these two gradients to the same configurable systolic array. This results in reuse of the variables from one computation to the other and eliminates unnecessary memory accesses. The proposed approach leads to 1.4 − 2.2× savings in terms of number of cycles and 1.9× savings in terms of memory accesses. Thus, the proposed accelerator reduces latency and energy consumption.

KW - Accelerator architectures

KW - Deep learning

KW - Gradient interleaving

KW - Neural network

KW - Processor scheduling

KW - Systolic array

UR - http://www.scopus.com/inward/record.url?scp=85109345040&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85109345040&partnerID=8YFLogxK

M3 - Conference contribution

AN - SCOPUS:85109345040

T3 - Proceedings - IEEE International Symposium on Circuits and Systems

BT - 2020 IEEE International Symposium on Circuits and Systems, ISCAS 2020 - Proceedings

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 52nd IEEE International Symposium on Circuits and Systems, ISCAS 2020

Y2 - 10 October 2020 through 21 October 2020

ER -

A gradient-interleaved scheduler for energy-efficient backpropagation for training neural networks

Abstract

Publication series

Conference

Bibliographical note

Keywords

OpenUrl availability

Other files and links

Fingerprint

Cite this