Publication: Spartan: a sparsity-adaptive framework to accelerate deep neural network training on GPUs
Authors
Dong, Shi ; Sun, Yifan ; Agostini, Nicolas Bohm ; Karimi, Elmira ; Lowell, Daniel ; Zhou, Jing ; Cano, José ; Abellán Miguel, José Luis ; Kaeli, David
item.page.secondaryauthor
Facultad de Informática
item.page.director
Publisher
Institute of Electrical and Electronics Engineers
publication.page.editor
publication.page.department
DOI
https://doi.org/10.1109/TPDS.2021.3067825
item.page.type
info:eu-repo/semantics/article
Description
Abstract
Deep Neural Networks (DNNs) have emerged as an important class of machine learning algorithms, providing accurate solutions to a broad range of applications. Sparsity in activation maps in DNN training presents an opportunity to reduce computations. However, exploiting activation sparsity presents two major challenges: i) profiling activation sparsity during training comes with significant overhead due to computing the degree of sparsity and the data movement; ii) the dynamic nature of activation maps requires dynamic dense-to-sparse conversion during training, leading to significant overhead. In this article, we present Spartan, a lightweight hardware/software framework to accelerate DNN training on a GPU. Spartan provides a cost-effective and programmer-transparent microarchitectural solution to exploit activation sparsity detected during training. Spartan provides an efficient sparsity monitor, a tile-based sparse GEMM algorithm, and a novel compaction engine designed for GPU workloads. Spartan can reduce sparsity profiling overhead by 52.5× on average. For the most compute-intensive layers, i.e., convolutional layers, we can speedup AlexNet by 3.4×, VGGNet-16 by 2.14×, and ResNet-18 by 2.02×, when training on the ImageNet dataset.
publication.page.subject
Monitoring , Sparse matrices , Graphics processing units , Acceleration , Market research , Engines , DNN , Sparsity , GPU , Training
Citation
IEEE Transactions on Parallel and Distributed Systems, Volume: 32, Issue: 10, 01 October 2021
item.page.embargo
Collections
Ir a Estadísticas
Este ítem está sujeto a una licencia Creative Commons. http://creativecommons.org/licenses/by/4.0/