Browsing by Subject "DNN"

Now showing 1 - 1 of 1
  • Thumbnail Image
    Publication
    Open Access
    Spartan: a sparsity-adaptive framework to accelerate deep neural network training on GPUs
    (Institute of Electrical and Electronics Engineers, 2021-03-22) Dong, Shi; Sun, Yifan; Agostini, Nicolas Bohm; Karimi, Elmira; Lowell, Daniel; Zhou, Jing; Cano, José; Abellán Miguel, José Luis; Kaeli, David; Ingeniería y Tecnología de Computadores; Facultad de Informática
    Deep Neural Networks (DNNs) have emerged as an important class of machine learning algorithms, providing accurate solutions to a broad range of applications. Sparsity in activation maps in DNN training presents an opportunity to reduce computations. However, exploiting activation sparsity presents two major challenges: i) profiling activation sparsity during training comes with significant overhead due to computing the degree of sparsity and the data movement; ii) the dynamic nature of activation maps requires dynamic dense-to-sparse conversion during training, leading to significant overhead. In this article, we present Spartan, a lightweight hardware/software framework to accelerate DNN training on a GPU. Spartan provides a cost-effective and programmer-transparent microarchitectural solution to exploit activation sparsity detected during training. Spartan provides an efficient sparsity monitor, a tile-based sparse GEMM algorithm, and a novel compaction engine designed for GPU workloads. Spartan can reduce sparsity profiling overhead by 52.5× on average. For the most compute-intensive layers, i.e., convolutional layers, we can speedup AlexNet by 3.4×, VGGNet-16 by 2.14×, and ResNet-18 by 2.02×, when training on the ImageNet dataset.