Deep Neural Networks (DNNs) are being used in an increasing number of applications. However, DNNs have high energy (computational and memory) requirements, limiting their usage on hand-held devices. Therefore, it is essential to decrease these requirements in DNNs, within the allowed degradation in DNN output quality. Towards this, smoothed LASSO (Least Absolute Shrinkage and Selection Operator)-based training procedures are used to induce sparsity in DNNs. The results obtained show that regardless of error profile, sparsity values obtained using various smoothed LASSO functions are similar, provided the maximum error of these functions with respect to the LASSO function is the same.

The smoothed LASSO-based training procedures must be accompanied by effective pruning heuristics to enhance DNN sparsity. A novel layer-wise DNN pruning algorithm is proposed in this direction, where the layers are pruned based on their individual allocated accuracy loss budget, which is determined by estimates of the reduction in number of multiply-accumulate operations (in convolutional layers) and weights (in fully connected layers). The proposed layer-wise pruning algorithm is extended to structured pruning, where groups of weights are removed as a whole. The various types of groupings and the trade-offs involved will be discussed. The results obtained on various DNN architectures of MNIST, SVHN, CIFAR-10, Imagenette datasets will be presented.