The increasing deployment of large language models (LLMs) has magnified the computational and memory bottlenecks of autoregressive decoding, where low compute intensity and bandwidth-bound kernels dominate inference cost. Weight pruning offers a promising remedy, but existing methods remain confined to either static pruning (SP)—which permanently removes redundant weights but lacks adaptivity—or dynamic pruning (DP)—which adapts to input sparsity but introduces runtime irregularity. This paper presents SPDP, a unified sparse-inference framework that integrates unstructured SP with input-adaptive DP for efficient LLM inference on GPUs.
SPDP co-designs a new Tiled-Column-wise Bitmap Compressed (Tiled-CBC) format and two complementary GPU kernels: (1) a CUDA-core spMspV kernel featuring Hybrid Activation-aware Dynamic Shared-Memory Bitmap Decoding (HAD-SMBD) for fine-grained, runtime activation skipping, and (2) a Tensor-Core SpMM kernel optimized for prefill computation. This joint format–kernel design harmonizes static and dynamic sparsity, maintaining bandwidth-efficient memory access and high compute intensity under both phases of LLM inference. Comprehensive evaluations on inference-optimized GPUs demonstrate that SPDP achieves 1.24×–1.37× average speedup (up to 2.51×) over state-of-the-art sparse frameworks such as SpInfer, while matching perplexity with up to 25% higher sparsity. SPDP advances the inference efficiency–quality Pareto frontier, showing that unified static–dynamic pruning can deliver substantial throughput and performance-per-watt improvements in large-scale LLM serving.
Static pruning (SP) permanently removes redundant weights but is input-agnostic, so quality erodes as sparsity grows. Dynamic pruning (DP) adaptively zeros activations per input, but is coarse-grained (column-level) and keeps the dense weights resident in memory. The two are complementary—so a natural question is whether they can be jointly employed to push the speed–quality Pareto frontier further than either alone.
The perplexity evidence says yes. On Llama2-7B (WikiText), combined pruning attains its lowest perplexity at moderate total sparsity (~0.3–0.4) rather than at either extreme, and unified SP+DP (SPDP) sustains the lowest perplexity across all sparsity levels. The margin over static-only Wanda widens sharply beyond ~0.5—precisely the aggressive-sparsity regime where meaningful speedups become practically attainable.
SPDP is built around a single design principle: make static and dynamic sparsity work through one representation and one execution model, so that both prefill and decode stay efficient. It maps three design challenges to three co-designed components.
We first benchmark the SPDP-spMspV decode kernel directly against six baselines—the
Tensor-Core dense cuBLAS, cuSPARSE, Sputnik, SparTA, Flash-LLM,
and the state-of-the-art SpInfer—with all speedups normalized to the
Tensor-Core cuBLAS baseline, over LLM projection-matrix shapes.
On the A10G GPU, SPDP-spMspV achieves 1.88× average
speedup over cuBLAS and 1.24× over SpInfer
(up to 3.32× and 1.70×). On the L4 GPU, average speedups reach
2.11× and 1.37× (up to 3.52× and
2.52×). The advantage is largest in the moderate 30–50%
static-sparsity regime targeted by practical LLM inference (shaded), where DP contributes
input-dependent column skipping on top of static compression.
cuBLAS baseline on A10G, sweeping
static sparsity 0–90% with DP fixed at 30%. The shaded band marks the moderate-sparsity
regime targeted by LLM inference.
At matched perplexity, SPDP reaches up to 25% higher total sparsity than static-only pruning (SpInfer / Wanda) while delivering lower time-per-output-token (TPOT), across A10G, L4, and L40S GPUs. Because dynamic pruning is essentially “free” in quality, SPDP moves further along the efficiency–quality Pareto frontier than either static or dynamic pruning alone.
@article{kim2026spdp,
author = {Jinhyeok Kim and Yejoon Lee and Jaeyoung Do},
title = {Unified Static--Dynamic Pruning for Efficient LLM Inference},
journal = {Proceedings of the VLDB Endowment (PVLDB)},
volume = {19},
number = {11},
pages = {XXX--XXX},
year = {2026},
doi = {XX.XX/XXX.XX}
}