Deep neural networks (DNNs) are essential for performing advanced tasks on edge or mobile devices, yet their deployment is often hindered by severe resource constraints, including limited memory, energy, and computational power. While uniform quantization provides a straightforward approach to compress model and reduce hardware requirement, it fails to fully leverage the varying robustness across layers, and often lead to accuracy degradation or suboptimal resource usage, particularly at low bitwidths. In contrast, heterogeneous quantization, which allocates different bitwidths to individual layers, can mitigate these drawbacks. Nonetheless, current heterogeneous quantization methods either needs huge brute-force design space search or lacks the adaptability to meet different hardware conditions, such as memory size, energy budget, and latency requirement. Filling these gaps, this work introduces SigmaQuant, an adaptive layer-wise heterogeneous quantization framework designed to efficiently balance accuracy and resource usage for varied edge environments without exhaustive search. SigmaQuant allocates layer-wise bitwidths based on weight standard deviation and KL divergence, enabling adaptive quantization under real hardware constraints. This strategy efficiently balances accuracy, memory, and latency without the use of exhaustive search. We validate its practicality through ASIC integration in a shift-add-based accelerator, analyzing power, performance, and area (PPA) tradeoffs for the resulting mixed-precision models. Experimentals on CIFAR-100 and ImageNet show that SigmaQuant consistently outperforms both uniform and state-of-the-art heterogeneous quantization. At an equal size model, it achieves up to 2.0% higher accuracy; at an equal accuracy, it reduces memory by up to 40.0%. Hardware evaluation demonstrates up to 22.3% area savings and 20.6% less energy cost compared to the widely used INT8 quantization and implementation, with slight latency overhead and comparable accuracy. These results confirm the effectiveness of SigmaQuant for edge AI deployment.
