English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Although prior art has demonstrated negligible accuracy drop in sub-bytequantization -- where weights and/or activations are represented by less than 8bits -- popular SIMD instructions of CPUs do not natively support thesedatatypes. While recent methods, such as ULPPACK, are already using sub-bytequantization on general-purpose CPUs with vector units, they leave out severalempty bits between the sub-byte values in memory and in vector registers toavoid overflow to the neighbours during the operations. This results in memoryfootprint and bandwidth-usage inefficiencies and suboptimal performance. Inthis paper, we present memory layouts for storing, and mechanisms forprocessing sub-byte (4-, 2-, or 1-bit) models that utilize all the bits in thememory as well as in the vector registers for the actual data. We providecompute kernels for the proposed layout for the GEMV (GEneral Matrix-Vectormultiplication) operations between weights and activations of differentdatatypes (e.g., 8-bit activations and 4-bit weights). For evaluation, weextended the TFLite package and added our methods to it, then ran the models onthe cycle-accurate gem5 simulator to compare detailed memory and CPU cycles ofeach method. We compare against nine other methods that are actively used inproduction including GEMLOWP, Ruy, XNNPack, and ULPPACK. Furthermore, weexplore the effect of different input and output sizes of deep learning layerson the performance of our proposed method. Experimental results show 0.96-2.1xspeedup for small sizes and 1.2-6.7x speedup for mid to large sizes. Applyingour proposal to a real-world speech recognition model, Mozilla DeepSpeech, weproved that our method achieves 1.56-2.11x end-to-end speedup compared to thestate-of-the-art, depending on the bit-width employed.

FullPack: Full Vector Utilization for Sub-Byte Quantized Inference on General Purpose CPUs