English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Recent years have witnessed significant success in Gradient Boosting DecisionTrees (GBDT) for a wide range of machine learning applications. Generally, aconsensus about GBDT's training algorithms is gradients and statistics arecomputed based on high-precision floating points. In this paper, we investigatean essentially important question which has been largely ignored by theprevious literature: how many bits are needed for representing gradients intraining GBDT? To solve this mystery, we propose to quantize all thehigh-precision gradients in a very simple yet effective way in the GBDT'straining algorithm. Surprisingly, both our theoretical analysis and empiricalstudies show that the necessary precisions of gradients without hurting anyperformance can be quite low, e.g., 2 or 3 bits. With low-precision gradients,most arithmetic operations in GBDT training can be replaced by integeroperations of 8, 16, or 32 bits. Promisingly, these findings may pave the wayfor much more efficient training of GBDT from several aspects: (1) speeding upthe computation of gradient statistics in histograms; (2) compressing thecommunication cost of high-precision statistical information during distributedtraining; (3) the inspiration of utilization and development of hardwarearchitectures which well support low-precision computation for GBDT training.Benchmarked on CPUs, GPUs, and distributed clusters, we observe up to 2$\times$speedup of our simple quantization strategy compared with SOTA GBDT systems onextensive datasets, demonstrating the effectiveness and potential of thelow-precision training of GBDT. The code will be released to the officialrepository of LightGBM.

Quantized Training of Gradient Boosting Decision Trees