Đang tải bảng tin…
NVIDIA Developer00 bình luận7 phút đọc
Boost Inference Performance up to 15x on NVIDIA Blackwell Using DFlash Speculative Decoding
DFlash is an open source block-diffusion speculative decoding method that replaces sequential autoregressive drafting with parallel block-level token prediction. On NVIDIA Blackwell hardware, it delivers up to 15x throughput improvement over autoregressive decoding for gpt-oss-120b at high interactivity targets, and outperforms EAGLE-3 speculative decoding by 1.5x. The technique uses three key mechanisms: block-diffusion drafting, target hidden-state conditioning, and KV injection. Twenty model checkpoints covering Qwen, Llama, Gemma, Kimi K2.6, and gpt-oss families are available on Hugging Face, with integration support for TensorRT-LLM, vLLM (via the Speculators library), and SGLang requiring minimal config changes and no application refactoring.
Nguồn: https://developer.nvidia.com/blog/boost-inference-performance-up-to-15x-on-nvidia-blackwell-using-dflash-speculative-decoding. 8sync News chỉ tóm tắt và dẫn link; bản quyền nội dung thuộc tác giả và nguồn gốc.
Đề xuất cho bạn
Towards Data Science017 phút
The Hot Path Belongs to GBDTs, Agents Own the Cold Path: A Payment-Fraud Benchmark
A reproducible benchmark comparing gradient-boosted decision trees (GBDTs) vs. LLM-based scoring for payment fraud detection across three dimensions: latency, cost, and determinism. On a single CPU core, GBDTs hit p99 latency of 0.15ms vs. ~1,200ms for LLMs — well outside the 100ms ISO 8583 authorization budget. Cost-wise, GBDTs run ~$54/hour at 50K TPS vs. $16,200–$351,000 for LLM tiers. Determinism is the most critical issue for regulated environments: GBDTs return identical scores on identical inputs while LLMs produce hundreds of distinct outputs even at temperature=0. The recommended architecture keeps deterministic tree ensembles on the synchronous hot path and deploys LLM agents on the asynchronous cold path for SAR drafting, evidence gathering, and agent-as-a-judge validation before human review. All benchmark code is open-source and reproducible on a laptop.
