Loading feed…
Towards Data Science0 Hot0 comments24 min read
Can an LLM Forget the Right Things?
A project called vla-edge-backend implements a hand-written CUDA inference runtime designed for robotics vision-language-action workloads, where a hard 33ms control-loop deadline and fixed VRAM budget are treated as first-class constraints. It uses an admission controller (exponential moving average cost estimation) that refuses to start reasoning chunks it can't finish in time, a semantic KV cache eviction policy that evicts the most redundant frame by cosine similarity instead of the oldest, and a lock-free double buffer so a 60Hz camera never blocks on slower reasoning. The entire Qwen2.5-Coder-1.5B-Instruct transformer (RMSNorm, RoPE, grouped-query attention, SwiGLU) is written in raw CUDA with no cuBLAS or libtorch, validated against HuggingFace output at ≥0.999 cosine similarity. The author is explicit this is an architecture demonstration built on a cloud Hopper GPU, not a benchmarked edge deployment, and the transformer itself currently runs about 100x too slow for the deadline it targets.
Source: https://towardsdatascience.com/can-an-llm-forget-the-right-things. 8 Sync News only summarizes and links out; content copyright belongs to the authors and original sources.
Recommended for you
Bình luận
Đang tải bình luận…
