Đang tải bảng tin…
Medium00 bình luận10 phút đọc
mlx-lm vs oMLX: I Was Wrong About the Winner
A hands-on benchmark comparison of mlx-lm (Apple's reference LLM runtime) and oMLX (a serving layer built on top of it) on a Mac Studio M4 Max with 64GB unified memory. The author's original thesis — that oMLX's tiered KV cache would dominate on repeated long prompts — was disproven: both tools tie on warm prefill (3.4s vs 3.6s) and decode throughput (~22-25 tok/s) because mlx-lm already caches prompts in memory. The real differentiator emerged only after testing server restarts: oMLX's SSD-backed cold cache survives process restarts, recovering to near-warm speed (7.2s) while mlx-lm must recompute from scratch (~17s). The post also details four critical benchmarking pitfalls: mlx-lm's prompt caching, its buffered (non-streaming) output, oMLX's lazy model loading, and oMLX's persistent SSD cache contaminating cold-start measurements. Practical guidance: use mlx-lm for single-user or scripting workflows; use oMLX when operational resilience, multi-model pooling, or cache persistence across restarts matters.
Nguồn: https://medium.com/macoclock/mlx-lm-vs-omlx-i-was-wrong-about-the-winner-8f36be328069. 8sync News chỉ tóm tắt và dẫn link; bản quyền nội dung thuộc tác giả và nguồn gốc.
Đề xuất cho bạn
Towards Data Science017 phút
The Hot Path Belongs to GBDTs, Agents Own the Cold Path: A Payment-Fraud Benchmark
A reproducible benchmark comparing gradient-boosted decision trees (GBDTs) vs. LLM-based scoring for payment fraud detection across three dimensions: latency, cost, and determinism. On a single CPU core, GBDTs hit p99 latency of 0.15ms vs. ~1,200ms for LLMs — well outside the 100ms ISO 8583 authorization budget. Cost-wise, GBDTs run ~$54/hour at 50K TPS vs. $16,200–$351,000 for LLM tiers. Determinism is the most critical issue for regulated environments: GBDTs return identical scores on identical inputs while LLMs produce hundreds of distinct outputs even at temperature=0. The recommended architecture keeps deterministic tree ensembles on the synchronous hot path and deploys LLM agents on the asynchronous cold path for SAR drafting, evidence gathering, and agent-as-a-judge validation before human review. All benchmark code is open-source and reproducible on a laptop.
