The Training Infrastructure Behind AI-Powered Job Search: 8X Faster Multi-Teacher Distillation

LinkedIn's engineering team details how they rebuilt the training infrastructure for AI-Powered Job Search using multi-teacher knowledge distillation. The system coordinates a relevance teacher, engagement teacher, and embedding model to train a compact 0.6B student SLM capable of serving ~22,000 requests/sec/GPU. Key optimizations include: migrating to FSDP2 with Hybrid Sharded Data Parallel (HSDP), switching to high-throughput inference engines with continuous batching and paged attention (2X faster teacher inference, 4X larger batches), and introducing offline distillation with per-shard caching on HDFS to decouple teacher inference from student training. The result is an 8X end-to-end speedup — from ~45 hours to under 5 hours — with no loss in model quality. The entire pipeline is unified under a Ray-based framework where online and offline distillation are per-teacher decisions within a single run, enabling same-day experimentation cycles.

Source: https://www.linkedin.com/blog/engineering/infrastructure/the-training-infrastructure-behind-ai-powered-job-search-eight-x-faster-multi-teacher-distillation. 8 Sync News only summarizes and links out; content copyright belongs to the authors and original sources.