Petals – Run LLMs at home, BitTorrent-style

Petals is an open-source project that lets you run large language models (Llama 3.1 up to 405B, Mixtral 8x22B, Falcon 180B, BLOOM 176B) on consumer-grade GPUs or Google Colab using a BitTorrent-style distributed network. Each participant loads a portion of the model and joins a peer network serving the remaining layers. Single-batch inference reaches up to 6 tokens/sec for Llama 2 70B, sufficient for chatbots and interactive apps. Unlike standard LLM APIs, Petals exposes full PyTorch and Hugging Face Transformers flexibility, enabling custom fine-tuning, sampling methods, and access to hidden states. It is part of the BigScience research workshop.

Source: https://petals.dev/. 8sync News only summarizes and links out; content copyright belongs to the authors and original sources.