AI

125B LLM Under the Desk: Why Agents Left the Cloud

A Habr author explains how clients' security rules led him to a 125B model on a workstation and what changed for his AI agents.

A Habr author going by 4KAP has described how he moved his working AI agents from cloud APIs to a local model with 125 billion parameters. By his account, it runs on his own workstation, generates 22–25 tokens per second and reads the prompt (prefill) at more than 500 tokens per second. He says no data leaves the company perimeter.

The reason is information security requirements. His clients are state corporations, banks and industrial holdings. By his list, a cloud API receives task descriptions with project names, counterparties and amounts, code and configs with internal system URLs (sometimes access keys), logs, and metadata on who asked what and when. He also names cloud rate limits for dozens of parallel subagents and token bills: a single investigation agent, in his words, can burn millions of input tokens in one evening.

The key quality figure: the chosen compressed version of the model (quantization) gives the same answer as the 8-bit reference in 93.1% of cases. On tuning, he highlights only a few things. An update to the program that runs the model gave +79% generation speed. On Windows, a larger pagefile added another 48%. A small draft model that speeds up generation is, in his view, a must. He calls masking data before sending it to the cloud necessary hygiene but only a partial fix: information can be reconstructed from indirect clues, and answer quality drops on masked text.

Running locally is possible thanks to open-weight models, which anyone can download and run on their own hardware. The MoE (mixture of experts) architecture uses only part of the parameters for each token, so a large model computes faster than a dense one of the same size. The legal background the author points to: an amendment to Russia’s Federal Law 152-FZ, in force since September 1, 2015, requires the primary database of Russian citizens’ personal data to be stored in Russia. It does not ban foreign APIs by itself, but it is a separate task for clients’ lawyers. For comparison, KRAKOBIT has covered which Mac and PC can run a local 27B model.

The author predicts that within one to two years a hybrid will become the norm for development teams: the cloud for design and hard decisions, a local model for mass task waves and anything sensitive. One more argument for a local setup, in his words: tuned workflows survive lost subscriptions, withdrawn deployments and a lost internet connection.

Source: Хабр