viiwork

LLM inference for a fleet of machines. One node per machine serves every model on it behind one OpenAI-compatible API. Nodes find each other and form a mesh; any node is an entry point, and a request goes to a free slot wherever one exists.

I had 50 Radeon VII cards sitting in servers in my mother-in-law's garage (who doesn't?) and wanted to do something useful with them. viiwork turns a pile of aging-but-capable GPUs into a practical inference cluster.

That fleet is still the reference deployment, but viiwork runs on anything from one gaming GPU to racks of cards — AMD, NVIDIA or an Apple Silicon Mac, all in one mesh. It doesn't do inference itself: it drives existing engines — llama.cpp, vLLM and FreeToken — and turns them into one fleet. Each node's GPU resources are available from any endpoint.

You need GGUF model files and one of:

Full requirements: setup.

v=vX.Y.Z                     # newest from https://github.com/janit/viiwork/releases
os=linux_amd64               # or linux_arm64, darwin_arm64
base=https://github.com/janit/viiwork/releases/download/$v
curl -fLO "$base/viiwork_${v}_${os}.tar.gz" -fLO "$base/SHA256SUMS"
sha256sum --check --ignore-missing SHA256SUMS    # Mac: shasum -a 256 --check ...
tar xzf "viiwork_${v}_${os}.tar.gz" && cd "viiwork_${v}_${os}"

sudo ./viiwork init          # Mac: ./viiwork init

The wizard finds your GPUs and models, proposes a layout, and shows every file before writing anything. → setup · verifying signatures

On gfx906, build the image from the same release and start it:

git clone https://github.com/janit/viiwork && cd viiwork && git checkout "$v"
make docker
sudo cp configs/docker-compose.v2.example.yaml /etc/viiwork/docker-compose.yaml
# set the models mount in that file, then:
sudo docker compose -f /etc/viiwork/docker-compose.yaml -p viiwork up -d

→ when it writes the config only

Add the next machine: print a join code on any node, run the wizard on the new one, paste the code. The code is the mesh secret.

sudo sh -c 'set -a; . /etc/viiwork/mesh.env; /usr/local/bin/viiwork join-code'   # Linux
~/.local/bin/viiwork join-code                                                    # Mac

Test it:

curl http://localhost:8086/v1/models
curl http://localhost:8086/v1/chat/completions -H "Content-Type: application/json" \
  -d '{"model":"<name from /v1/models>","messages":[{"role":"user","content":"Hello"}]}'

Day to day: viiwork top, viiwork update (updating), viiwork stop / start, viiwork uninstall (keeps your models).

Skip the wizard: copy viiwork.yaml.example to /etc/viiwork/viiwork.yaml, put a VIIWORK_MESH_SECRET in /etc/viiwork/mesh.env (or set mesh.open: true), then build and start the image as above. → configuration

One file per machine. Each model has a name, its GPUs and its slots:

models:
  - name: Qwen3.8-27B
    engine: llamacpp
    path: /models/Qwen3.8-27B-UD-Q4_K_XL.gguf
    gpus: [0, 1]
    gpus_per_backend: 2      # one backend across both cards
    context: 49152           # tokens PER SLOT
    parallel: 2

• A GPU belongs to one model
• SIGHUP reloads: added models start, removed drain, changed restart
• Validated at startup and reload, errors name the field

→ docs/configuration.md

• Ports: 8086 tcp (API, dashboards), 7946 tcp+udp (gossip)
• Discovery: Tailscale (default) or LAN mDNS; optional seeds
• Secured or open: set VIIWORK_MESH_SECRET, or declare mesh.open: true
• Routing: local free slot → member with most free slots → queue (20 s)

→ docs/mesh.md

→ docs/dashboards.md

All three can run on one node. Adding an engine is one package plus one line. → docs/adding-an-engine.md

• gfx906 rule: a dense model that doesn't fit one card costs ~3× throughput
• FreeToken exception: big sparse MoE models on a single card, at the cost of load time

What the Radeon VII hosts serve today:

Throughput, validated configs, failed bring-ups, tuning rules → docs/models.md

viiwork authenticates nothing. Reachability is the authorization; run it on a tailnet. A CORS allowlist (default: your own tailnet) stops browser pages driving the fleet. → docs/security.md

→ docs/api-integration.md

Releases also ship signed static binaries for linux/amd64, linux/arm64 and darwin/arm64. → BUILDS.md · docs/releases.md

make build        # binary
make test         # unit tests
make docker       # ROCm image
go test -tags=integration ./mesh/... ./internal/proxy/ ./internal/alias/ ./internal/node/   # multi-node, no GPU
go test -bench=. -benchmem ./internal/proxy ./internal/route                                 # compare allocs, not time

Go 1.27.1. Dependencies: yaml.v3, memberlist, mdns, x/term.