harald/nixcfg
1
0
Fork 0
Code Issues Pull requests 4 Projects Releases Packages Wiki Activity Actions

do-not-merge: feat(llm): offload the PLE table, resize the cap, cap the pools #21

Open
hermes wants to merge 1 commit from hermes/ple-offload-day-v2 into main AGit
pull from: hermes/ple-offload-day-v2
merge into: harald:main
Conversation 0 Commits 1 Files changed 2 +71 -31
hermes commented 2026-08-29 02:32:14 +02:00
Contributor
Copy link

Turns the documented offload-day change into a deployable commit, still
gated on hardware: amd has NOT been rebooted since 2026-08-26 (journal
boots + kernel "Memory: 62962476K" on 2026-08-29) so the 2x64 GiB
Kingston sticks are not populated yet. Deploy only after the sticks are
in, trained at speed, and memtest passed, in a restart window the opencode
traffic can survive.

  • --no-ple-offload-embedding -> --ple-offload-embedding: moves the
    48-52 GiB pinned n-gram table out of VRAM (~24-26 GiB/board freed).
  • --memory=48g -> 64g (swap 80g): pinned bytes are charged to the
    container's cgroup (measured 2026-08-27), so the cap must sit above
    48-52 GiB pinned + ~6.3 GiB anon; 16 GiB swap allowance kept.
  • --max-total-tokens 300000 + --max-mamba-cache-size 24 in the SAME
    commit: without hard caps the KV pool instantly re-eats the freed
    VRAM. Measured safe set ~3.6 GiB/board, above observed peaks (131k
    tokens, 16 mamba slots).

Verified: nix eval on nixosConfigurations.amd renders exactly these
flags. Deploy verification (per card t_044d1243): engine up, decode
~88 tok/s @18k, free VRAM per board clearly above the 3.7/4.4 GiB
baseline, qwen-tts containers untouched. Activating the restart also
switches on the already-committed --sleep-on-idle.

Turns the documented offload-day change into a deployable commit, still gated on hardware: amd has NOT been rebooted since 2026-08-26 (journal boots + kernel "Memory: 62962476K" on 2026-08-29) so the 2x64 GiB Kingston sticks are not populated yet. Deploy only after the sticks are in, trained at speed, and memtest passed, in a restart window the opencode traffic can survive. - --no-ple-offload-embedding -> --ple-offload-embedding: moves the 48-52 GiB pinned n-gram table out of VRAM (~24-26 GiB/board freed). - --memory=48g -> 64g (swap 80g): pinned bytes are charged to the container's cgroup (measured 2026-08-27), so the cap must sit above 48-52 GiB pinned + ~6.3 GiB anon; 16 GiB swap allowance kept. - --max-total-tokens 300000 + --max-mamba-cache-size 24 in the SAME commit: without hard caps the KV pool instantly re-eats the freed VRAM. Measured safe set ~3.6 GiB/board, above observed peaks (131k tokens, 16 mamba slots). Verified: nix eval on nixosConfigurations.amd renders exactly these flags. Deploy verification (per card t_044d1243): engine up, decode ~88 tok/s @18k, free VRAM per board clearly above the 3.7/4.4 GiB baseline, qwen-tts containers untouched. Activating the restart also switches on the already-committed --sleep-on-idle.
do-not-merge: feat(llm): offload the PLE table, resize the cap, cap the pools
Some checks failed
nix flake check / flake-check (pull_request) Failing after 3m49s
Details
3ca92f6080 
Turns the documented offload-day change into a deployable commit, still
gated on hardware: amd has NOT been rebooted since 2026-08-26 (journal
boots + kernel "Memory: 62962476K" on 2026-08-29) so the 2x64 GiB
Kingston sticks are not populated yet. Deploy only after the sticks are
in, trained at speed, and memtest passed, in a restart window the opencode
traffic can survive.

- --no-ple-offload-embedding -> --ple-offload-embedding: moves the
  48-52 GiB pinned n-gram table out of VRAM (~24-26 GiB/board freed).
- --memory=48g -> 64g (swap 80g): pinned bytes are charged to the
  container's cgroup (measured 2026-08-27), so the cap must sit above
  48-52 GiB pinned + ~6.3 GiB anon; 16 GiB swap allowance kept.
- --max-total-tokens 300000 + --max-mamba-cache-size 24 in the SAME
  commit: without hard caps the KV pool instantly re-eats the freed
  VRAM. Measured safe set ~3.6 GiB/board, above observed peaks (131k
  tokens, 16 mamba slots).

Verified: nix eval on nixosConfigurations.amd renders exactly these
flags. Deploy verification (per card t_044d1243): engine up, decode
~88 tok/s @18k, free VRAM per board clearly above the 3.7/4.4 GiB
baseline, qwen-tts containers untouched. Activating the restart also
switches on the already-committed --sleep-on-idle.
Some checks failed
nix flake check / flake-check (pull_request) Failing after 3m49s
Details
This pull request can be merged automatically.
This branch is out-of-date with the base branch
You are not authorized to merge this pull request.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin +refs/pull/21/head:hermes/ple-offload-day-v2
git switch hermes/ple-offload-day-v2
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
harald/nixcfg!21
No description provided.