harald/nixcfg
1
0
Fork 0
Code Issues Pull requests 5 Projects Releases Packages Wiki Activity Actions

fix(cache-warm): push the lock commit even when a target failed to build #36

Closed
hermes wants to merge 3 commits from hermes/cache-warm-fixes into main AGit
pull from: hermes/cache-warm-fixes
merge into: harald:main
Conversation 2 Commits 3 Files changed 5 +1244 -45
hermes commented 2026-09-06 19:13:07 +02:00
Contributor
Copy link

Gating the flake.lock push on every target building turned one broken target
into a fleet-wide input freeze: from 2026-08-30 to 2026-09-06 the freshly
updated lock lived only in amd's local clone while origin/main stayed on
b4cc9dd, so every other host's nixos-upgrade built against a lock that was
days stale -- the exact opposite of what cache-warm is for.

Push first, then exit with the accumulated build rc: the run still goes red
in the journal (visible), the lock still ships (unblocked). A rejected push
now counts towards rc as well, since it genuinely failed.

Gating the flake.lock push on every target building turned one broken target into a fleet-wide input freeze: from 2026-08-30 to 2026-09-06 the freshly updated lock lived only in amd's local clone while origin/main stayed on b4cc9dd, so every other host's nixos-upgrade built against a lock that was days stale -- the exact opposite of what cache-warm is for. Push first, then exit with the accumulated build rc: the run still goes red in the journal (visible), the lock still ships (unblocked). A rejected push now counts towards rc as well, since it genuinely failed. 
The overlay appended a fetchurl patch pointing at a GitHub *compare* range
(dd1ea52...9921e01, llama.cpp PR #26856). Compare ranges are not immutable:
the PR head was force-pushed and the PR has since been closed unmerged, so
the endpoint 9921e01 no longer exists and the URL now serves an HTML 404
page. Worse, even while it served content, the nightly nixpkgs-unstable roll
(0.2.0 -> 0.3.0, 2026-08-31) drifted the base under the tests/test-backend-
ops.cpp hunk: 'Hunk #1 FAILED at 9568' broke cache-warm on amd every night
since, blocking the halo closure.

Vendored as llama-cpp-bf16-fattn-tile.patch, regenerated against the current
pin (v0.3.0) from the still-immutable pull/26856.diff via
regen-bf16-fattn-tile.sh. The test-backend-ops.cpp hunks are dropped on
purpose: the file is never built (doCheck off) and churns daily upstream --
it was pure breakage surface. Verified to apply cleanly against both v0.3.0
(the current nixpkgs-unstable pin, 17de0b97) and b10408 (previous pin).
Upstream moved the MAX_PLATFORM_OUTPUT cap logic out of the else-branch of
the splits_long_messages gate into a _cap_oversized_output helper that
early-returns for chunking adapters (hermes-agent main; the nightly
cache-warm input bump pulled it in on 2026-09-05, failing
nextcloud-talk-platform's checkPhase and blocking the hermes closure since).

The behaviour the contract cares about is unchanged -- chunking adapters
never reach the cap, non-chunking ones always do -- but the old test
hard-coded the else-branch *shape* and saw a rename where there was none.

The gate check now walks whatever form delivery.py uses: locate the gate,
require the flag-true branch never references the cap, and require the cap
on the flag-false path (else-branch, or -- only when the true branch
exits -- after the gate). A fall-through true branch with post-gate
truncation still fails, because that would cut chunking adapters' output.

Verified: 140 tests pass against the pinned rev 4209d371 (else-branch form)
and against upstream main (early-return form); mutation tests (gate
deleted/inverted/renamed, cap dropped, cap moved past a fall-through gate)
all still fail as intended.
fix(cache-warm): push the lock commit even when a target failed to build
All checks were successful
nix flake check / flake-check (pull_request) Successful in 3m57s
Details
a6bd13a556 
Gating the flake.lock push on every target building turned one broken target
into a fleet-wide input freeze: from 2026-08-30 to 2026-09-06 the freshly
updated lock lived only in amd's local clone while origin/main stayed on
b4cc9dd, so every other host's nixos-upgrade built against a lock that was
days stale -- the exact opposite of what cache-warm is for.

Push first, then exit with the accumulated build rc: the run still goes red
in the journal (visible), the lock still ships (unblocked). A rejected push
now counts towards rc as well, since it genuinely failed.
harald commented 2026-09-14 13:37:39 +02:00
Owner
Copy link

no, I want it to fail here

no, I want it to fail here
harald closed this pull request 2026-09-14 13:37:39 +02:00
hermes commented 2026-10-11 18:54:39 +02:00
Author
Contributor
Copy link

Closing decision confirmed, leaving this for the record so neither a human nor cache-warm-adjacent automation re-rolls this fix: push-before-exit was rejected deliberately (Harald: "no, I want it to fail here", 14.09). The exit-before-push gate stays.

The Oct lock freeze this PR was opened for is now being fixed at the cause instead: cache-warm failed nightly on 06.-10.10 (journal: exit-code failures 23:0x UTC all five nights) because (a) the simple-nixos-mailserver gitlab: input hit an anonymous-API 403 (GitLab Cloudflare challenge on /repository/commits) during the weekly nix flake update, so the run died before pushing a new lock, and (b) from 09.10. the x1 target was also dead (our flatpak overlay patch failed to apply against flatpak 1.18.4). Both fixed in PR #61: mailserver follows the branch as an API-archive tarball (numeric project id, served anonymously), the overlay applies via postPatch sed anchored on the line itself. With those gone the nightly run should go green and the push gate stops mattering.

Closing decision confirmed, leaving this for the record so neither a human nor cache-warm-adjacent automation re-rolls this fix: push-before-exit was rejected deliberately (Harald: "no, I want it to fail here", 14.09). The exit-before-push gate stays. The Oct lock freeze this PR was opened for is now being fixed at the cause instead: cache-warm failed nightly on 06.-10.10 (journal: exit-code failures 23:0x UTC all five nights) because (a) the simple-nixos-mailserver gitlab: input hit an anonymous-API 403 (GitLab Cloudflare challenge on /repository/commits) during the weekly `nix flake update`, so the run died before pushing a new lock, and (b) from 09.10. the x1 target was also dead (our flatpak overlay patch failed to apply against flatpak 1.18.4). Both fixed in PR #61: mailserver follows the branch as an API-archive tarball (numeric project id, served anonymously), the overlay applies via postPatch sed anchored on the line itself. With those gone the nightly run should go green and the push gate stops mattering.
All checks were successful
nix flake check / flake-check (pull_request) Successful in 3m57s
Details

Pull request closed

Please reopen this pull request to perform a merge.
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
harald/nixcfg!36
No description provided.