harald/nixcfg
1
0
Fork 0
Code Issues Pull requests 4 Projects Releases Packages Wiki Activity Actions

fix(journal-upload): stop transient upload blips from failing nixos-upgrade #32

Open
hermes wants to merge 1 commit from hermes/journal-upload-blip-fix into main AGit
pull from: hermes/journal-upload-blip-fix
merge into: harald:main
Conversation 0 Commits 1 Files changed 1 +89
hermes commented 2026-08-30 18:55:39 +02:00
Contributor
Copy link

The daily nixos-upgrade.timer on amd reported failure on 2026-08-29 and
2026-08-30 even though the build and the switch completed. Cause: at the
exact second of activation, sgx's systemd-journal-remote (microhttpd
"Application reported internal error, closing connection") closed the
uploader's long-lived POST; systemd-journal-upload exited 1, was picked
up in "auto-restart" substate by switch-to-configuration, whose failed-
unit scan counts a service in auto-restart with ExecMainStatus != 0 as
failed -> "warning: the following units failed: systemd-journal-upload"
-> activation exits 4 -> nixos-upgrade inherits the failure.

SuccessExitStatus=1 does not fix this: verified against a live systemd,
it maps Result to "success" but leaves ExecMainStatus at 1, which is the
property the activation report reads. So the exit code itself must become
0. Run the uploader through a wrapper that exits 0 when exit 1 is
accompanied by the expected "Upload to failed" transport error, and
forwards signals so unit stops don't stall until TimeoutStopSec. Other
exit codes are passed through unchanged.

Same fix applies fleet-wide via metacfg.services.journalUpload (amd, mx,
attic, x1, halo, hermes all use the uploader); the sink-side blips are
observed on both amd and mx.

The daily nixos-upgrade.timer on amd reported failure on 2026-08-29 and 2026-08-30 even though the build and the switch completed. Cause: at the exact second of activation, sgx's systemd-journal-remote (microhttpd "Application reported internal error, closing connection") closed the uploader's long-lived POST; systemd-journal-upload exited 1, was picked up in "auto-restart" substate by switch-to-configuration, whose failed- unit scan counts a service in auto-restart with ExecMainStatus != 0 as failed -> "warning: the following units failed: systemd-journal-upload" -> activation exits 4 -> nixos-upgrade inherits the failure. SuccessExitStatus=1 does not fix this: verified against a live systemd, it maps Result to "success" but leaves ExecMainStatus at 1, which is the property the activation report reads. So the exit code itself must become 0. Run the uploader through a wrapper that exits 0 when exit 1 is accompanied by the expected "Upload to <url> failed" transport error, and forwards signals so unit stops don't stall until TimeoutStopSec. Other exit codes are passed through unchanged. Same fix applies fleet-wide via metacfg.services.journalUpload (amd, mx, attic, x1, halo, hermes all use the uploader); the sink-side blips are observed on both amd and mx.
fix(journal-upload): stop transient upload blips from failing nixos-upgrade
All checks were successful
nix flake check / flake-check (pull_request) Successful in 3m52s
Details
495f355c3a 
The daily nixos-upgrade.timer on amd reported failure on 2026-08-29 and
2026-08-30 even though the build and the switch completed. Cause: at the
exact second of activation, sgx's systemd-journal-remote (microhttpd
"Application reported internal error, closing connection") closed the
uploader's long-lived POST; systemd-journal-upload exited 1, was picked
up in "auto-restart" substate by switch-to-configuration, whose failed-
unit scan counts a service in auto-restart with ExecMainStatus != 0 as
failed -> "warning: the following units failed: systemd-journal-upload"
-> activation exits 4 -> nixos-upgrade inherits the failure.

SuccessExitStatus=1 does not fix this: verified against a live systemd,
it maps Result to "success" but leaves ExecMainStatus at 1, which is the
property the activation report reads. So the exit code itself must become
0. Run the uploader through a wrapper that exits 0 when exit 1 is
accompanied by the expected "Upload to <url> failed" transport error, and
forwards signals so unit stops don't stall until TimeoutStopSec. Other
exit codes are passed through unchanged.

Same fix applies fleet-wide via metacfg.services.journalUpload (amd, mx,
attic, x1, halo, hermes all use the uploader); the sink-side blips are
observed on both amd and mx.
All checks were successful
nix flake check / flake-check (pull_request) Successful in 3m52s
Details
This pull request can be merged automatically.
This branch is out-of-date with the base branch
You are not authorized to merge this pull request.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin +refs/pull/32/head:hermes/journal-upload-blip-fix
git switch hermes/journal-upload-blip-fix
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
harald/nixcfg!32
No description provided.