zeroclaw/docs/ci-cost-optimization.md

# CI Cost Optimization — February 2026

> **Date:** 2026-02-18
> **Status:** Implemented
> **Impact:** ~60-65% reduction in estimated monthly GitHub Actions billable minutes

---

## Executive Summary

On February 17, 2026, the ZeroClaw repository consumed **400+ workflow runs** in a single day, totaling an estimated **398 billable minutes** (~6.6 hours). At this rate, monthly costs were projected at **~200 hours/month** (~12,000 billable minutes). This document describes the analysis performed, optimizations implemented, and the revised CI/CD architecture.

---

## Analysis Methodology

A Python script (`scripts/ci/fetch_actions_data.py`) was created to programmatically fetch and analyze all GitHub Actions workflow runs from the GitHub API for February 17, 2026. The script:

1. Fetched all completed workflow runs for the date via the GitHub REST API
2. Grouped runs by workflow name
3. Sampled job-level timing (up to 3 runs per workflow) to compute per-job durations
4. Extrapolated to estimate total billable minutes per workflow

### Raw Data Summary (February 17, 2026)

| Rank | Workflow | Runs/Day | Est. Minutes/Day | Primary Trigger |
|------|----------|----------|-------------------|-----------------|
| 1 | Rust Package Security Audit | 57 | 102 | Every PR + push |
| 2 | CI Run | 57 | 70 | Every PR + push |
| 3 | Performance Benchmarks | 15 | 63 | Every push to main |
| 4 | Docker | 20 | 63 | PR + push |
| 5 | PR Labeler | 69 | 20 | Every PR event |
| 6 | Feature Matrix | 3 | 19 | Push to main |
| 7 | Integration / E2E Tests | 15 | 17 | Every push to main |
| 8 | Workflow Sanity | 31 | 16 | Push + PR |
| 9 | Copilot Code Review | 6 | 14 | Dynamic |
| 10 | PR Intake Checks | 70 | 7 | Every PR event |
| 11 | PR Auto Responder | 47 | 4 | PR + issues |
| | **Total** | **400+** | **~398** | |

### Key Findings

- **15 pushes to main in ~2 hours** on Feb 17, each triggering 6-8 parallel workflows
- **Security Audit** was the single largest cost driver (102 min/day) with no path filtering
- **PR Auto Responder** had an **81% failure rate** (38/47 runs failing) — wasting runner time
- **CodeQL** runs twice daily (not captured in Feb 17 data since it's schedule-only) — adding ~3.5h/week
- **Benchmarks** ran on every push to main (15x in one day) despite being regression-focused
- **Dependabot** could generate up to 11 PRs/week, each triggering the full CI cascade

---

## Changes Implemented

### 1. Security Audit — Path Filters Added

**File:** `.github/workflows/sec-audit.yml`

**Before:** Ran on every PR and every push to main, regardless of what files changed.

**After:** Only runs when dependency or source files change:
- `Cargo.toml`, `Cargo.lock`, `src/**`, `crates/**`, `deny.toml`

**Weekly schedule retained** as a safety net for advisory database updates.

**Estimated savings:** ~60-70% of security audit runs eliminated (~30-35 hours/month)

### 2. Performance Benchmarks — Moved to Weekly Schedule

**File:** `.github/workflows/test-benchmarks.yml`

**Before:** Ran on every push to main (15x/day on Feb 17).

**After:** Runs weekly (Monday 3am UTC) + on-demand via `workflow_dispatch`.

**Artifact retention** reduced from 30 days to 7 days to lower storage costs.

**Rationale:** Benchmark regressions don't need per-commit detection. Weekly cadence catches regressions within one development cycle.

**Estimated savings:** ~90% reduction (~28 hours/month)

### 3. Docker PR Smoke Builds — Tightened Path Filters

**File:** `.github/workflows/pub-docker-img.yml`

**Before:** PR smoke builds triggered on any change to `src/**`, `crates/**`, `benches/**`, `firmware/**`, etc.

**After:** PR smoke builds only trigger on Docker-specific files:
- `Dockerfile`, `.dockerignore`, `docker-compose.yml`, `rust-toolchain.toml`, `dev/config.template.toml`, `.github/workflows/pub-docker-img.yml`

**Push-to-main triggers unchanged** — production Docker images still rebuild on source changes.

**Estimated savings:** ~40-50% fewer Docker smoke builds (~12-15 hours/month)

### 4. CodeQL — Reduced from Twice-Daily to Weekly

**File:** `.github/workflows/sec-codeql.yml`

**Before:** Ran twice daily at 6am and 6pm UTC (14 runs/week), each performing a full `cargo build --workspace --all-targets`.

**After:** Runs weekly (Monday 6am UTC) + on-demand.

**Rationale:** CodeQL for Rust is still maturing. Weekly scans are standard practice for security-focused projects. On-demand dispatch available for urgent scans.

**Estimated savings:** ~12 hours/month

### 5. CI Run — Merged Lint Jobs + Dropped `--release` Build

**File:** `.github/workflows/ci-run.yml`

**Changes:**
1. **Merged `lint` and `lint-strict-delta` into a single job** — Previously these were two separate parallel jobs, each requiring a full runner spin-up, Rust toolchain install, and cache restore. Now they run sequentially in one job.
2. **Dropped `--release` flag from smoke build** — `cargo build --release` is 2-3x slower than debug due to optimizations. For a smoke check validating compilation, debug mode is equivalent.

**Estimated savings:** ~1 runner job per CI invocation + faster build times

### 6. Feature Matrix — Weekly-Only + Check-Only

**File:** `.github/workflows/feature-matrix.yml`

**Before:** Ran on every push to main touching `src/**` (3x on Feb 17) with 4 matrix entries, each running both `cargo check` AND `cargo test`.

**After:**
1. **Removed push trigger** — Now weekly-only (Monday 4:30am UTC) + on-demand
2. **Removed `cargo test`** — Only runs `cargo check --locked` per feature combination. Tests are already covered by the main CI Run workflow.

**Estimated savings:** ~50-75% of feature matrix compute eliminated

### 7. Lightweight Jobs Moved to `ubuntu-latest`

**Files affected:**
- `.github/workflows/pr-check-stale.yml`
- `.github/workflows/pr-check-status.yml`
- `.github/workflows/pr-auto-response.yml`
- `.github/workflows/pr-intake-checks.yml`
- `.github/workflows/pr-labeler.yml`
- `.github/workflows/sync-contributors.yml`

**Before:** All jobs used `blacksmith-2vcpu-ubuntu-2404` runners, even for lightweight API-only operations (labeling, stale checks, greetings).

**After:** Moved to `ubuntu-latest` (GitHub-hosted runners). These jobs only make API calls and run JavaScript scripts — they don't need Rust toolchains or specialized runners.

**Additional change:** `pr-check-status.yml` schedule reduced from every 12 hours to once daily (8:15am UTC).

### 8. Dependabot — Reduced Frequency and PR Limits

**File:** `.github/dependabot.yml`

**Before:**
- Cargo: weekly, 5 open PRs max
- GitHub Actions: weekly, 3 open PRs max
- Docker: weekly, 3 open PRs max
- Total: up to 11 Dependabot PRs/week, each triggering full CI

**After:**
- Cargo: **monthly**, 3 open PRs max, all deps grouped into single PR
- GitHub Actions: **monthly**, 1 open PR max, all grouped
- Docker: **monthly**, 1 open PR max, all grouped
- Total: up to 5 Dependabot PRs/month

**Rationale:** Each Dependabot PR triggers the full CI pipeline. Reducing from weekly to monthly and grouping updates into fewer PRs dramatically reduces CI cascade costs while still keeping dependencies current.

---

## Known Issues to Investigate

### PR Auto Responder — 81% Failure Rate

The `pr-auto-response.yml` workflow had 38 failures out of 47 runs on Feb 17. The `contributor-tier-issues` job fires on every issue `labeled`/`unlabeled` event, even when the label is not contributor-tier related. While the JavaScript handler exits early for non-tier labels, the runner still spins up and checks out the repository.

**Recommendations for further investigation:**
1. Add more specific event filtering at the workflow level to reduce unnecessary runs
2. Check if the failures are related to GitHub API rate limiting on the search endpoint
3. Consider whether `continue-on-error: true` should be added to non-critical jobs

---

## Revised Workflow Architecture

### Workflow Frequency Overview

| Workflow | Trigger | Runner |
|----------|---------|--------|
| **CI Run** | Push to main + PR | Blacksmith |
| **Sec Audit** | Push/PR (path-filtered) + weekly schedule | Blacksmith |
| **Sec CodeQL** | Weekly schedule | Blacksmith |
| **Test E2E** | Push to main | Blacksmith |
| **Test Benchmarks** | Weekly schedule | Blacksmith |
| **Test Fuzz** | Weekly schedule | Blacksmith |
| **Feature Matrix** | Weekly schedule | Blacksmith |
| **Docker Publish** | Push to main (broad paths) + PR (Docker-only paths) | Blacksmith |
| **Release** | Tag push only | GitHub-hosted |
| **Workflow Sanity** | Push/PR (workflow paths only) | Blacksmith |
| **Label Policy** | Push/PR (policy paths only) | Blacksmith |
| **PR Labeler** | PR events | **ubuntu-latest** |
| **PR Intake Checks** | PR events | **ubuntu-latest** |
| **PR Auto Responder** | PR + issue events | **ubuntu-latest** |
| **PR Check Stale** | Daily schedule | **ubuntu-latest** |
| **PR Check Status** | Daily schedule | **ubuntu-latest** |
| **Sync Contributors** | Weekly schedule | **ubuntu-latest** |

### Weekly Schedule Summary

| Day | Time (UTC) | Workflow |
|-----|-----------|----------|
| Monday | 03:00 | Test Benchmarks |
| Monday | 04:30 | Feature Matrix |
| Monday | 06:00 | Sec Audit (schedule) |
| Monday | 06:00 | Sec CodeQL |
| Sunday | 00:00 | Sync Contributors |
| Sunday | 02:00 | Test Fuzz |
| Daily | 02:20 | PR Check Stale |
| Daily | 08:15 | PR Check Status |

### CI Run Job Dependency Graph

```
changes ──┬── lint (Format + Clippy + Strict Delta)
           │     └── test
           ├── build (Smoke, debug mode)
           ├── docs-only (fast path)
           ├── non-rust (fast path)
           ├── docs-quality
           └── workflow-owner-approval

All above ──── ci-required (final gate)
```

### Push-to-Main Trigger Cascade

When code is pushed to `main`, the following workflows trigger:

1. **CI Run** — Always (change-detection gates individual jobs)
2. **Sec Audit** — Only if `Cargo.toml`, `Cargo.lock`, `src/**`, `crates/**`, or `deny.toml` changed
3. **Test E2E** — Always
4. **Docker Publish** — Only if broad source paths changed
5. **Workflow Sanity** — Only if workflow files changed

**No longer triggered on push:**
- ~~Performance Benchmarks~~ → Weekly only
- ~~Feature Matrix~~ → Weekly only

---

## Estimated Impact

| Metric | Before | After | Savings |
|--------|--------|-------|---------|
| Daily workflow runs | 400+ | ~150-180 | ~55-60% |
| Daily billable minutes | ~400 min | ~120-150 min | ~60-65% |
| Monthly billable hours | ~200 hours | ~60-75 hours | ~60-65% |
| Dependabot PRs/month | ~44 | ~5 | ~89% |
| CodeQL runs/week | 14 | 1 | ~93% |
| Benchmark runs/day | ~15 | 0 (weekly: ~1) | ~99% |

---

## Rollback Strategy

Each change is isolated to a single workflow file. To rollback any specific optimization:

1. **Revert the specific file** using `git checkout <commit>^ -- <file-path>`
2. Changes are backward-compatible — no downstream code or configuration depends on the CI schedule/trigger changes
3. All workflows retain `workflow_dispatch` triggers for manual invocation when needed

---

## Validation Checklist

- [ ] Verify CI Run workflow passes on next PR with Rust changes
- [ ] Verify Security Audit skips docs-only PRs
- [ ] Verify Docker smoke build only triggers on Dockerfile changes in PRs
- [ ] Verify weekly schedules fire correctly (check after first Monday)
- [ ] Monitor PR Auto Responder failure rate after switching to `ubuntu-latest`
- [ ] Verify Dependabot respects new monthly schedule and limits

---

## Files Modified

| File | Change Summary |
|------|---------------|
| `.github/workflows/sec-audit.yml` | Added path filters for push and PR triggers |
| `.github/workflows/test-benchmarks.yml` | Changed to weekly schedule; reduced artifact retention to 7 days |
| `.github/workflows/pub-docker-img.yml` | Tightened PR path filters to Docker-specific files |
| `.github/workflows/sec-codeql.yml` | Changed from twice-daily to weekly schedule |
| `.github/workflows/ci-run.yml` | Merged lint jobs; dropped `--release` from smoke build |
| `.github/workflows/feature-matrix.yml` | Removed push trigger; removed `cargo test` step |
| `.github/workflows/pr-check-stale.yml` | Switched to `ubuntu-latest` |
| `.github/workflows/pr-check-status.yml` | Switched to `ubuntu-latest`; reduced to daily schedule |
| `.github/workflows/pr-auto-response.yml` | Switched all jobs to `ubuntu-latest` |
| `.github/workflows/pr-intake-checks.yml` | Switched to `ubuntu-latest` |
| `.github/workflows/pr-labeler.yml` | Switched to `ubuntu-latest` |
| `.github/workflows/sync-contributors.yml` | Switched to `ubuntu-latest` |
| `.github/dependabot.yml` | Changed to monthly schedule; reduced PR limits; grouped all deps |
| `scripts/ci/fetch_actions_data.py` | New: cost analysis script for GitHub Actions runs |