diff --git a/README.md b/README.md index 4f06263..9427ff1 100644 --- a/README.md +++ b/README.md @@ -15,7 +15,7 @@ The fastest, smallest, fully autonomous AI assistant — deploy anywhere, swap anything. ``` -~3MB binary · <10ms startup · 657 tests · 22+ providers · Pluggable everything +~3.4MB binary · <10ms startup · 943 tests · 22+ providers · 8 traits · Pluggable everything ``` ## Quick Start @@ -37,6 +37,10 @@ cargo run --release -- agent -m "Hello, ZeroClaw!" # Interactive mode cargo run --release -- agent +# Start the gateway (webhook server) +cargo run --release -- gateway # default: 127.0.0.1:8080 +cargo run --release -- gateway --port 0 # random port (security hardened) + # Check status cargo run --release -- status --verbose @@ -46,6 +50,9 @@ cargo run --release -- tools list # Test a tool directly cargo run --release -- tools test memory_store '{"key": "lang", "content": "User prefers Rust"}' cargo run --release -- tools test memory_recall '{"query": "Rust"}' + +# List integrations +cargo run --release -- integrations list ``` > **Tip:** Run `cargo install --path .` to install `zeroclaw` globally, then use `zeroclaw` instead of `cargo run --release --`. @@ -60,30 +67,94 @@ Every subsystem is a **trait** — swap implementations with a config change, ze | Subsystem | Trait | Ships with | Extend | |-----------|-------|------------|--------| -| **AI Models** | `Provider` | 22+ providers (OpenRouter, Anthropic, OpenAI, Venice, Groq, Mistral, etc.) | `custom:https://your-api.com` — any OpenAI-compatible API | +| **AI Models** | `Provider` | 22+ providers (OpenRouter, Anthropic, OpenAI, Ollama, Venice, Groq, Mistral, xAI, DeepSeek, Together, Fireworks, Perplexity, Cohere, Bedrock, etc.) | `custom:https://your-api.com` — any OpenAI-compatible API | | **Channels** | `Channel` | CLI, Telegram, Discord, Slack, iMessage, Matrix, Webhook | Any messaging API | -| **Memory** | `Memory` | SQLite (default), Markdown | Any persistence | +| **Memory** | `Memory` | SQLite with hybrid search (FTS5 + vector cosine similarity), Markdown | Any persistence backend | | **Tools** | `Tool` | shell, file_read, file_write, memory_store, memory_recall, memory_forget | Any capability | | **Observability** | `Observer` | Noop, Log, Multi | Prometheus, OTel | | **Runtime** | `RuntimeAdapter` | Native (Mac/Linux/Pi) | Docker, WASM | -| **Security** | `SecurityPolicy` | Sandbox + allowlists + rate limits | — | +| **Security** | `SecurityPolicy` | Gateway pairing, sandbox, allowlists, rate limits, filesystem scoping | — | | **Tunnel** | `Tunnel` | None, Cloudflare, Tailscale, ngrok, Custom | Any tunnel binary | | **Heartbeat** | Engine | HEARTBEAT.md periodic tasks | — | +| **Skills** | Loader | TOML manifests + SKILL.md instructions | Community skill packs | +| **Integrations** | Registry | 50+ integrations across 9 categories | Plugin system | ### Memory System ZeroClaw has a built-in brain. The agent automatically: -1. **Recalls** relevant memories before each prompt (context injection) -2. **Saves** conversation turns to memory (auto-save) +1. **Recalls** relevant memories before each prompt (hybrid FTS5 + vector search with context injection) +2. **Saves** conversation turns to memory (auto-save with embeddings) 3. **Manages** its own memory via tools (store/recall/forget) -Two backends — **SQLite** (default, searchable, upsert, delete) and **Markdown** (human-readable, append-only, git-friendly). Switch with one config line. +The default **SQLite** backend includes: +- **FTS5 full-text search** with BM25 ranking for keyword queries +- **Vector embeddings** (OpenAI or pluggable) with cosine similarity for semantic search +- **Hybrid merge** — weighted combination of keyword + vector results (configurable: 0.3/0.7 default) +- **Embedding cache** with LRU eviction (default: 10,000 entries) +- **Markdown-aware chunking** — splits documents by headings, respects token limits +- **LIKE fallback** when FTS5 and vector return no results +- **Upsert, delete, reindex** — full CRUD with automatic embedding refresh -### Security Architecture +**Markdown** backend available for human-readable, append-only, git-friendly storage. -ZeroClaw enforces security at **every layer** — not just the sandbox. Every message passes through authentication and rate limiting before reaching the agent. +Switch with one config line: -#### Layer 1: Channel Authentication +```toml +[memory] +backend = "sqlite" # "sqlite", "markdown", "none" +auto_save = true +embedding_provider = "openai" +vector_weight = 0.7 +keyword_weight = 0.3 +``` + +## Security + +ZeroClaw enforces security at **every layer** — not just the sandbox. It passes all items from the community security checklist. + +### Security Checklist + +| # | Item | Status | How | +|---|------|--------|-----| +| 1 | **Gateway not publicly exposed** | ✅ | Binds `127.0.0.1` by default. Refuses `0.0.0.0` without tunnel or explicit `allow_public_bind = true`. | +| 2 | **Pairing required** | ✅ | 6-digit one-time code on startup. Exchange via `POST /pair` for bearer token. All `/webhook` requests require `Authorization: Bearer `. | +| 3 | **Filesystem scoped (no /)** | ✅ | `workspace_only = true` by default. 14 system dirs + 4 sensitive dotfiles blocked. Null byte injection blocked. Symlink escape detection via canonicalization. | +| 4 | **Access via tunnel only** | ✅ | Gateway refuses public bind without active tunnel. Supports Tailscale, Cloudflare, ngrok, or any custom tunnel. | + +> **Run your own nmap:** `nmap -p 1-65535 ` — ZeroClaw binds to localhost only, so nothing is exposed unless you explicitly configure a tunnel. + +### Layer 1: Gateway Hardening + +```bash +# Default — localhost only, pairing required +zeroclaw gateway + +# Random port — OS assigns ephemeral port (49152-65535) +zeroclaw gateway --port 0 + +# With tunnel — public access via secure tunnel only +zeroclaw gateway # with [tunnel] configured +``` + +On startup, the gateway prints a **6-digit pairing code**: + +``` +🔐 PAIRING REQUIRED — use this one-time code: + ┌──────────────┐ + │ 482917 │ + └──────────────┘ + Send: POST /pair with header X-Pairing-Code: 482917 +``` + +After pairing, all subsequent requests use `Authorization: Bearer zc_`. + +```toml +[gateway] +require_pairing = true # default: true +allow_public_bind = false # default: false — NEVER set true without tunnel +``` + +### Layer 2: Channel Authentication Every channel validates the sender **before** the message reaches the agent loop: @@ -99,15 +170,26 @@ Every channel validates the sender **before** the message reaches the agent loop > **Note:** An empty `allowed_users` list or `["*"]` allows all users (open mode). Set specific IDs for production. -#### Layer 2: Rate Limiting +### Layer 3: Rate Limiting - **Sliding-window tracker** — counts actions within a 1-hour rolling window - **`max_actions_per_hour`** — hard cap on tool executions (default: 20) - **`max_cost_per_day_cents`** — daily cost ceiling (default: $5.00) -#### Layer 2.5: Agnostic Tunnel +### Layer 4: Filesystem Sandbox -Expose your gateway securely to the internet — **bring your own tunnel provider**. ZeroClaw doesn't lock you into Cloudflare or any single vendor. +- **Workspace-only mode** (default) — all paths must be relative to workspace +- **14 system directories blocked** — `/etc`, `/root`, `/home`, `/usr`, `/bin`, `/sbin`, `/lib`, `/opt`, `/boot`, `/dev`, `/proc`, `/sys`, `/var`, `/tmp` +- **4 sensitive dotfiles blocked** — `~/.ssh`, `~/.gnupg`, `~/.aws`, `~/.config` +- **Null byte injection** — blocked at the path validation layer +- **Path traversal** — `..` in any position is rejected +- **Symlink escape detection** — `is_resolved_path_allowed()` verifies canonicalized paths stay inside workspace +- **Command allowlisting** — only approved shell commands (`git`, `cargo`, `ls`, etc.) +- **Autonomy levels** — `ReadOnly`, `Supervised` (default), `Full` + +### Layer 5: Secure Tunnels + +Expose your gateway securely — **bring your own tunnel provider**: | Provider | Binary | Use Case | |----------|--------|----------| @@ -148,18 +230,74 @@ funnel = true # true = public internet, false = tailnet only The tunnel starts automatically with `zeroclaw gateway` and prints the public URL. -#### Layer 3: Tool Sandbox - -- **Workspace sandboxing** — can't escape workspace directory -- **Command allowlisting** — only approved shell commands (`git`, `cargo`, `ls`, etc.) -- **Path traversal blocking** — `..` and absolute paths blocked -- **Forbidden paths** — `/etc`, `/root`, `~/.ssh`, `~/.gnupg` always blocked -- **Autonomy levels** — `ReadOnly` (observe only), `Supervised` (acts with policy), `Full` (autonomous within bounds) - ## Configuration Config: `~/.zeroclaw/config.toml` (created by `onboard`) +```toml +api_key = "sk-..." +default_provider = "openrouter" +default_model = "anthropic/claude-sonnet-4-20250514" +default_temperature = 0.7 + +[memory] +backend = "sqlite" # "sqlite", "markdown", "none" +auto_save = true +embedding_provider = "openai" # "openai", "noop" +vector_weight = 0.7 +keyword_weight = 0.3 + +[gateway] +require_pairing = true # require pairing code on first connect +allow_public_bind = false # refuse 0.0.0.0 without tunnel + +[autonomy] +level = "supervised" # "readonly", "supervised", "full" +workspace_only = true +allowed_commands = ["git", "npm", "cargo", "ls", "cat", "grep"] +forbidden_paths = ["/etc", "/root", "/proc", "/sys", "~/.ssh", "~/.gnupg", "~/.aws"] + +[heartbeat] +enabled = false +interval_minutes = 30 + +[tunnel] +provider = "none" # "none", "cloudflare", "tailscale", "ngrok", "custom" +``` + +## Gateway API + +| Endpoint | Method | Auth | Description | +|----------|--------|------|-------------| +| `/health` | GET | None | Health check (always public, no secrets leaked) | +| `/pair` | POST | `X-Pairing-Code` header | Exchange one-time code for bearer token | +| `/webhook` | POST | `Authorization: Bearer ` | Send message: `{"message": "your prompt"}` | + +### Random Port Mode + +Use `--port 0` for OS-assigned random ephemeral ports (security hardening against port scanning): + +```bash +zeroclaw gateway --port 0 +# Output: 🦀 ZeroClaw Gateway listening on http://127.0.0.1:54321 (random port) +``` + +The actual port is printed on startup and passed to the tunnel system automatically. + +## Commands + +| Command | Description | +|---------|-------------| +| `onboard` | Interactive setup wizard | +| `agent -m "..."` | Single message mode | +| `agent` | Interactive chat mode | +| `gateway` | Start webhook server (default: `127.0.0.1:8080`) | +| `gateway --port 0` | Random port mode | +| `status -v` | Show full system status | +| `tools list` | List all 6 tools | +| `tools test ` | Test a tool directly | +| `integrations list` | List all 50+ integrations | + ## Documentation Index Fetch the complete documentation index at: https://docs.openclaw.ai/llms.txt @@ -176,11 +314,10 @@ ZeroClaw assembles its own system prompt on every run. It includes: * Tool list + short descriptions * Skills list (only metadata; instructions are loaded on demand with `read`) -* Self-update instructions -* Workspace + bootstrap files (`AGENTS.md`, `SOUL.md`, `TOOLS.md`, `IDENTITY.md`, `USER.md`, `HEARTBEAT.md`, `BOOTSTRAP.md` when new, plus `MEMORY.md` and/or `memory.md` when present). Large files are truncated by `agents.defaults.bootstrapMaxChars` (default: 20000). `memory/*.md` files are on-demand via memory tools and are not auto-injected. +* Safety guardrails +* Workspace + bootstrap files (`AGENTS.md`, `SOUL.md`, `TOOLS.md`, `IDENTITY.md`, `USER.md`, `HEARTBEAT.md`, `BOOTSTRAP.md` when new, plus `MEMORY.md`). Large files are truncated at 20,000 characters. `memory/*.md` files are on-demand via memory tools and are not auto-injected. * Time (UTC + user timezone) -* Reply tags + heartbeat behavior -* Runtime metadata (host/OS/model/thinking) +* Runtime metadata (host/OS/model) ### What counts in the context window @@ -189,152 +326,134 @@ Everything the model receives counts toward the context limit: * System prompt (all sections listed above) * Conversation history (user + assistant messages) * Tool calls and tool results -* Attachments/transcripts (images, audio, files) -* Compaction summaries and pruning artifacts +* Memory context (injected before each prompt via hybrid recall) * Provider wrappers or safety headers (not visible, but still counted) -### How to see current token usage - -Use these in chat: - -* `/status` → **emoji-rich status card** with the session model, context usage, - last response input/output tokens, and **estimated cost** (API key only). -* `/usage off|tokens|full` → appends a **per-response usage footer** to every reply. - * Persists per session (stored as `responseUsage`). - * OAuth auth **hides cost** (tokens only). -* `/usage cost` → shows a local cost summary from ZeroClaw session logs. - -Other surfaces: - -* **TUI/Web TUI:** `/status` + `/usage` are supported. -* **CLI:** `zeroclaw status --usage` and `zeroclaw channels list` show - provider quota windows (not per-response costs). - -### Cost estimation (when shown) - -Costs are estimated from your model pricing config: - -``` -models.providers..models[].cost -``` - -These are **USD per 1M tokens** for `input`, `output`, `cacheRead`, and -`cacheWrite`. If pricing is missing, ZeroClaw shows tokens only. OAuth tokens -never show dollar cost. - -### Cache TTL and pruning impact - -Provider prompt caching only applies within the cache TTL window. ZeroClaw can -optionally run **cache-ttl pruning**: it prunes the session once the cache TTL -has expired, then resets the cache window so subsequent requests can re-use the -freshly cached context instead of re-caching the full history. This keeps cache -write costs lower when a session goes idle past the TTL. - -Configure it in Gateway configuration and see the behavior details in -[Session pruning](/concepts/session-pruning). - -Heartbeat can keep the cache **warm** across idle gaps. If your model cache TTL -is `1h`, setting the heartbeat interval just under that (e.g., `55m`) can avoid -re-caching the full prompt, reducing cache write costs. - -For Anthropic API pricing, cache reads are significantly cheaper than input -tokens, while cache writes are billed at a higher multiplier. See Anthropic's -prompt caching pricing for the latest rates and TTL multipliers: -[https://docs.anthropic.com/docs/build-with-claude/prompt-caching](https://docs.anthropic.com/docs/build-with-claude/prompt-caching) - -#### Example: keep 1h cache warm with heartbeat - -```yaml -agents: - defaults: - model: - primary: "anthropic/claude-opus-4-6" - models: - "anthropic/claude-opus-4-6": - params: - cacheRetention: "long" - heartbeat: - every: "55m" -``` - ### Tips for reducing token pressure -* Use `/compact` to summarize long sessions. -* Trim large tool outputs in your workflows. -* Keep skill descriptions short (skill list is injected into the prompt). -* Prefer smaller models for verbose, exploratory work. - -```toml -api_key = "sk-..." -default_provider = "openrouter" -default_model = "anthropic/claude-sonnet-4-20250514" -default_temperature = 0.7 - -[memory] -backend = "sqlite" # "sqlite", "markdown", "none" -auto_save = true - -[autonomy] -level = "supervised" # "readonly", "supervised", "full" -workspace_only = true -allowed_commands = ["git", "npm", "cargo", "ls", "cat", "grep"] - -[heartbeat] -enabled = false -interval_minutes = 30 -``` - -## Commands - -| Command | Description | -|---------|-------------| -| `onboard` | Initialize workspace and config | -| `agent -m "..."` | Single message mode | -| `agent` | Interactive chat mode | -| `status -v` | Show full system status | -| `tools list` | List all 6 tools | -| `tools test ` | Test a tool directly | -| `gateway` | Start webhook/WebSocket server | +* Use smaller models for verbose, exploratory work +* Trim large tool outputs in your workflows +* Keep skill descriptions short (skill list is injected into the prompt) +* Adjust `auto_save` to avoid excessive memory growth ## Development ```bash cargo build # Dev build -cargo build --release # Release build (~3MB) -cargo test # 657 tests +cargo build --release # Release build (~3.4MB) +cargo test # 943 tests cargo clippy # Lint (0 warnings) +cargo fmt # Format # Run the SQLite vs Markdown benchmark cargo test --test memory_comparison -- --nocapture ``` +### Test Coverage + +| Module | Tests | Covers | +|--------|-------|--------| +| **Memory (SQLite)** | 100+ | FTS5, vector search, hybrid merge, embeddings, chunker, SQL injection, unicode | +| **Security (Policy)** | 50+ | Path traversal, null bytes, forbidden dirs, workspace scoping, symlink escapes | +| **Security (Pairing)** | 20+ | Code generation, token issuance, constant-time comparison, replay prevention | +| **Gateway** | 20+ | Port 0, random port allocation, header extraction, port conflicts | +| **Config** | 30+ | Serde roundtrip, backward compat, secure defaults, gateway config | +| **Providers** | 30+ | Factory, custom URLs, auth styles | +| **Tools** | 20+ | Schema validation, tool specs, serde | +| **Integrations** | 15+ | Registry completeness, status functions, categories | +| **Tunnel** | 20+ | Factory, constructors, async behavior | + ## Project Structure ``` src/ -├── main.rs # CLI (clap) -├── lib.rs # Library exports -├── agent/ # Agent loop + context injection -├── channels/ # Channel trait + CLI -├── config/ # TOML config schema -├── cron/ # Scheduled tasks -├── heartbeat/ # HEARTBEAT.md engine -├── memory/ # Memory trait + SQLite + Markdown -├── observability/ # Observer trait + Noop/Log/Multi -├── providers/ # Provider trait + 22 providers -├── runtime/ # RuntimeAdapter trait + Native -├── security/ # Sandbox + allowlists + autonomy -├── tools/ # Tool trait + shell/file/memory tools -└── tunnel/ # Tunnel trait + Cloudflare/Tailscale/ngrok/Custom +├── main.rs # CLI (clap) — 10 subcommands +├── lib.rs # Library exports (8 modules) +├── agent/ # Agent loop + memory context injection +│ ├── mod.rs +│ └── loop_.rs +├── channels/ # Channel trait + 7 implementations +│ ├── traits.rs # Channel trait definition +│ ├── cli.rs # Local terminal +│ ├── telegram.rs # Telegram Bot API +│ ├── discord.rs # Discord bot +│ ├── slack.rs # Slack bot +│ ├── matrix.rs # Matrix protocol +│ ├── imessage.rs # macOS iMessage +│ └── mod.rs # System prompt builder +├── config/ # TOML config schema +│ ├── schema.rs # All config structs + defaults +│ └── mod.rs +├── cron/ # Scheduled tasks +├── gateway/ # HTTP gateway (raw TCP + tokio) +│ └── mod.rs # /health, /pair, /webhook endpoints +├── heartbeat/ # Periodic task engine +│ ├── engine.rs +│ └── mod.rs +├── integrations/ # 50+ integration registry +│ ├── registry.rs # All integrations across 9 categories +│ └── mod.rs +├── memory/ # Memory trait + hybrid search engine +│ ├── traits.rs # Memory trait definition +│ ├── sqlite.rs # SQLite + FTS5 + vector embeddings +│ ├── markdown.rs # Append-only markdown +│ ├── embeddings.rs # EmbeddingProvider trait + OpenAI + Noop +│ ├── vector.rs # Cosine similarity + serialization + hybrid merge +│ ├── chunker.rs # Markdown-aware document splitting +│ └── mod.rs # Factory +├── observability/ # Observer trait + 3 backends +│ ├── traits.rs +│ ├── noop.rs +│ ├── log.rs +│ └── multi.rs +├── onboard/ # Interactive setup wizard +│ └── wizard.rs +├── providers/ # Provider trait + 22+ providers +│ ├── traits.rs # Provider trait definition +│ ├── openrouter.rs # OpenRouter (default) +│ ├── anthropic.rs # Anthropic direct +│ ├── openai.rs # OpenAI direct +│ ├── ollama.rs # Local Ollama +│ ├── compatible.rs # OpenAI-compatible adapter (18+ providers) +│ └── mod.rs # Factory +├── runtime/ # RuntimeAdapter trait + Native +│ ├── traits.rs +│ └── native.rs +├── security/ # Security policy + gateway pairing +│ ├── policy.rs # SecurityPolicy, path validation, rate limiting +│ ├── pairing.rs # PairingGuard, OTP, bearer tokens +│ └── mod.rs +├── skills/ # Skill loader (TOML manifests) +│ └── mod.rs +├── tools/ # Tool trait + 6 tools +│ ├── traits.rs # Tool trait definition +│ ├── shell.rs # Shell command execution +│ ├── file_read.rs # Sandboxed file reading +│ ├── file_write.rs # Sandboxed file writing +│ ├── memory_store.rs # Store to memory +│ ├── memory_recall.rs # Search memory +│ ├── memory_forget.rs # Delete from memory +│ └── mod.rs # Registry +└── tunnel/ # Tunnel trait + 5 implementations + ├── none.rs # Local-only (default) + ├── cloudflare.rs # Cloudflare Zero Trust + ├── tailscale.rs # Tailscale serve/funnel + ├── ngrok.rs # ngrok + ├── custom.rs # Bring your own + └── mod.rs # Factory + examples/ ├── custom_provider.rs ├── custom_channel.rs ├── custom_tool.rs └── custom_memory.rs + tests/ └── memory_comparison.rs # SQLite vs Markdown benchmark ``` +**62 source files · 16,500 lines of Rust · 943 tests · 0 clippy warnings** + ## License MIT — see [LICENSE](LICENSE) @@ -347,6 +466,8 @@ See [CONTRIBUTING.md](CONTRIBUTING.md). Implement a trait, submit a PR: - New `Observer` → `src/observability/` - New `Tool` → `src/tools/` - New `Memory` → `src/memory/` +- New `Tunnel` → `src/tunnel/` +- New `Skill` → `~/.zeroclaw/workspace/skills//` ---