harald/zeroclaw
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 3

docs: comprehensive README rewrite — security checklist, gateway API, memory system, project structure

Browse source 
Updated README to reflect current state of ZeroClaw:
- Test count: 943 (was 657)
- Binary size: ~3.4MB
- Security checklist table (all 4 items passing)
- Gateway hardening docs (pairing, random port, tunnel)
- Gateway API reference (/health, /pair, /webhook)
- Memory system docs (FTS5 + vector + hybrid merge)
- Full config example with [gateway] and [tunnel] sections
- Test coverage table by module
- Complete project structure (62 files, 16.5k lines)
- Commands table updated with gateway --port 0
- Contributing section updated with Tunnel + Skill entries
This commit is contained in:
argenis de la rosa 2026-02-14 01:40:30 -05:00
parent c8d4ceee71
commit 0b5b49537a
1 changed files with 264 additions and 143 deletions
Download patch file Download diff file Expand all files Collapse all files

407
README.md
View file

@ -15,7 +15,7 @@
The fastest, smallest, fully autonomous AI assistant — deploy anywhere, swap anything.
```
~3MB binary · <10ms startup · 657 tests · 22+ providers · Pluggable everything
~3.4MB binary · <10ms startup · 943 tests · 22+ providers · 8 traits · Pluggable everything
```
## Quick Start
@ -37,6 +37,10 @@ cargo run --release -- agent -m "Hello, ZeroClaw!"
# Interactive mode
cargo run --release -- agent
# Start the gateway (webhook server)
cargo run --release -- gateway # default: 127.0.0.1:8080
cargo run --release -- gateway --port 0 # random port (security hardened)
# Check status
cargo run --release -- status --verbose
@ -46,6 +50,9 @@ cargo run --release -- tools list
# Test a tool directly
cargo run --release -- tools test memory_store '{"key": "lang", "content": "User prefers Rust"}'
cargo run --release -- tools test memory_recall '{"query": "Rust"}'
# List integrations
cargo run --release -- integrations list
```
> **Tip:** Run `cargo install --path .` to install `zeroclaw` globally, then use `zeroclaw` instead of `cargo run --release --`.
@ -60,30 +67,94 @@ Every subsystem is a **trait** — swap implementations with a config change, ze
| Subsystem | Trait | Ships with | Extend |
|-----------|-------|------------|--------|
| **AI Models** | `Provider` | 22+ providers (OpenRouter, Anthropic, OpenAI, Venice, Groq, Mistral, etc.) | `custom:https://your-api.com` — any OpenAI-compatible API |
| **AI Models** | `Provider` | 22+ providers (OpenRouter, Anthropic, OpenAI, Ollama, Venice, Groq, Mistral, xAI, DeepSeek, Together, Fireworks, Perplexity, Cohere, Bedrock, etc.) | `custom:https://your-api.com` — any OpenAI-compatible API |
| **Channels** | `Channel` | CLI, Telegram, Discord, Slack, iMessage, Matrix, Webhook | Any messaging API |
| **Memory** | `Memory` | SQLite (default), Markdown | Any persistence |
| **Memory** | `Memory` | SQLite with hybrid search (FTS5 + vector cosine similarity), Markdown | Any persistence backend |
| **Tools** | `Tool` | shell, file_read, file_write, memory_store, memory_recall, memory_forget | Any capability |
| **Observability** | `Observer` | Noop, Log, Multi | Prometheus, OTel |
| **Runtime** | `RuntimeAdapter` | Native (Mac/Linux/Pi) | Docker, WASM |
| **Security** | `SecurityPolicy` | Sandbox + allowlists + rate limits | — |
| **Security** | `SecurityPolicy` | Gateway pairing, sandbox, allowlists, rate limits, filesystem scoping | — |
| **Tunnel** | `Tunnel` | None, Cloudflare, Tailscale, ngrok, Custom | Any tunnel binary |
| **Heartbeat** | Engine | HEARTBEAT.md periodic tasks | — |
| **Skills** | Loader | TOML manifests + SKILL.md instructions | Community skill packs |
| **Integrations** | Registry | 50+ integrations across 9 categories | Plugin system |
### Memory System
ZeroClaw has a built-in brain. The agent automatically:
1. **Recalls** relevant memories before each prompt (context injection)
2. **Saves** conversation turns to memory (auto-save)
1. **Recalls** relevant memories before each prompt (hybrid FTS5 + vector search with context injection)
2. **Saves** conversation turns to memory (auto-save with embeddings)
3. **Manages** its own memory via tools (store/recall/forget)
Two backends — **SQLite** (default, searchable, upsert, delete) and **Markdown** (human-readable, append-only, git-friendly). Switch with one config line.
The default **SQLite** backend includes:
- **FTS5 full-text search** with BM25 ranking for keyword queries
- **Vector embeddings** (OpenAI or pluggable) with cosine similarity for semantic search
- **Hybrid merge** — weighted combination of keyword + vector results (configurable: 0.3/0.7 default)
- **Embedding cache** with LRU eviction (default: 10,000 entries)
- **Markdown-aware chunking** — splits documents by headings, respects token limits
- **LIKE fallback** when FTS5 and vector return no results
- **Upsert, delete, reindex** — full CRUD with automatic embedding refresh
### Security Architecture
**Markdown** backend available for human-readable, append-only, git-friendly storage.
ZeroClaw enforces security at **every layer** — not just the sandbox. Every message passes through authentication and rate limiting before reaching the agent.
Switch with one config line:
#### Layer 1: Channel Authentication
```toml
[memory]
backend = "sqlite" # "sqlite", "markdown", "none"
auto_save = true
embedding_provider = "openai"
vector_weight = 0.7
keyword_weight = 0.3
```
## Security
ZeroClaw enforces security at **every layer** — not just the sandbox. It passes all items from the community security checklist.
### Security Checklist
| # | Item | Status | How |
|---|------|--------|-----|
| 1 | **Gateway not publicly exposed** | ✅ | Binds `127.0.0.1` by default. Refuses `0.0.0.0` without tunnel or explicit `allow_public_bind = true`. |
| 2 | **Pairing required** | ✅ | 6-digit one-time code on startup. Exchange via `POST /pair` for bearer token. All `/webhook` requests require `Authorization: Bearer <token>`. |
| 3 | **Filesystem scoped (no /)** | ✅ | `workspace_only = true` by default. 14 system dirs + 4 sensitive dotfiles blocked. Null byte injection blocked. Symlink escape detection via canonicalization. |
| 4 | **Access via tunnel only** | ✅ | Gateway refuses public bind without active tunnel. Supports Tailscale, Cloudflare, ngrok, or any custom tunnel. |
> **Run your own nmap:** `nmap -p 1-65535 <your-host>` — ZeroClaw binds to localhost only, so nothing is exposed unless you explicitly configure a tunnel.
### Layer 1: Gateway Hardening
```bash
# Default — localhost only, pairing required
zeroclaw gateway
# Random port — OS assigns ephemeral port (49152-65535)
zeroclaw gateway --port 0
# With tunnel — public access via secure tunnel only
zeroclaw gateway # with [tunnel] configured
```
On startup, the gateway prints a **6-digit pairing code**:
```
🔐 PAIRING REQUIRED — use this one-time code:
┌──────────────┐
│ 482917 │
└──────────────┘
Send: POST /pair with header X-Pairing-Code: 482917
```
After pairing, all subsequent requests use `Authorization: Bearer zc_<token>`.
```toml
[gateway]
require_pairing = true # default: true
allow_public_bind = false # default: false — NEVER set true without tunnel
```
### Layer 2: Channel Authentication
Every channel validates the sender **before** the message reaches the agent loop:
@ -99,15 +170,26 @@ Every channel validates the sender **before** the message reaches the agent loop
> **Note:** An empty `allowed_users` list or `["*"]` allows all users (open mode). Set specific IDs for production.
#### Layer 2: Rate Limiting
### Layer 3: Rate Limiting
- **Sliding-window tracker** — counts actions within a 1-hour rolling window
- **`max_actions_per_hour`** — hard cap on tool executions (default: 20)
- **`max_cost_per_day_cents`** — daily cost ceiling (default: $5.00)
#### Layer 2.5: Agnostic Tunnel
### Layer 4: Filesystem Sandbox
Expose your gateway securely to the internet — **bring your own tunnel provider**. ZeroClaw doesn't lock you into Cloudflare or any single vendor.
- **Workspace-only mode** (default) — all paths must be relative to workspace
- **14 system directories blocked** — `/etc`, `/root`, `/home`, `/usr`, `/bin`, `/sbin`, `/lib`, `/opt`, `/boot`, `/dev`, `/proc`, `/sys`, `/var`, `/tmp`
- **4 sensitive dotfiles blocked** — `~/.ssh`, `~/.gnupg`, `~/.aws`, `~/.config`
- **Null byte injection** — blocked at the path validation layer
- **Path traversal** — `..` in any position is rejected
- **Symlink escape detection** — `is_resolved_path_allowed()` verifies canonicalized paths stay inside workspace
- **Command allowlisting** — only approved shell commands (`git`, `cargo`, `ls`, etc.)
- **Autonomy levels** — `ReadOnly`, `Supervised` (default), `Full`
### Layer 5: Secure Tunnels
Expose your gateway securely — **bring your own tunnel provider**:
| Provider | Binary | Use Case |
|----------|--------|----------|
@ -148,18 +230,74 @@ funnel = true # true = public internet, false = tailnet only
The tunnel starts automatically with `zeroclaw gateway` and prints the public URL.
#### Layer 3: Tool Sandbox
- **Workspace sandboxing** — can't escape workspace directory
- **Command allowlisting** — only approved shell commands (`git`, `cargo`, `ls`, etc.)
- **Path traversal blocking** — `..` and absolute paths blocked
- **Forbidden paths** — `/etc`, `/root`, `~/.ssh`, `~/.gnupg` always blocked
- **Autonomy levels** — `ReadOnly` (observe only), `Supervised` (acts with policy), `Full` (autonomous within bounds)
## Configuration
Config: `~/.zeroclaw/config.toml` (created by `onboard`)
```toml
api_key = "sk-..."
default_provider = "openrouter"
default_model = "anthropic/claude-sonnet-4-20250514"
default_temperature = 0.7
[memory]
backend = "sqlite" # "sqlite", "markdown", "none"
auto_save = true
embedding_provider = "openai" # "openai", "noop"
vector_weight = 0.7
keyword_weight = 0.3
[gateway]
require_pairing = true # require pairing code on first connect
allow_public_bind = false # refuse 0.0.0.0 without tunnel
[autonomy]
level = "supervised" # "readonly", "supervised", "full"
workspace_only = true
allowed_commands = ["git", "npm", "cargo", "ls", "cat", "grep"]
forbidden_paths = ["/etc", "/root", "/proc", "/sys", "~/.ssh", "~/.gnupg", "~/.aws"]
[heartbeat]
enabled = false
interval_minutes = 30
[tunnel]
provider = "none" # "none", "cloudflare", "tailscale", "ngrok", "custom"
```
## Gateway API
| Endpoint | Method | Auth | Description |
|----------|--------|------|-------------|
| `/health` | GET | None | Health check (always public, no secrets leaked) |
| `/pair` | POST | `X-Pairing-Code` header | Exchange one-time code for bearer token |
| `/webhook` | POST | `Authorization: Bearer <token>` | Send message: `{"message": "your prompt"}` |
### Random Port Mode
Use `--port 0` for OS-assigned random ephemeral ports (security hardening against port scanning):
```bash
zeroclaw gateway --port 0
# Output: 🦀 ZeroClaw Gateway listening on http://127.0.0.1:54321 (random port)
```
The actual port is printed on startup and passed to the tunnel system automatically.
## Commands
| Command | Description |
|---------|-------------|
| `onboard` | Interactive setup wizard |
| `agent -m "..."` | Single message mode |
| `agent` | Interactive chat mode |
| `gateway` | Start webhook server (default: `127.0.0.1:8080`) |
| `gateway --port 0` | Random port mode |
| `status -v` | Show full system status |
| `tools list` | List all 6 tools |
| `tools test <name> <json>` | Test a tool directly |
| `integrations list` | List all 50+ integrations |
## Documentation Index
Fetch the complete documentation index at: https://docs.openclaw.ai/llms.txt
@ -176,11 +314,10 @@ ZeroClaw assembles its own system prompt on every run. It includes:
* Tool list + short descriptions
* Skills list (only metadata; instructions are loaded on demand with `read`)
* Self-update instructions
* Workspace + bootstrap files (`AGENTS.md`, `SOUL.md`, `TOOLS.md`, `IDENTITY.md`, `USER.md`, `HEARTBEAT.md`, `BOOTSTRAP.md` when new, plus `MEMORY.md` and/or `memory.md` when present). Large files are truncated by `agents.defaults.bootstrapMaxChars` (default: 20000). `memory/*.md` files are on-demand via memory tools and are not auto-injected.
* Safety guardrails
* Workspace + bootstrap files (`AGENTS.md`, `SOUL.md`, `TOOLS.md`, `IDENTITY.md`, `USER.md`, `HEARTBEAT.md`, `BOOTSTRAP.md` when new, plus `MEMORY.md`). Large files are truncated at 20,000 characters. `memory/*.md` files are on-demand via memory tools and are not auto-injected.
* Time (UTC + user timezone)
* Reply tags + heartbeat behavior
* Runtime metadata (host/OS/model/thinking)
* Runtime metadata (host/OS/model)
### What counts in the context window
@ -189,152 +326,134 @@ Everything the model receives counts toward the context limit:
* System prompt (all sections listed above)
* Conversation history (user + assistant messages)
* Tool calls and tool results
* Attachments/transcripts (images, audio, files)
* Compaction summaries and pruning artifacts
* Memory context (injected before each prompt via hybrid recall)
* Provider wrappers or safety headers (not visible, but still counted)
### How to see current token usage
Use these in chat:
* `/status` → **emoji-rich status card** with the session model, context usage,
last response input/output tokens, and **estimated cost** (API key only).
* `/usage off|tokens|full` → appends a **per-response usage footer** to every reply.
* Persists per session (stored as `responseUsage`).
* OAuth auth **hides cost** (tokens only).
* `/usage cost` → shows a local cost summary from ZeroClaw session logs.
Other surfaces:
* **TUI/Web TUI:** `/status` + `/usage` are supported.
* **CLI:** `zeroclaw status --usage` and `zeroclaw channels list` show
provider quota windows (not per-response costs).
### Cost estimation (when shown)
Costs are estimated from your model pricing config:
```
models.providers.<provider>.models[].cost
```
These are **USD per 1M tokens** for `input`, `output`, `cacheRead`, and
`cacheWrite`. If pricing is missing, ZeroClaw shows tokens only. OAuth tokens
never show dollar cost.
### Cache TTL and pruning impact
Provider prompt caching only applies within the cache TTL window. ZeroClaw can
optionally run **cache-ttl pruning**: it prunes the session once the cache TTL
has expired, then resets the cache window so subsequent requests can re-use the
freshly cached context instead of re-caching the full history. This keeps cache
write costs lower when a session goes idle past the TTL.
Configure it in Gateway configuration and see the behavior details in
[Session pruning](/concepts/session-pruning).
Heartbeat can keep the cache **warm** across idle gaps. If your model cache TTL
is `1h`, setting the heartbeat interval just under that (e.g., `55m`) can avoid
re-caching the full prompt, reducing cache write costs.
For Anthropic API pricing, cache reads are significantly cheaper than input
tokens, while cache writes are billed at a higher multiplier. See Anthropic's
prompt caching pricing for the latest rates and TTL multipliers:
[https://docs.anthropic.com/docs/build-with-claude/prompt-caching](https://docs.anthropic.com/docs/build-with-claude/prompt-caching)
#### Example: keep 1h cache warm with heartbeat
```yaml
agents:
defaults:
model:
primary: "anthropic/claude-opus-4-6"
models:
"anthropic/claude-opus-4-6":
params:
cacheRetention: "long"
heartbeat:
every: "55m"
```
### Tips for reducing token pressure
* Use `/compact` to summarize long sessions.
* Trim large tool outputs in your workflows.
* Keep skill descriptions short (skill list is injected into the prompt).
* Prefer smaller models for verbose, exploratory work.
```toml
api_key = "sk-..."
default_provider = "openrouter"
default_model = "anthropic/claude-sonnet-4-20250514"
default_temperature = 0.7
[memory]
backend = "sqlite" # "sqlite", "markdown", "none"
auto_save = true
[autonomy]
level = "supervised" # "readonly", "supervised", "full"
workspace_only = true
allowed_commands = ["git", "npm", "cargo", "ls", "cat", "grep"]
[heartbeat]
enabled = false
interval_minutes = 30
```
## Commands
| Command | Description |
|---------|-------------|
| `onboard` | Initialize workspace and config |
| `agent -m "..."` | Single message mode |
| `agent` | Interactive chat mode |
| `status -v` | Show full system status |
| `tools list` | List all 6 tools |
| `tools test <name> <json>` | Test a tool directly |
| `gateway` | Start webhook/WebSocket server |
* Use smaller models for verbose, exploratory work
* Trim large tool outputs in your workflows
* Keep skill descriptions short (skill list is injected into the prompt)
* Adjust `auto_save` to avoid excessive memory growth
## Development
```bash
cargo build # Dev build
cargo build --release # Release build (~3MB)
cargo test # 657 tests
cargo build --release # Release build (~3.4MB)
cargo test # 943 tests
cargo clippy # Lint (0 warnings)
cargo fmt # Format
# Run the SQLite vs Markdown benchmark
cargo test --test memory_comparison -- --nocapture
```
### Test Coverage
| Module | Tests | Covers |
|--------|-------|--------|
| **Memory (SQLite)** | 100+ | FTS5, vector search, hybrid merge, embeddings, chunker, SQL injection, unicode |
| **Security (Policy)** | 50+ | Path traversal, null bytes, forbidden dirs, workspace scoping, symlink escapes |
| **Security (Pairing)** | 20+ | Code generation, token issuance, constant-time comparison, replay prevention |
| **Gateway** | 20+ | Port 0, random port allocation, header extraction, port conflicts |
| **Config** | 30+ | Serde roundtrip, backward compat, secure defaults, gateway config |
| **Providers** | 30+ | Factory, custom URLs, auth styles |
| **Tools** | 20+ | Schema validation, tool specs, serde |
| **Integrations** | 15+ | Registry completeness, status functions, categories |
| **Tunnel** | 20+ | Factory, constructors, async behavior |
## Project Structure
```
src/
├── main.rs # CLI (clap)
├── lib.rs # Library exports
├── agent/ # Agent loop + context injection
├── channels/ # Channel trait + CLI
├── config/ # TOML config schema
├── cron/ # Scheduled tasks
├── heartbeat/ # HEARTBEAT.md engine
├── memory/ # Memory trait + SQLite + Markdown
├── observability/ # Observer trait + Noop/Log/Multi
├── providers/ # Provider trait + 22 providers
├── runtime/ # RuntimeAdapter trait + Native
├── security/ # Sandbox + allowlists + autonomy
├── tools/ # Tool trait + shell/file/memory tools
└── tunnel/ # Tunnel trait + Cloudflare/Tailscale/ngrok/Custom
├── main.rs # CLI (clap) — 10 subcommands
├── lib.rs # Library exports (8 modules)
├── agent/ # Agent loop + memory context injection
│ ├── mod.rs
│ └── loop_.rs
├── channels/ # Channel trait + 7 implementations
│ ├── traits.rs # Channel trait definition
│ ├── cli.rs # Local terminal
│ ├── telegram.rs # Telegram Bot API
│ ├── discord.rs # Discord bot
│ ├── slack.rs # Slack bot
│ ├── matrix.rs # Matrix protocol
│ ├── imessage.rs # macOS iMessage
│ └── mod.rs # System prompt builder
├── config/ # TOML config schema
│ ├── schema.rs # All config structs + defaults
│ └── mod.rs
├── cron/ # Scheduled tasks
├── gateway/ # HTTP gateway (raw TCP + tokio)
│ └── mod.rs # /health, /pair, /webhook endpoints
├── heartbeat/ # Periodic task engine
│ ├── engine.rs
│ └── mod.rs
├── integrations/ # 50+ integration registry
│ ├── registry.rs # All integrations across 9 categories
│ └── mod.rs
├── memory/ # Memory trait + hybrid search engine
│ ├── traits.rs # Memory trait definition
│ ├── sqlite.rs # SQLite + FTS5 + vector embeddings
│ ├── markdown.rs # Append-only markdown
│ ├── embeddings.rs # EmbeddingProvider trait + OpenAI + Noop
│ ├── vector.rs # Cosine similarity + serialization + hybrid merge
│ ├── chunker.rs # Markdown-aware document splitting
│ └── mod.rs # Factory
├── observability/ # Observer trait + 3 backends
│ ├── traits.rs
│ ├── noop.rs
│ ├── log.rs
│ └── multi.rs
├── onboard/ # Interactive setup wizard
│ └── wizard.rs
├── providers/ # Provider trait + 22+ providers
│ ├── traits.rs # Provider trait definition
│ ├── openrouter.rs # OpenRouter (default)
│ ├── anthropic.rs # Anthropic direct
│ ├── openai.rs # OpenAI direct
│ ├── ollama.rs # Local Ollama
│ ├── compatible.rs # OpenAI-compatible adapter (18+ providers)
│ └── mod.rs # Factory
├── runtime/ # RuntimeAdapter trait + Native
│ ├── traits.rs
│ └── native.rs
├── security/ # Security policy + gateway pairing
│ ├── policy.rs # SecurityPolicy, path validation, rate limiting
│ ├── pairing.rs # PairingGuard, OTP, bearer tokens
│ └── mod.rs
├── skills/ # Skill loader (TOML manifests)
│ └── mod.rs
├── tools/ # Tool trait + 6 tools
│ ├── traits.rs # Tool trait definition
│ ├── shell.rs # Shell command execution
│ ├── file_read.rs # Sandboxed file reading
│ ├── file_write.rs # Sandboxed file writing
│ ├── memory_store.rs # Store to memory
│ ├── memory_recall.rs # Search memory
│ ├── memory_forget.rs # Delete from memory
│ └── mod.rs # Registry
└── tunnel/ # Tunnel trait + 5 implementations
├── none.rs # Local-only (default)
├── cloudflare.rs # Cloudflare Zero Trust
├── tailscale.rs # Tailscale serve/funnel
├── ngrok.rs # ngrok
├── custom.rs # Bring your own
└── mod.rs # Factory
examples/
├── custom_provider.rs
├── custom_channel.rs
├── custom_tool.rs
└── custom_memory.rs
tests/
└── memory_comparison.rs # SQLite vs Markdown benchmark
```
**62 source files · 16,500 lines of Rust · 943 tests · 0 clippy warnings**
## License
MIT — see [LICENSE](LICENSE)
@ -347,6 +466,8 @@ See [CONTRIBUTING.md](CONTRIBUTING.md). Implement a trait, submit a PR:
- New `Observer` → `src/observability/`
- New `Tool` → `src/tools/`
- New `Memory` → `src/memory/`
- New `Tunnel` → `src/tunnel/`
- New `Skill` → `~/.zeroclaw/workspace/skills/<name>/`
---