A laptop-local chatbot that answers domain-scoped questions by retrieving relevant markdown "skills" and calling a configured tenant API.
Built as a learning vehicle for: FastAPI + LangGraph ReAct + Chroma RAG + OAuth client_credentials + SQLite session memory + multi-layer guardrails.
The
skills/directory is gitignored. This repo ships the framework — you bring your own troubleshooting docs asskills/<name>/SKILL.mdfiles.Each
SKILL.mdneeds YAML frontmatter:--- name: my-skill description: One-line summary used for retrieval and shown to the LLM. --- # My skill body — markdown anything below.Without skills, the agent will refuse every question (no context to ground replies). See Adding skills below.
| Layer | Purpose | Cost | Latency |
|---|---|---|---|
| L0 — Intent classifier | Cheap LLM call labels message greeting / in_scope / off_topic |
~$0.00001 | ~300ms |
| L1 — RAG threshold gate | Embed query, find nearest skill, refuse if cosine score < 0.15 | $0 local + ~$0.00001 embed | ~50ms |
| L2 — Main agent | LangGraph ReAct: LLM + tool calls, system prompt injects matched skill body | ~$0.001-0.003 | ~3s |
| Tools | call_tenant_api(method, path, body) — read-only HTTP to your tenant w/ OAuth bearer |
tenant-dependent | varies |
| Memory | SQLite-backed chat history via SQLChatMessageHistory, survives restarts |
local | none |
# 1. Install dependencies (uv is a fast Python package manager)
uv sync
# 2. Configure
cp .env.example .env
# Edit .env — set OPENAI_API_KEY (required), TENANT_BASE_URL + OAUTH_CLIENT_ID/SECRET (for the tool), optional LangSmith
# 3. Drop your skills under skills/<name>/SKILL.md (see warning above)
# 4. Build the retrieval index from your skills
uv run python scripts/build_index.py
# Prints: "Indexed N chunks → chroma_db/"
# 5. Run the server
uv run uvicorn app.main:app --port 8000 --reloadOpen http://127.0.0.1:8000. Type a question. Done.
You type a question
│
▼
┌─────────────────────────────────────────┐
│ L0 — Intent classifier (LLM) │ ← cheap, fast
│ greeting | in_scope | off_topic │
└─────────────────────────────────────────┘
│ greeting → canned hi (no main LLM)
│ off_topic → canned refusal (no main LLM)
│ in_scope ↓
┌─────────────────────────────────────────┐
│ L1 — RAG retrieval (Chroma) │ ← local, no LLM
│ Find nearest skill, score it 0..1 │
└─────────────────────────────────────────┘
│ score < 0.15 → canned refusal (no main LLM)
│ score ≥ 0.15 ↓
┌─────────────────────────────────────────┐
│ L2 — Main agent (LangGraph ReAct) │ ← LLM + tools
│ System prompt = base rules + skill body│
│ May call call_tenant_api(...) │
└─────────────────────────────────────────┘
│
▼
Reply rendered in browser + persisted to SQLite for next turn
.env keys:
| Key | Required | Purpose |
|---|---|---|
OPENAI_API_KEY |
yes | Embeddings (Chroma) — even when chat goes to OpenRouter |
LLM_MODEL |
yes | Chat model name. gpt-4o-mini (OpenAI) or google/gemini-2.0-flash-001 (OpenRouter) |
LLM_BASE_URL |
no | If set, ChatOpenAI routes here (e.g. https://openrouter.ai/api/v1). Empty = OpenAI direct. |
LLM_API_KEY |
conditional | Key for LLM_BASE_URL. Falls back to OPENAI_API_KEY if empty. |
TENANT_BASE_URL |
for Phase 1D tool | Base URL of the tenant API (e.g. https://my-tenant.example.com) |
OAUTH_CLIENT_ID |
for Phase 1D tool | OAuth client_credentials grant client id |
OAUTH_CLIENT_SECRET |
for Phase 1D tool | OAuth client_credentials grant client secret |
LANGSMITH_* |
no | Tracing to LangSmith — recommended for debugging |
RETRIEVAL_TOP_K |
no | Skills to fetch per turn (default 3) |
RETRIEVAL_THRESHOLD |
no | Cosine score gate, 0..1 (default 0.15) |
Restart the server after editing .env — --reload only watches Python files.
- Create
skills/<your-skill-name>/SKILL.md - Add YAML frontmatter (
name,description) + markdown body - Rebuild the index:
rm -rf chroma_db && uv run python scripts/build_index.py - Restart the server
The loader walks skills/*/SKILL.md. Sub-folders inside a skill (e.g. skills/foo/references/*.md) are NOT indexed — only top-level SKILL.md. Markdown-header chunking is planned for a later phase.
---
name: my-skill
description: One-sentence summary. Used for retrieval and shown to the LLM.
---
# My Skill
Markdown body. The LLM sees the full body when this skill is retrieved.
If your skill instructs hitting an API endpoint, write it like a curl command:
```bash
GET /api/users?limit=10
POST /api/search/indexsearch body={"query": ...}The LLM will translate these into call_tenant_api(method, path, body) calls.
---
## File layout
. ├── app/ │ ├── config.py ← pydantic-settings, reads .env │ ├── main.py ← FastAPI /chat endpoint + logs │ ├── tenant_client.py ← OAuth token manager wrapper │ ├── graph/ │ │ └── agent.py ← prompts, classifier, ReAct agent build │ ├── skills/ │ │ ├── loader.py ← parses SKILL.md frontmatter │ │ └── indexer.py ← Chroma vector index wrapper │ ├── tools/ │ │ ├── registry.py ← TOOL_REGISTRY = [...] │ │ └── tenant_api.py ← call_tenant_api @tool │ └── static/index.html ← chat UI (vanilla JS) ├── scripts/build_index.py ← offline CLI to embed skills/ ├── skills/ ← (gitignored) your SKILL.md files ├── chroma_db/ ← (gitignored) vector index, rebuilt from skills/ ├── agent.db ← (gitignored) SQLite chat memory ├── .env ← (gitignored) your secrets ├── pyproject.toml └── README.md
Gitignored paths are local artifacts. The repo is the framework, not the data.
---
## Cost reference
Numbers below assume `gpt-4o-mini` or `gemini-2.0-flash` as chat model.
| Action | Approx cost |
|--------|-------------|
| Build retrieval index (one-time, depends on skill count) | ~$0.0003 per ~5K-char skill |
| Off-topic query (caught by L0/L1, no main LLM) | ~$0.00001 |
| In-scope query (full agent run, 1 tool call) | ~$0.001-0.005 |
| 100 typical chat turns | ~$0.20-0.50 |
Real numbers visible per turn in terminal logs (`[reply] ... usage={...}`) and in LangSmith if configured.
---
## Glossary
- **LLM** — Large Language Model. Generates the reply. e.g. `gpt-4o-mini`, `gemini-2.0-flash`.
- **Embedding** — Text → 1536-dimensional vector capturing meaning. Used for similarity search.
- **Chroma** — Local vector database. Stores skill embeddings, returns nearest matches for a query.
- **RAG** — Retrieval-Augmented Generation. Look up a relevant doc, inject it into the prompt, ask the LLM.
- **LangGraph** — Library wrapping LLM calls in a graph. We use `create_react_agent` (LLM → tool → LLM loop).
- **LangSmith** — Observability for LLM apps. Per-turn trace, cost, latency.
- **OAuth client_credentials** — Service-to-service auth. Client ID + secret → bearer token. Token represents the service, not a person.
- **Session** — One ongoing chat. Identified by UUID in browser localStorage, persisted in `agent.db`.
---
## Storage
| Thing | Location | Survives restart |
|-------|----------|------------------|
| Chat history | `agent.db` (SQLite) | yes |
| Skill embeddings | `chroma_db/` (HNSW + SQLite) | yes (rebuild when skills change) |
| Session ID | Browser `localStorage["debug-agent.sid"]` | yes (until cleared) |
| `.env` secrets | `.env` file (gitignored) | yes |
| LLM traces | LangSmith cloud (if enabled) | yes |
---
## Security
- Server binds `127.0.0.1` only — not reachable from network
- `.env` gitignored — secrets never committed
- `call_tenant_api` tool is **read-only**: GET allowed; POST only on a small allowlist of search/audit endpoints; PUT/PATCH/DELETE rejected by validator
- OAuth bearer never logged
- Tool output truncated to 16K chars to control prompt cost / leakage
- `agent.db` may contain customer-typed content — don't share without redaction
Known gaps (deferred):
- No PII / secret redaction in LLM traces (whatever the user types lands in LangSmith)
- No per-user impersonation — single service-account identity
- No audit table separate from chat history
- No rate limit on `/chat`
- No tests yet
---
## Roadmap
| Phase | Adds | Status |
|-------|------|--------|
| 1A | Walking-skeleton chat UI | done |
| 1B | Skill RAG + 3-layer guardrail | done |
| 1C | SQLite session memory | done |
| 1D | First real tenant API tool + OAuth | done |
| 1E | SSE streaming + second tool | next |
| 1F | Tests + audit table + secret redaction | |
| 2+ | Markdown-header chunking, write tools, long-term memory, OTel | |
---
## FAQ
**Q: I sent a message but the agent refused. Why?**
Check terminal:
- `[classify] label=off_topic` — L0 classifier said it's not in-scope
- `[retrieve] out_of_scope=True` — L1 threshold gate fired (no skill matched closely enough)
- LLM refusal in the body — L2 prompt-driven refusal
**Q: My `.env` change isn't picked up.**
`uvicorn --reload` watches `.py` files only. Kill server (Ctrl-C) and restart.
**Q: Index seems stale after I edited a skill.**
The build is insert-only by default. To fully refresh:
```bash
rm -rf chroma_db && uv run python scripts/build_index.py
Q: How do I reset chat memory?
Delete agent.db, or in browser DevTools clear localStorage["debug-agent.sid"] for just one session.
Q: Why are scores in 0..1 and not 0..2 (cosine distance)?
Chroma's similarity_search_with_relevance_scores normalizes when hnsw:space=cosine is set at collection creation. Already configured in indexer.py.
Q: Can I use a different LLM provider?
Yes if it's OpenAI-API-compatible. Set LLM_BASE_URL + LLM_API_KEY in .env. Embeddings still need OpenAI (OpenRouter doesn't provide embeddings).
Q: Why does the agent need OAuth creds?
The call_tenant_api tool needs to authenticate against your tenant. Use any OAuth2 client_credentials provider that issues bearer tokens via standard /protocol/openid-connect/token endpoint.
Q: Can the tool write data?
No, by design. The path/method validator in app/tools/tenant_api.py blocks PUT/PATCH/DELETE and restricts POST to a known read-only allowlist. Extending to writes requires explicit code changes + careful review.