172 lines
11 KiB
Markdown
172 lines
11 KiB
Markdown
# AI Agent App — product specification
|
|
|
|
This is the contract the project is built against. It is written for the agent that scaffolds
|
|
the repository, but it doubles as the human-readable brief: every requirement below is meant to
|
|
be implementable and verifiable.
|
|
|
|
Values in angle brackets (`<Agent name>`, `<model>`, `<retrieval>`) come from the
|
|
project-creation dialog and are recorded in the project record; replace them as you read.
|
|
|
|
## 1. Goal
|
|
|
|
A chat application built around an LLM agent: a user holds a conversation, the agent streams its
|
|
answer token by token, may consult the workspace's own documents through `<retrieval>`, and may
|
|
call tools. Answers that use documents cite the source chunks. It must be **runnable
|
|
end-to-end on day one** (even with a mock provider and seeded documents), not a set of screens
|
|
wired together later.
|
|
|
|
Non-goals (do not build unless the must-have list says otherwise): multi-agent orchestration,
|
|
fine-tuning, a public model marketplace, voice I/O, autonomous background agents.
|
|
|
|
## 2. Roles
|
|
|
|
| Role | Can do |
|
|
|------|--------|
|
|
| End user | Create conversations, chat with streaming answers, upload attachments, see citations and their own usage. |
|
|
| Workspace admin | Manage the knowledge base (upload, re-index, delete documents), configure the agent (system prompt, model, temperature, tools) and review usage/cost. |
|
|
| Platform operator | Set provider credentials and per-tenant quotas, inspect provider health and the audit log. |
|
|
|
|
Authentication is required for every conversation and for the whole admin surface; there is no
|
|
anonymous chat. Provide a seeded admin account and a seeded user for local development.
|
|
|
|
## 3. Information architecture (routes)
|
|
|
|
- `/` — conversations list: newest-first, title, last message snippet, model, new-conversation
|
|
action.
|
|
- `/conversations/:conversationId` — one conversation: streaming chat, message history,
|
|
attachments, stop/regenerate, rename/delete.
|
|
- `/knowledge` — knowledge base: document list with status (uploaded → processing → indexed →
|
|
failed), upload, re-index, delete, per-document chunk count.
|
|
- `/knowledge/:documentId` — one document: metadata, ingestion progress, chunk preview.
|
|
- `/agent` — agent configuration: system prompt, model, temperature, retrieval on/off, tool
|
|
selection.
|
|
- `/usage` — usage and cost: tokens and cost per period, per conversation and per model.
|
|
- `/settings` — profile, provider keys (BYOK), default model, export/delete account data.
|
|
- `/login`, `/logout` (admin surface guarded by the admin role).
|
|
|
|
## 4. Data model
|
|
|
|
Minimum viable entities (add fields the requirements imply; keep them typed and validated):
|
|
|
|
- **User** — id, email, passwordHash, name, role (`user` | `admin` | `operator`), createdAt.
|
|
- **Conversation** — id, userId, title, model, systemPromptSnapshot, createdAt, updatedAt,
|
|
archivedAt?.
|
|
- **Message** — id, conversationId, role (`system` | `user` | `assistant` | `tool`), content,
|
|
status (`pending` | `streaming` | `complete` | `error` | `cancelled`), inputTokens,
|
|
outputTokens, model, toolCallId?, createdAt.
|
|
- **Attachment** — id, messageId, kind (`image` | `file`), filename, mimeType, sizeBytes,
|
|
storagePath.
|
|
- **Document** — id, workspaceId, filename, mimeType, sizeBytes, status (`uploaded` |
|
|
`processing` | `indexed` | `failed`), error?, chunkCount, uploadedBy, createdAt.
|
|
- **DocumentChunk** — id, documentId, ordinal, text, tokenCount, embeddingRef (vector column for
|
|
`pgvector`, else the external collection/point id), createdAt.
|
|
- **AgentConfig** — id, workspaceId, systemPrompt, model, temperature, topP, retrievalEnabled,
|
|
retrievalTopK, enabledToolIds[], updatedAt.
|
|
- **ToolDefinition** — id, workspaceId, name, description, parameters (JSON schema), handler,
|
|
sideEffecting (boolean), enabled.
|
|
- **ToolCall** — id, messageId, toolId, arguments (JSON), result (JSON), status (`pending` |
|
|
`awaiting_confirmation` | `running` | `succeeded` | `failed` | `denied`), latencyMs.
|
|
- **UsageRecord** — id, userId, conversationId?, messageId?, model, inputTokens, outputTokens,
|
|
costMinor, currency, createdAt.
|
|
|
|
Retrieval storage is chosen in the dialog: `pgvector` keeps `embeddingRef` as a vector column;
|
|
`Qdrant`/`OpenSearch` store the external collection and point id; `None (chat only)` disables
|
|
ingestion and citations. Keep this behind one retrieval interface so the choice is a
|
|
configuration, not a rewrite.
|
|
|
|
Embeddings and model calls always run **server-side**; the browser never holds a provider key.
|
|
|
|
## 5. Key flows
|
|
|
|
1. **Streamed reply.** The user sends a message; the server persists it, calls the model through
|
|
the provider abstraction, and streams tokens back over the response stream. The assistant
|
|
message is `streaming` while tokens arrive and `complete` when the stream ends. The user can
|
|
cancel mid-stream (`cancelled`, partial text kept); a provider error ends the message as
|
|
`error` with a retry action.
|
|
2. **Document ingestion.** The admin uploads a document; the server stores it, extracts text,
|
|
chunks it, embeds each chunk, writes it to `<retrieval>`, and marks the document `indexed`.
|
|
Progress is visible per document (status + chunk count); a failure records the error and the
|
|
document stays re-indexable.
|
|
3. **Retrieval-augmented answer.** When retrieval is enabled, the question is embedded, the top-k
|
|
chunks are fetched, and they are passed to the model as untrusted context. The answer includes
|
|
**citations** that map back to the source chunk (document + ordinal), shown next to the
|
|
message and linking to `/knowledge/:documentId`.
|
|
4. **Tool calling.** The model may request a tool. Read-only tools run immediately; a
|
|
side-effecting tool pauses the turn in `awaiting_confirmation` and the user confirms or denies
|
|
before it executes. The result is fed back to the model and the turn continues; every call and
|
|
its status is recorded on the message.
|
|
5. **Conversation lifecycle.** A conversation is auto-titled from its first exchange, can be
|
|
renamed, and can be deleted (with confirmation) along with its messages and usage links.
|
|
|
|
## 6. Functional requirements
|
|
|
|
- **Chat:** streaming token-by-token with visible typing state; stop/regenerate; markdown
|
|
rendering with code blocks; per-message status; retry after an error without losing history.
|
|
- **Provider abstraction:** one interface with `DeepSeek`, `OpenAI`, `Anthropic` and `BYOK`
|
|
implementations, plus a **mock provider** used when no key is configured, so the app runs
|
|
without secrets.
|
|
- **Knowledge base:** upload (multiple files), list with status and filters, re-index, delete;
|
|
extraction for at least text/PDF/markdown; chunk size and overlap documented.
|
|
- **Retrieval:** configurable top-k; citations on every grounded answer; a clear "no relevant
|
|
documents" state instead of an invented answer; retrieval can be toggled per workspace and per
|
|
conversation.
|
|
- **Tools:** a registry with JSON-schema parameters; per-workspace enable/disable; the
|
|
side-effecting confirmation step; timeouts on every handler.
|
|
- **Usage:** record input/output tokens and cost per request; aggregate by period, conversation
|
|
and model; surface remaining quota to the user.
|
|
- **Admin:** edit the system prompt, model, temperature and tools; guard the surface with the
|
|
admin role; no destructive action without confirmation.
|
|
- **Seed data:** one workspace, one admin, one user, a few documents already indexed, and a
|
|
couple of conversations so the app is presentable on first run.
|
|
|
|
## 7. Non-functional requirements
|
|
|
|
- **Secrets:** never expose model/API keys to the client — all provider calls are server-side and
|
|
proxied; keys live in server env only and are documented in `.env.example`; BYOK keys are
|
|
encrypted at rest and never returned to the browser.
|
|
- **Limits:** per-user and per-tenant rate limits and token/cost quotas enforced server-side; a
|
|
request over quota is rejected with a clear message, not a silent failure.
|
|
- **Untrusted input:** documents, tool output and retrieved chunks are untrusted data — never
|
|
concatenated into instructions, delimited and labelled as data in the prompt, and never allowed
|
|
to trigger side-effecting tools without the confirmation step.
|
|
- **Resilience:** provider timeouts, rate-limit and partial-stream failures degrade with a clear
|
|
message and a retry path; a dropped connection keeps the partial answer.
|
|
- **Context budget:** conversation history sent to the model is bounded with a documented
|
|
truncation/summarisation strategy; the full transcript is always kept server-side.
|
|
- **Observability:** structured logs for model calls, retrieval and tool calls (no prompt or key
|
|
material in logs); a health endpoint that reports provider reachability.
|
|
- **Accessibility:** semantic landmarks, labelled inputs, keyboard-operable chat, visible focus,
|
|
contrast at least AA, and an announced streamed message for screen readers.
|
|
|
|
## 8. Acceptance criteria (definition of done)
|
|
|
|
- [ ] Install, dev server, lint, typecheck, tests and production build all pass.
|
|
- [ ] A streamed answer works end-to-end: tokens appear progressively, cancellation stops the
|
|
stream, and a provider error shows a retry without losing history.
|
|
- [ ] A question over an ingested document returns a grounded answer with citations to the source
|
|
chunks.
|
|
- [ ] An unavailable provider degrades with a clear, actionable message instead of a blank screen.
|
|
- [ ] Model/API keys are never present in the client bundle or any client response.
|
|
- [ ] Usage (input/output tokens and cost) is recorded per request and visible in `/usage`.
|
|
- [ ] A side-effecting tool cannot run without an explicit user confirmation.
|
|
- [ ] Seeded data makes the app presentable immediately; `.env.example` documents every secret.
|
|
- [ ] README quickstart (install, run, test, env) is accurate; empty/loading/error states exist.
|
|
- [ ] CI runs install + lint + typecheck + tests + build.
|
|
|
|
## 9. Suggested build order
|
|
|
|
Follow this order and finish (and verify) a layer before starting the next:
|
|
|
|
1. **Scaffold** the chosen stack, install dependencies, get the dev server and the empty shell
|
|
running, set up env handling for model keys, and commit the skeleton.
|
|
2. **Streaming chat shell**: the message list, the composer and token streaming end-to-end with
|
|
the mock provider.
|
|
3. **Conversation persistence**: conversations and messages in the database, auto-titling,
|
|
rename/delete, history loading.
|
|
4. **Document ingestion + retrieval**: upload → chunk → embed → index over `<retrieval>`, and
|
|
retrieval-augmented answers with citations.
|
|
5. **Tool calling**: the tool registry, the run loop, and the confirmation step for side-effecting
|
|
tools.
|
|
6. **Usage + limits**: token/cost accounting, quotas and rate limits.
|
|
7. **Quality**: tests for the flows above, CI, README, `.env.example`, accessibility pass.
|