SemianiakaVY/openpencil
1
0
Fork 0
Code Issues Pull requests Actions 1 Packages Projects Releases Wiki Activity
openpencil/packages/docs/programmable/ai-chat.md
Danila Poyarkov 8d132ce070
feat(app): show who works on each page (#815) 
* fix(vue): keep the command palette open when a command opens a step

CommandPaletteRoot emitted select for every item, including one that only opens its children, so a host that closes on select closed the palette instead of showing the step. useCommandPalette.select now reports whether a command ran, and the root emits only then.

Disabled items were marked only with Reka's data-disabled; expose aria-disabled so assistive technology announces them.

* feat(app): jump between pages from the command palette

The palette had no way to reach a page. It now lists the pages visited recently in the tab, offers a Go to page step with every page, and finds any page by name.

Recent pages are tracked per editor session from page changes and reset when the document is replaced. Palette items can be search-only, so pages beyond the recent ones appear only when the query matches them. The divider-page rule moves out of PageListRoot so the palette skips dividers the same way, and useCommandPalette is exported from the package root.

* feat(canvas): draw agents' cursors as outlined sparkles

Editor state's remoteCursors becomes presenceCursors with a kind, since the list now includes local agents. People keep the filled arrow; an agent is a sparkle outlined in its owner's color, with an outlined name pill, so whose agent it is reads from the outline. Cursor drawing moves out of the pen overlay into canvas/overlays/presence.ts.

* feat(app): publish AI agent presence to collaborators

The built-in chat now appears as an agent with a callsign while it replies, at the nodes its tools touch on the run's page, and goes idle (off the canvas) when the reply ends. Agents live in a per-document presence registry and are published in their owner's awareness state, so collaborators see each other's agents in the owner's color; the payload is metadata only.

Peer awareness was cast without checks. It is now validated with Valibot, invalid fields are dropped rather than the peer, and names, selections, and agent counts are bounded.

* feat(app): follow agents and list them in the share panel

Following lived in collab and only knew people. It moves into the presence registry with a person-or-agent target, so you can follow anyone's agent, including your own outside a room: the view goes to the agent's page and keeps its cursor centered, stays attached while it idles between replies, and lets go when it leaves. A new editor action, centerOn, replaces reading the canvas size from the DOM. Peer cursors keep their zoom so following a person still matches it.

The share panel lists everyone in the room with their agents, each with its status, page, and a follow toggle, and your own agents can be renamed inline. CollabPanel moves to collab-panel, and the two-browser relay helpers move out of the collab spec into tests/helpers/collab.

* feat(app): show who works on each page

Agents now publish the page they work on, set when a reply starts on its pinned page and moved by switch_page, so a page is marked before the agent's first edit. presenceByPage groups people and working agents by page.

The Pages panel marks those pages with people's dots and agents' outlined sparkles in their owner colors, the command palette names who is on each page, and the chat says which page a reply is working on, with Go to page, while you view another one.

* docs(collaboration): list the agent model among shared presence

* test(app): stories for page presence markers and the chat's run location

The run location notice reads app state, so it moves into
useChatRunLocation and the component takes the agent and page as props.

* test(vue): a canvas story for presence cursors

Storybook now serves CanvasKit, so a story can render the real canvas:
people's arrows and agents' outlined sparkles, with controls for names,
colors, and zoom.

* feat(canvas): mark agents with a sparkle label instead of a sparkle cursor

A sparkle on its own did not read as a pointer. Agents now point with
the same filled arrow as people, in their owner's color, and their
outlined label starts with a sparkle.

* refactor(app): split the collaboration theme by component

One 18-slot theme served five components that each used a few slots,
with variants that applied to one slot. Avatars, the share button, the
presence list, page markers, and the mobile presence popover now have
their own themes, exported as tv() like the rest of src/theme.

* fix(app): truncate an agent's status before its name in the presence list

In a narrow share panel the status kept its width and the callsign
shrank to its first letter.

* feat(app): right-align page badges in a trailing area of the page row

Presence markers followed the page name. The row now has a trailing area,
right-aligned with its own spacing, where markers and later page badges
go.

* fix(app): key page markers by person or agent, not by name

Two people with the same name on a page, such as two Anonymous peers,
gave page markers duplicate keys. Entries now carry a stable id.
2026-10-04 01:18:16 +04:00

104 lines
9.4 KiB
Markdown
Raw Blame History

---
title: AI Chat
description: Built-in AI assistant with 90+ tools for creating and modifying designs.
---
# AI Chat
Press <kbd>⌘</kbd><kbd>J</kbd> (<kbd>Ctrl</kbd> + <kbd>J</kbd>) to open the AI assistant. Describe what you want — it creates shapes, sets styles, manages layout, works with components, and analyzes your design.
## Setup
1. Open the AI chat panel (<kbd>⌘</kbd><kbd>J</kbd>)
2. Click the settings icon
3. Add a model and configure its provider, model ID, credentials, and capabilities
4. Save the model and assign it to **Design agent**
You can configure multiple reusable models and separately assign models for design work, reviews, fast tasks, and image input. Models using the same provider connection reuse its stored credential.
The chat composer grows with multiline prompts and can pin the current canvas selection as explicit node context. Assistant messages show provider reasoning in collapsible sections and provide a per-response copy action. Image attachments remain available for visual references when a Vision model is configured. Streaming responses use a hardened Markdown renderer with Shiki-highlighted code blocks; unsafe link protocols and embedded data images are blocked.
## Step limit
In **Settings → AI & agents → Chat**, set **Maximum steps per message** to a whole number from 1 to 1,000. The default is 50. Press Enter or leave the field to save a valid value; invalid drafts do not replace the saved preference. Higher limits allow longer tool-driven tasks but can increase latency and provider cost.
The built-in AI captures this limit when each message starts. Stopping, remaining-step warnings, and the **Continue** action use that same budget. Changing it does not interrupt an ongoing request; the next message or continuation uses the new limit. A step is one model iteration and can include multiple tool calls. ACP and Pi agents manage their own limits.
## Tool access
Open **Settings → Tool access** to choose which tools direct AI model connections can use. Search by name or description, expand read-only or side-effect groups, and toggle individual tools or an entire group. Group switches affect all tools in that group, not only search results. **Restore defaults** restores the compact default tool set; extended tools such as `create_component` can be enabled individually.
Preferences are saved locally and apply to the next message, including in an existing conversation. They do not change an already-running request. Enabling many tools increases the schemas sent to the model.
The **Local MCP** segment has independent settings for clients connected to OpenPencil's MCP server, including ACP and Pi agents. Restart the server and reconnect stdio clients after changing those settings. Remote MCP connections, WebMCP access, and Pi's shell/filesystem permissions remain separate.
Tool toggles control which tools are offered, not which operations scripts may perform. An enabled `eval` or other script-capable tool can perform design operations whose dedicated tools are disabled; these switches are not a sandbox.
## Saved Conversations
Use **Conversation history** to return to a saved chat, start a **New chat**, or rename or delete a conversation. History and attachment previews are stored locally; **All chats** lets you browse transcripts from other documents.
A conversation belonging to another document is read-only until you open that document. A saved agent transcript is not a guarantee that its external agent session can resume: when resumption is unavailable, start a new chat. Local history is not cloud synchronization or a backup.
In the Chat settings beside the model overview, choose whether reasoning is **Collapsed by default**, **Expand while thinking**, or **Expanded by default**. Disclosure animations follow the app's reduced-motion preference. Expanding older reasoning does not force the conversation to scroll to the bottom.
## Supported Providers
| Provider | Models | Setup |
| ------------------------ | ----------------------------------------------- | ----------------------------------------------------------------------------------------------------------- |
| **OpenRouter** | Claude, GPT, Gemini, DeepSeek, Qwen, and others | API key from [openrouter.ai](https://openrouter.ai) |
| **Anthropic** | Claude Sonnet 4.6, Claude Opus 4.6 | API key from [console.anthropic.com](https://console.anthropic.com) |
| **OpenAI** | GPT-5.3 Codex, GPT-4.1, o3, o4-mini | API key from [platform.openai.com](https://platform.openai.com) |
| **Google AI** | Gemini 3.1 Pro, Gemini 3 Flash | API key from [aistudio.google.dev](https://aistudio.google.dev) |
| **Z.ai** | GLM-5.1, GLM-5, GLM-4.7, GLM-4.5 family | API key from [docs.z.ai](https://docs.z.ai/devpack/quick-start) |
| **MiniMax** | MiniMax M3, M2.7, M2.7-highspeed, M2.5, M2.1 | API key from [platform.minimax.io](https://platform.minimax.io/user-center/basic-information/interface-key) |
| **OpenAI-compatible** | Any endpoint with OpenAI API format | Custom base URL + key. Supports Completions and Responses API toggle. |
| **Anthropic-compatible** | Any endpoint with Anthropic API format | Custom base URL + key |
No backend, no subscription — your key talks directly to the provider. Browser requests are subject to each provider's CORS policy, and model deployments vary in how reliably they stream tool calls. See [BYOK provider and model compatibility](./byok-provider-compatibility) for measured results and reproduction steps.
## External MCP connections
Desktop ACP agents can also use trusted remote [Model Context Protocol](https://modelcontextprotocol.io/) servers. In **Settings → MCP**, under MCP connections, add a named Streamable HTTP endpoint, optionally save a bearer token, and enable the connection. OpenPencil stores the token in the configured credential backend rather than ordinary settings and resolves it only when starting the ACP session.
Remote servers must use HTTPS. Loopback HTTP endpoints are accepted for local development. Review and trust a server before enabling it: its tools may read external data or perform actions with the credentials you provide. OpenPencil's built-in design MCP server remains attached automatically and does not need to be added here.
## What It Can Do
The configurable tool catalog covers these categories; the tools offered to a model depend on your Tool access settings:
- **Create** — frames, shapes, text, components, pages. Renders JSX for complex layouts.
- **Style** — fills, strokes, effects, opacity, corner radius, blend modes.
- **Layout** — auto-layout, grid, alignment, spacing, sizing.
- **Components** — create components, instances, component sets. Manage overrides.
- **Variables** — create/edit variables, collections, modes. Bind to fills.
- **Query** — find nodes, XPath selectors, read properties, list pages, fonts, selection.
- **Inspect** — `get_jsx` for JSX roundtrip view, `diff_create` and `diff_jsx` for structural diffs, `diff_visual` for pixel diffs, `describe` for semantic role and design issue detection.
- **Analyze** — color palette, typography audit, spacing consistency, cluster detection.
- **Export** — PNG, SVG, JSX with Tailwind classes. Vision-based verification via `export_image`.
- **Vector** — boolean operations, path manipulation.
## Visual Verification
The assistant can verify its work visually. When `export_image` is enabled, it can capture a screenshot after creating or modifying designs and checks the result against the original request. This catches layout issues, missing elements, and color mismatches that text-only responses would miss. `diff_visual`, enabled by default, compares an edited node with a reference copy and returns the changed pixels and region, so the assistant can confirm an edit stayed within its target.
## Example Prompts
- "Create a card with a title, description, and a blue button"
- "Make all buttons on this page use the same border radius"
- "What fonts are used in this file?"
- "Change the background of the selected frame to a gradient from blue to purple"
- "Export the selected frame as SVG"
- "Find all text nodes with font size less than 12"
- "Describe the selected component — what role does it look like?"
- "Show me the JSX for this frame"
## Tips
- Select nodes before asking — the assistant knows what's selected.
- Be specific about colors, sizes, and positions for precise results.
- The assistant can modify multiple nodes in one message.
- You can browse other pages while a reply runs: the assistant keeps working on the page where the message started, and its previews show when you return. While you're away, the chat says which page it is working on, with **Go to page** to return. If the assistant switches pages itself, your view follows.
- Use "undo" in the editor if you don't like the result — AI mutations support full undo.
- All layout is recomputed automatically after each tool execution.
Reference in a new issue View git blame Copy permalink