Danila Poyarkov 8bf71bd92f

feat(ai): add guided AI setup for providers, coding agents, and Pi (#916 )

* feat(storybook): prototype guided AI setup and task assignments

* refactor(storybook): adopt shared control foundations

* refactor(storybook): build AI setup on current settings foundations

Move the prototype to settings/ai-setup and compose SettingsSection, SettingsGroup, SettingsRow, AppAlert, AppBadge, and AppCheckbox instead of local section, status, and badge markup. Replace the nonexistent danger color and raw amber with the error and warning tokens.

* feat(ui): add a shared radio group

AppRadioGroup wraps the Reka radio group with typed options, labels each radio by its option text with any description as its accessible description, and supports all arrow keys unless an orientation is set. The AI setup wizard uses it for the spending choice, named by the step heading, and shares the choice card style with its checkboxes.

* fix(ui): draw unchecked checkboxes on the field background

AppCheckbox filled its box with the surface (text) color, so unchecked boxes were nearly black in the light theme and nearly white in the dark theme. Use the panel field background and accent hover border shared with the radio group and switch.

* refactor(storybook): drop the simplified AI connections panel

AI setup has two modes: skippable guided onboarding for most people and the existing advanced settings for power users, both editing the same model settings. Remove the third, simplified connections and tasks panel. The wizard's Advanced settings action and the returning-user screen now stand in for ModelsPanel, which offers Run guided setup. Removing Gateway's own Back button also fixes the blank screen it led to.

* feat(ai): plan guided AI setup from the model catalog

planOnboarding proposes design and vision models from the access a person already has, using the real provider and agent catalog, and falls back to OpenRouter only when pay-as-you-go is allowed. applyOnboardingPlan merges a confirmed plan into the model settings, reusing matching connections and profiles, keeping roles it was not asked about, and dropping only the empty fresh-install profile.

* refactor(ai): share model provider display names

Move the provider, agent, and Pi display-name lookup out of the model settings workflow so guided setup can reuse it.

* feat(ai): offer guided AI setup over the real model settings

Guided setup asks what AI should help with, what access the person
already has, and whether pay-as-you-go models are allowed, then
proposes design and vision models from the provider and agent catalog.
Connections reuse the provider key field and connection test, keys are
saved through the credential manager, and the confirmed plan is merged
into the model settings, keeping anything configured by hand. Saving
reports saved, partial, or failed like the profile editor.

A fresh install whose model settings are still the empty placeholder is
offered setup once with a skippable welcome; existing setups never see
it. Settings → AI & agents can run it again, and Advanced settings hands
off to the model editor. The Storybook fixtures for agents, OpenRouter
sign-in, Vercel AI Gateway, and the local server are replaced by the
real flow, with all copy translated.

Browser tests start with the offer dismissed through the shared
Playwright storage state; the first-run spec clears it.

* fix(ai): keep configured models and credentials safe in guided setup

Running guided setup again planned from the catalog defaults, so it
replaced hand-configured design and vision models and dropped their
settings; it now keeps a configured model while its access is still
selected or onboarding cannot offer that provider, and keeps vision when
nothing new covers it. Reused profiles must have the capabilities the
plan relies on, and an explicit vision assignment without image input is
cleared.

A server's saved key and its status now apply only to the connection at
the address being entered, and its connection test uses that
connection's API type. Entered keys are copied before saving, so closing
setup mid-save no longer drops them, and the Models list refreshes key
status after setup saves a key. Servers that do not check keys get a
hint to enter any value, and a step that only keeps configured models
says so instead of showing nothing.

* test(ai): check key status right after guided setup

The Models list must show a key saved by guided setup as connected without reopening Settings.

* feat(ai): sign in to OpenRouter from guided setup

OpenRouter can now be connected with its OAuth PKCE flow instead of a
pasted key. In the browser, sign-in opens in a popup that returns to a
static callback page on the app's origin, which relays the redirect to
the editor over a BroadcastChannel, so the editor never navigates away.
The desktop app opens the system browser and receives the redirect on
a one-shot 127.0.0.1 listener, the localhost callback OpenRouter
documents. Either way the editor checks the state, exchanges the code
for a key directly with OpenRouter, fills it in, and runs the
connection test. Waiting, blocked pop-ups, cancellation, expiry, and
failures are reported in the step, which keeps the pasted-key path.

Setup no longer offers Clear for a saved key, since removing keys
belongs to the advanced settings, and the service worker leaves
/oauth/ pages to the network.

* fix(settings): report unreadable model keys as unavailable

A saved key the browser credential store could not read, for example one left from an older session on the same origin, rejected the model status refresh and the startup credential check, which surfaced as a global error toast. Each read failure now marks only that connection as unavailable.

* feat(ai): map every role in guided setup and verify OpenRouter sign-in

Guided setup now proposes a model for design, vision, review, and fast
work, and the review step is a map of those roles with every suitable
model from the connected providers and a "Use recommended setup"
shortcut. Fast work defaults to the provider's catalog model tagged as
fast; behind an agent, review and fast work use the API model chosen
for vision. Review and fast work are never asked about, so a configured
choice, including none, stays unless it follows a design model it can
no longer follow.

The pay-as-you-go question only appears when the access already
selected leaves a requested role without a model, so choosing
OpenRouter or another account no longer asks it.

After signing in with OpenRouter, setup checks the key with
OpenRouter's key endpoint, which costs no credits, instead of a text
generation test. The step then says it is signed in, names the key,
warns when the account has no credits yet, and offers another account
in place of the key field and test button.

* feat(ai): offer OpenRouter only for goals nothing selected covers

The separate pay-as-you-go step asked an abstract question even when
the selected access already covered every goal. The connect step now
names a goal nothing selected can cover, such as visual review behind
a coding agent or a local server, and offers to add OpenRouter for it;
once added, it says OpenRouter fills the gap and can be removed again.
Setup can finish without visual review, but not without a design model.

A local or company server can be marked as able to read images, which
lets it cover visual review and makes it the preferred vision model
over a paid account.

* feat(ai): show provider logos and more providers in guided setup

Guided setup shows monochrome logos for coding agents, API accounts,
and local servers, from LobeHub's MIT-licensed static SVG set loaded as
an `ai` icon collection, so they follow the theme like Lucide icons.
DeepSeek, Z.ai, and MiniMax are offered under "More providers", and a
local server can start from the Ollama or LM Studio address instead of
typing it.

* feat(ai): guide coding agent setup in guided setup

Choosing Claude Code, Codex, or Gemini CLI in the desktop app now checks
whether the agent's ACP program and OpenPencil's MCP server, which
agents use to reach the canvas, are installed. An allowlisted
agent_lookup command finds the program on the same widened PATH as the
MCP lookup. The card shows install commands only for what is missing,
checks again on request, links a new setup guide, and copies a prompt
that asks an agent the person already uses to install both, confirm
they are on PATH, and sign in. In the browser, the agent section links
to the desktop app.

* fix(ai): space the More providers toggle like a group heading

The toggle sat flush against the account cards above and below it; it now reads as a group heading with the same rhythm as the other sections.

* feat(ai): detect and install coding agents in guided setup

Adopt the local agent discovery from #847. A desktop agent_lookup
command reports each agent's own CLI, its ACP adapter, npm, and
OpenPencil's MCP server on the widened PATH without starting any of
them, and the app can install a missing adapter or the MCP server with
npm, limited by the shell capability to those exact packages and the
MCP version that matches the app. Guided setup now tells "installed but
the OpenPencil adapter is missing" apart from "not installed", offers
one-click installs, links each vendor's own setup guide, and keeps the
manual commands and setup prompt for the browser, missing npm, or a
failed install. Codex install instructions move to
@agentclientprotocol/codex-acp, which replaces @zed-industries/codex-acp.

Kiro CLI support from the same pull request is left for a separate
change, since it needs ACP transport work.

Co-authored-by: GitttHomie <134371845+GitttHomie@users.noreply.github.com>

* build(app): resolve LobeHub icons with import.meta.resolve

The architecture lint forbids createRequire in ESM build code.

* test(app): seed AI setup specs through storageState

Follows the storage seeding used by other browser specs and the import type rule.

* feat(ai): set up Pi with its own sign-ins in guided setup

Pi now runs with the providers signed in to in the Pi CLI and Pi's
default model. The Harness companion reuses ~/.pi/agent; the app reads
only Pi's settings.json for the default model and never auth.json. A
saved key is still used as an AI Gateway key.

Guided setup offers Pi next to the other coding agents. On the desktop
it checks the Harness companion, the MCP server, and Pi's default
model, and installs the companion with one click through npm.

Agent discovery now reads the installed versions of the MCP server and
the Harness companion from their package.json without starting them.
Setup flags a version that does not match the app and shows the update
command for the package manager that installed it, instead of
reporting the server as installed and failing at the first message.

* feat(ai): check agent companions before a chat starts

A Pi chat without the Harness companion, or any agent chat whose
companion or MCP server version does not match the app, failed when the
process started and showed only the generic request error. The chat now
checks the companions through agent discovery first and names the fix,
with an action that opens guided setup. Pi sign-in and model problems
use the same path.

The Pi model editor shows the same companion, MCP server, and default
model status as guided setup, and no longer requires a model ID, since
Pi falls back to the default model set in Pi.

Supersedes the companion detection in #566, which ran the companion to
read its version and required an exact version match.

* fix(harness): start Pi sessions with MCP tools and keep unsent messages

Pi chats in the desktop app always configure OpenPencil's MCP server,
and three companion problems stopped them:

- @ai-sdk/harness-pi imports pi-mcp-adapter, which publishes TypeScript
sources. Node refuses to strip types under node_modules, so the
companion now strips them through a module load hook limited to
TypeScript dependencies. Bun runs them as is.
- pi-mcp-adapter imports @earendil-works/pi-tui, declared only as an
optional peer, so npm left it out. The companion depends on it at the
version pi-coding-agent uses.
- Pi reports live-process resume, yet the service handed it state saved
by an earlier session, and the just-bash sandbox cannot resume, so
every later session with that ID failed. Live-process backends now
start fresh and drop saved state.

When a chat cannot start, the composer now keeps the typed message
instead of discarding it.

* fix(harness): keep companion stdout for protocol messages

Pi prepares the packages listed in a person's Pi settings with npm,
which inherits the companion's stdout, and libraries log through
console.log. Both landed in the JSONL protocol stream, where the app
discarded them with warnings. The companion now keeps the real stdout
for protocol messages, sends other stdout writes to stderr, and quiets
npm on success through its environment.

The type-stripping hook no longer prints Node's experimental warning,
and the app logs companion stderr as diagnostics rather than errors,
since failures arrive as protocol errors.

Document Pi in the coding agents guide, the AI chat page, and the
README: guided setup installs the companion, Pi uses the Pi CLI's
sign-ins and default model, an AI Gateway key is optional, and the
companion needs Node.js 22.15 or later.

* test(harness): keep the pi-tui pin in step with pi-coding-agent

The companion depends on pi-tui only because pi-mcp-adapter imports it while declaring it optional (nicobailon/pi-mcp-adapter#805). Upgrading @ai-sdk/harness-pi moves pi-coding-agent, and a pin left behind would make npm install a second, mismatched pi-tui. The test fails until the pin matches.

* test(ai): follow the model catalog in guided setup tests

The plan and apply tests repeated catalog default and fast model IDs, so master's model update broke them without any change in setup behavior. They now read those models from the catalog.

* test(ai): keep the model catalog helper with the shared test helpers

Unit test homes under tests/app accept only *.test.ts files, so the onboarding tests' catalog helper moves to tests/helpers/ai.

* docs: tighten the guided setup and Pi changelog entries

Name every provider and server preset guided setup offers, describe the role step as it now works, and shorten the Pi entry.

* test(ai): type the guided setup test stubs for the test typecheck

Master now typechecks the test suites: fetch fakes go through fetchStub, the chat ref is shallow like the real one, and mocks declare the arguments the tests inspect.

* test: type the tabs module in the closed-documents spec

The spec imported the tabs module by its served URL without a type, which fails the test type check on master.

* test(ai): assert outcomes instead of copy in guided setup tests

Drop the setup-prompt test, which checked prompt prose, the onboarding wrapper cases that restated discovery, and the coversGoals case. Story plays and the OpenRouter E2E flow now assert controls and saved models instead of sentences and catalog model names, and the fast-model helper checks the planned model's catalog entry instead of recomputing the choice. tests/AGENTS.md states the rule.

* feat(ai): return desktop OpenRouter sign-in through a deep link

The desktop app ran a hand-written HTTP server on a localhost port to receive OpenRouter's redirect, and OpenRouter labels apps with a localhost callback by host and port. OpenRouter now redirects to a page on the web app that opens openpencil://oauth/openrouter with the same query, and the desktop shell forwards that link to the webview as an oauth-callback event. The attempt that started sign-in checks the state and exchanges the code with its PKCE verifier, which never leaves the app.

* feat(ai): ask OpenPencil's companions for their version

The desktop app read a companion's version by following its executable's symlink up to a package.json. That only worked for the Unix npm and bun layouts: Windows .cmd and .exe shims and version-manager shims such as Volta and mise are not links into the package, so the version was always unknown and an outdated companion went unreported. The MCP server, stdio bridge, and Harness companion now print their version for --version, and the app runs each installed one with --version --help under a timeout. A release older than --version prints its help or exits without a version line, which reads as outdated. A bun global install on Windows now gets the bun update command too.

* fix(ai): ask for a Pi sign-in when Pi has none

readPiAccount returned an account whenever a home folder existed, so a chat with no Pi sign-in reached the Harness and failed with a provider error instead of the guided pi-sign-in fix. It now reports whether Pi's auth.json exists, without reading it, and the capability allows that one check.

* fix(ai): keep the attachments of a message that was not sent

A message that never reached the chat came back to the composer as text only: its image previews were revoked and its referenced layers dropped. The composer now takes back the whole submission, or releases the previews when newer text replaced it. A message counts as sent once the chat holds it, so a failure after that no longer hands it back to be sent twice.

* refactor(app): read the app version from one constant

Four modules each derived the app version from the build define with the same test fallback.

* refactor(ai): report chat submission errors from their own module

Reverting turns from master and keeping unsent drafts together took useChatSubmission past the composition-root limit. The test for reverted turns now passes the setup messages the submission reports.

* refactor(app): keep the app version with the runtime config

Tools typecheck src/constants.ts through app imports without the Vite defines, so the version constant moves to src/app/runtime/version.ts.

* test(desktop): check npm installs of the companions at the app's release version

The scope test named the companion packages and version literally, so it would keep passing if the app requested something else. It now builds them from the app's package names and the release version a build embeds.

* refactor(ai): parse OpenRouter, Pi, and sign-in callback data with Valibot

The OpenRouter key info and code exchange checked their JSON with typeof chains, Pi's settings parsed JSON in a try before validating it, and the desktop sign-in trusted the shell's callback payload as typed. Each now goes through one schema.

* fix(ai): take back a message whose images could not be prepared

A message with images appears in the chat before its images are prepared, so a preparation failure counted as sent: the draft did not come back and its previews were already revoked. A message now counts as sent once it is dispatched; a failure before that removes the shown message and hands the draft back, and the composer's previews are revoked only after dispatch.

* fix(ai): restore an unsent message only in its own conversation

Switching conversations while a message was being sent could restore it into the newly opened one. The draft now comes back only if the conversation is unchanged, and its previews are released otherwise.

* refactor(app): keep one app version constant

The update window added an APP_VERSION to src/constants.ts beside the one in src/app/runtime/version.ts. The tools typecheck reaches src/constants.ts without the Vite defines, so the update window now reads the runtime one.

---------

Co-authored-by: GitttHomie <134371845+GitttHomie@users.noreply.github.com>

2026-10-07 08:46:57 +00:00

10 KiB

Raw Blame History

title	description
AI Chat	Built-in AI assistant with 90+ tools for creating and modifying designs.

AI Chat

Press ⌘J (Ctrl + J) to open the AI assistant. Describe what you want — it creates shapes, sets styles, manages layout, works with components, and analyzes your design.

Setup

The first time you open OpenPencil, guided setup asks what AI should help with and what you already use: a coding agent such as Claude Code, Codex, Gemini CLI, or Pi in the desktop app, an API account, or a local or company server. It connects that access and assigns it to the Design agent and Vision roles, keeping anything you configured by hand. With OpenRouter you can sign in instead of pasting an API key. Skip it to start designing, and run it again from Settings → AI & agents → Run guided setup. To configure models by hand:

Open the AI chat panel (⌘J)
Click the settings icon
Add a model and configure its provider, model ID, credentials, and capabilities
Save the model and assign it to Design agent

You can configure multiple reusable models and separately assign models for design work, reviews, fast tasks, and image input. Models using the same provider connection reuse its stored credential.

The chat composer grows with multiline prompts and can pin the current canvas selection as explicit node context. Assistant messages show provider reasoning in collapsible sections and provide a per-response copy action. Image attachments remain available for visual references when a Vision model is configured. Streaming responses use a hardened Markdown renderer with Shiki-highlighted code blocks; unsafe link protocols and embedded data images are blocked.

Step limit

In Settings → AI & agents → Chat, set Maximum steps per message to a whole number from 1 to 1,000. The default is 50. Press Enter or leave the field to save a valid value; invalid drafts do not replace the saved preference. Higher limits allow longer tool-driven tasks but can increase latency and provider cost.

The built-in AI captures this limit when each message starts. Stopping, remaining-step warnings, and the Continue action use that same budget. Changing it does not interrupt an ongoing request; the next message or continuation uses the new limit. A step is one model iteration and can include multiple tool calls. ACP and Pi agents manage their own limits.

Tool access

Open Settings → Tool access to choose which tools direct AI model connections can use. Search by name or description, expand read-only or side-effect groups, and toggle individual tools or an entire group. Group switches affect all tools in that group, not only search results. Restore defaults restores the compact default tool set; extended tools such as create_component can be enabled individually.

Preferences are saved locally and apply to the next message, including in an existing conversation. They do not change an already-running request. Enabling many tools increases the schemas sent to the model.

The Local MCP segment has independent settings for clients connected to OpenPencil's MCP server, including ACP and Pi agents. Restart the server and reconnect stdio clients after changing those settings. Remote MCP connections, WebMCP access, and Pi's shell/filesystem permissions remain separate.

Tool toggles control which tools are offered, not which operations scripts may perform. An enabled eval or other script-capable tool can perform design operations whose dedicated tools are disabled; these switches are not a sandbox.

Saved Conversations

Use Conversation history to return to a saved chat, start a New chat, or rename or delete a conversation. History and attachment previews are stored locally; All chats lets you browse transcripts from other documents.

A conversation belonging to another document is read-only until you open that document. A saved agent transcript is not a guarantee that its external agent session can resume: when resumption is unavailable, start a new chat. Local history is not cloud synchronization or a backup.

In the Chat settings beside the model overview, choose whether reasoning is Collapsed by default, Expand while thinking, or Expanded by default. Disclosure animations follow the app's reduced-motion preference. Expanding older reasoning does not force the conversation to scroll to the bottom.

Supported Providers

Provider	Models	Setup
OpenRouter	Claude, GPT, Gemini, DeepSeek, Qwen, and others	API key from openrouter.ai
Anthropic	Claude Sonnet 4.6, Claude Opus 4.6	API key from console.anthropic.com
OpenAI	GPT-5.3 Codex, GPT-4.1, o3, o4-mini	API key from platform.openai.com
Google AI	Gemini 3.1 Pro, Gemini 3 Flash	API key from aistudio.google.dev
Z.ai	GLM-5.1, GLM-5, GLM-4.7, GLM-4.5 family	API key from docs.z.ai
MiniMax	MiniMax M3, M2.7, M2.7-highspeed, M2.5, M2.1	API key from platform.minimax.io
OpenAI-compatible	Any endpoint with OpenAI API format	Custom base URL + key. Supports Completions and Responses API toggle.
Anthropic-compatible	Any endpoint with Anthropic API format	Custom base URL + key

No backend, no subscription — your key talks directly to the provider. Browser requests are subject to each provider's CORS policy, and model deployments vary in how reliably they stream tool calls. See BYOK provider and model compatibility for measured results and reproduction steps.

External MCP connections

Desktop ACP agents can also use trusted remote Model Context Protocol servers. In Settings → MCP, under MCP connections, add a named Streamable HTTP endpoint, optionally save a bearer token, and enable the connection. OpenPencil stores the token in the configured credential backend rather than ordinary settings and resolves it only when starting the ACP session.

Remote servers must use HTTPS. Loopback HTTP endpoints are accepted for local development. Review and trust a server before enabling it: its tools may read external data or perform actions with the credentials you provide. OpenPencil's built-in design MCP server remains attached automatically and does not need to be added here.

What It Can Do

The configurable tool catalog covers these categories; the tools offered to a model depend on your Tool access settings:

Create — frames, shapes, text, components, pages. Renders JSX for complex layouts.
Style — fills, strokes, effects, opacity, corner radius, blend modes.
Layout — auto-layout, grid, alignment, spacing, sizing.
Components — create components, instances, component sets, and slots. Manage overrides. Give components Reka UI behaviours so they work in preview.
Variables — create/edit variables, collections, modes. Bind to fills.
Query — find nodes, XPath selectors, read properties, list pages, fonts, selection.
Inspect — get_jsx for JSX roundtrip view, diff_create and diff_jsx for structural diffs, diff_visual for pixel diffs, describe for semantic role and design issue detection.
Analyze — color palette, typography audit, spacing consistency, cluster detection.
Export — PNG, SVG, JSX with Tailwind classes. Vision-based verification via export_image.
Vector — boolean operations, path manipulation.

Visual Verification

The assistant can verify its work visually. When export_image is enabled, it can capture a screenshot after creating or modifying designs and checks the result against the original request. This catches layout issues, missing elements, and color mismatches that text-only responses would miss. diff_visual, enabled by default, compares an edited node with a reference copy and returns the changed pixels and region, so the assistant can confirm an edit stayed within its target.

Example Prompts

"Create a card with a title, description, and a blue button"
"Make all buttons on this page use the same border radius"
"What fonts are used in this file?"
"Change the background of the selected frame to a gradient from blue to purple"
"Export the selected frame as SVG"
"Find all text nodes with font size less than 12"
"Describe the selected component — what role does it look like?"
"Show me the JSX for this frame"
"Make this component a switch that I can try in preview"

Tips

Select nodes before asking — the assistant knows what's selected.
Be specific about colors, sizes, and positions for precise results.
The assistant can modify multiple nodes in one message.
You can browse other pages while a reply runs: the assistant keeps working on the page where the message started, and its previews show when you return. While you're away, the chat says which page it is working on, with Go to page to return. If the assistant switches pages itself, your view follows.
Use "undo" in the editor if you don't like the result — AI mutations support full undo.
All layout is recomputed automatically after each tool execution.

10 KiB Raw Blame History