* Fix text rendering: always use paragraph wrapping, font fallback, batch tools, severity levels
Rendering:
- renderText: always use paragraph when fontProvider available, not just
when node-specific font loaded — fixes single-line overflow for Georgia/
non-Inter fonts
- buildParagraph: add Inter as fallback fontFamily so paragraph wraps text
even when requested font not yet loaded
- measureTextNode/buildTextPicture: remove isNodeFontLoaded gate — measure
with fallback font for consistent layout↔rendering
- TEXT_PICTURE_KEYS: add width/height so textPicture invalidates on resize
- renderText drawText fallback: clip to node bounds
- onAfterExecute: invalidate textPictures after font load
Layout:
- estimateTextSize: accept maxWidth, compute multi-line height
- toJSON: depth parameter with childCount fallback
AI tools:
- batch_update tool: multiple set_layout/set_layout_child in one call
- describe: accept ids array for multi-node inspection
- Issue severity (error/warning/info) with pattern classification
- checkNestedFlexWithoutFill: cross-axis fill tolerance
- CORE_TOOLS trimmed 34→27, rest moved to EXTENDED
- System prompt: batch_update, describe ids, severity rules, font/text
wrapping guidance, card grid patterns
* Add Pexels stock photo integration for AI chat
- stock_photo tool: search Pexels by query, download image, apply as
IMAGE fill on any node. Picks resolution based on node dimensions.
- Pexels API key stored in localStorage, synced to core via
setPexelsApiKey on change
- ProviderSettings UI: Pexels API key input with link to get free key
- System prompt: stock photo usage guidance
- Added to CORE_TOOLS so AI can use it in every session
* Batch stock_photo + calc, 4-phase workflow, fix wrap false positives
- stock_photo: accepts JSON array of requests, all fetched in parallel.
Rejects nodes with children (leaf shapes only). One call replaces 14.
- calc: accepts JSON array of expressions or single string. One call
replaces 3 separate calc invocations.
- System prompt: 4-phase workflow (plan → skeleton → content → polish).
Batch stock_photo and calc examples. Updated step budget.
- describe-layout-issues: skip 'nested flex may collapse' for wrap
containers (pill/tag frames are intentionally HUG). Extract
childNeedsFill helper to reduce complexity.
* Fix nested flex false positive for pills, single-root render rule
- describe-layout-issues: skip 'nested flex may collapse' when a sibling
already has grow or fill — pill/badge frames next to flex-grow content
are intentionally small (e.g. BreakingLabel next to ticker text)
- System prompt: render must have exactly one root element. Stock_photo
401 fallback: tell user, don't eval gradients.
* Rework workflow: render complete sections, not empty frames
Phase 2 now prescribes 5-7 renders, each producing a complete section
(nav, hero, stories, opinions, sidebar, footer) with placeholder images,
real text, proper auto-layout. No more empty frames to fill later.
Removes the fill content phase — skeleton IS the content, just without
real photos. Phase 3 is now just polish (batch stock_photo + fixes).
Step budget: 12-15 steps instead of 15-30.
* Pexels key input: match AI key behavior
Same pattern as the AI provider key — intermediate ref, 'Key saved —
enter new to replace' placeholder, Clear button, save on change. No
longer exposes the raw key in the input field.
* No empty frames in skeleton, select nodes after each tool
- System prompt: explicit rules against empty frames, no pre-creating
wrapper frames for future content. Each render = complete section with
visible placeholders. Render 1 = only nav+ticker, not the whole page
with empty containers.
- tools.ts: set selection to flash node IDs after each mutating tool,
so user sees which section AI is working on (selection border persists
vs flash which is transient).
* AI overlay: pulsing blue while working, green flash on done; skeleton workflow
Rendering:
- ai-overlays.ts: new module with pulsing blue border (sinusoidal
opacity 0.3-0.8 at 1.5s period) for nodes AI is working on, green
fade-out flash (800ms) when done
- renderer.ts: aiMarkActive/aiMarkDone/aiClearAll methods, drawn after
regular flashes
- constants.ts: AI_ACTIVE_COLOR, AI_DONE_COLOR, timing constants
Integration:
- ai-adapter.ts: onBeforeExecute now receives args (for target ID)
- tools.ts: marks target node (id or parent_id) active before tool
executes, marks done via onFlashNodes after completion
- editor.ts: aiMarkActive/aiMarkDone/aiClearAll forwarded to renderer
Workflow:
- System prompt: 3-phase skeleton workflow. Phase 2 renders gray
placeholder blocks (Rectangle bg=#E2E8F0) for every section — page
looks like wireframe before any real content. Phase 3 replaces
skeletons with real content. Phase 4 batch stock_photo + polish.
* render replace_id: atomic skeleton-to-content swap, fix AI overlay lifecycle
render tool:
- New replace_id parameter — renders new content, places it at the
skeleton's position in parent, then deletes the skeleton. No visual
gap between skeleton removal and content appearance.
AI overlay fixes:
- Track lastActiveIds across before/after execute — aiMarkDone clears
the exact nodes that were marked active, not just the result IDs
- aiFlashDone: direct green flash on result nodes (not gated by
_aiActiveNodes membership)
- replace_id included in target ID detection for active marking
System prompt:
- Phase 3 uses render({ replace_id }) instead of delete_node + render
* Fix gradient fills, nested flex false positives, render insert_index, h=hug prompt
- set_fill: gradient support with color_end + direction (top-bottom,
left-right, etc). Uses proper Matrix type. No more eval for gradients.
- describe-layout-issues: skip nested flex warning for all non-MIN
alignments (SPACE_BETWEEN, CENTER, MAX) — not just SPACE_BETWEEN
- render: insert_index parameter for controlling child order
- system prompt: page frame h="hug" not fixed height
- AI overlay: mark active on any tool with target ID (not just
mutating), clear previous active in onBeforeExecute
* Add desktop business news site example to system prompt
Full workflow example: skeleton with gray placeholders → replace_id
content fills → batch stock_photo. Shows 12-col grid pattern, card
components, section headers, sidebar widgets, h=hug page frame.
Complements the existing mobile app UI example.
* Drop blue active pulse, keep only green done flash
Blue active marking was invisible — tool execution is 20-80ms (one
frame), AI thinking happens between tool calls where no hooks fire.
Green done flash on result nodes works and is visible.
* Blue pulse on skeleton during AI tool streaming
Watch streaming message parts in ChatPanel — when a tool part appears
with state 'call'/'partial-call' and has replace_id or parent_id in
args, mark that node active (blue pulse). When tool transitions to
'output-available', mark done (green flash). This shows the pulsing
blue border on the skeleton wireframe frame while AI generates JSX,
and switches to green when the content replaces it.
* AI overlay debug log section, watch tool parts by toolCallId
- chat-debug.ts: aiOverlayLog array + AI OVERLAY LOG section in debug
output showing every part/mark-active/mark-done event with timestamps
- ChatPanel.vue: check parts by toolCallId instead of isToolUIPart,
log all overlay events to aiOverlayLog for debugging
* Fix AI overlay: use 'input' not 'args' for tool part params
Vercel AI SDK UIToolInvocation uses 'input' field (not 'args') for
tool call parameters. The watch was checking p.args which was always
undefined, so no overlay events ever fired. Also use p.type instead
of p.toolName for the log.
* Move AI overlay to onBeforeExecute/onFlashNodes, reorder render params
- onBeforeExecute: mark replace_id/parent_id active (blue pulse)
- onFlashNodes: clear active, flash done on new nodes (green)
- Reorder render tool params: replace_id first so streaming partial
input reveals target ID before JSX body
- Remove ChatPanel watch — all overlay logic in tools.ts now
- Add aiClearActive() to renderer (clears active without touching
done flashes)
* Wait 2 rAF frames after marking active before tool execute
onBeforeExecute marks the target node active (blue pulse), but
execute ran immediately after without yielding to the render loop.
The blue frame was never painted because markActive → execute →
clearActive all happened in the same JS task. Now await two
requestAnimationFrame callbacks so the canvas renders at least one
blue-pulse frame before execution starts.
* Add comprehensive AI overlay debug logging to chat debug log
Both streaming watch (ChatPanel) and tool execution hooks (tools.ts)
now log every event to aiOverlayLog with full state: renderer exists,
activeNodes count, hasFlashes, node existence. This will show exactly
what happens and why the blue pulse is or isn't visible.
* Filter partial node IDs in streaming overlay watch
During input-streaming, replace_id may be partially received (e.g.
'0:' without the number part). Guard with isValidNodeId check.
Also refactor watch callback into collectPendingTargets +
syncOverlayState to stay under complexity limit.
Analysis of last debug log:
- Hero: only 235ms active (streaming didn't deliver replace_id early)
- Stories: 14.8s active — should be clearly visible
- Opinions: 12.5s, LatestNews: 11.5s, Stocks: 9.8s — all long enough
- Hero was the only one too fast to notice
* Fix undo race, JSX pragma conflict, partial ID overlay, add deletion log
- Fix undo snapshot race condition: replace shared `let beforeSnapshot`
with a per-tool stack. Vercel AI SDK runs tool calls from one step via
Promise.all — concurrent mutating tools overwrote the shared snapshot,
causing wrong undo entries and potential content loss.
- Fix 'h is not a function' JSX error: rename sucrase pragma from `h`
to `__h`. When AI writes `.map((h, i) => ...)`, the loop variable
shadowed the pragma, crashing render.
- Fix partial node ID `0:1` in overlay: strengthen `isValidNodeId` to
require both parts ≥2 chars and verify node exists in graph. During
streaming, `replace_id: "0:1811"` arrives as `0:1` → `0:18` → etc.
- Add deletion log to SceneGraph: `_deletionLog` records id, name,
type, parentId, timestamp, and truncated stack for last 200 deletions.
Included in debug log as NODE DELETION LOG section to diagnose the
intermittent frame disappearance bug.
* Reduce find_nodes spam and describe bloat in system prompt
- Instruct AI to reuse IDs from render results and describe output
instead of calling find_nodes to rediscover them (saves 8+ calls)
- Recommend depth=1 for full-page describe, depth=2 only for sections
* Auto-depth describe + validate-after-render workflow
Describe tool now auto-picks depth based on subtree size when depth
is omitted: ≤15 nodes → depth 4, ≤40 → 3, ≤100 → 2, larger → 1.
Max raised to 5 for explicit overrides.
Prompt updated: after each render replace_id, call describe on that
block (auto depth gives full detail on small sections). Every 2-3
content renders, describe root at depth=1 to catch cross-section
layout issues early.
* Enforce describe-after-every-render in system prompt
Show explicit render → describe → fix pattern with code example.
AI was skipping per-block validation and batching all describes
at the end, causing compounding layout errors.
* Add describe rules: nested text, missing padding, button padding, row centering
Four new layout issue detectors in describe-issues.ts:
- Nested Text (error): Text node inside Text causes overflow.
Suggestion: split into separate Text elements in flex row.
- Zero padding container (warning): Frame with fill + text children
but no padding — content touches edges.
- Button without padding (warning): Button-like frame with children
but zero horizontal padding.
- Row without vertical centering (warning): Horizontal flex with
items=start, tall container, and short children — suggests
items=center.
All verified with bun -e tests including false positive checks.
* Fix gradient alpha loss, add undersize child detection
Gradient fix: set_fill with color_end was overwriting alpha to 1
on both gradient stops (`{ ...c, a: 1 }`). Now preserves original
alpha from parsed hex — #00000000 → a:0, #000000AA → a:0.667.
This caused transparent-to-dark overlays to render as solid black.
New describe rule: detect child significantly narrower than parent
in a container without auto-layout (e.g. 100px Rectangle inside
896px Frame). Suggests adding flex + w=fill. Caught the HeroImg
bug that describe previously missed.
* Auto-include issues in render result, no separate describe needed
Render tool now scans the created subtree for errors/warnings and
includes them in the response. AI gets issues automatically without
needing to call describe — eliminates the pattern where AI skips
describe on 'simple' sidebar blocks.
Example response:
{ id: '0:42', name: 'Card', issues: [
{ node: 'Text', id: '0:45', message: 'no color', severity: 'error' }
] }
* Run layout before collecting render issues
Issues were checked on raw node sizes (default 100px) before
computeAllLayouts ran in onAfterExecute. Now render calls
computeAllLayouts inline before collectSubtreeIssues, so issues
reflect actual computed dimensions.
This fixes hundreds of false 'X 100px overflows parent' warnings
that made AI chase nonexistent problems.
* Revert auto-issues in render and layout-before-issues
Reverts 09a9b89 and 70c116a. Auto-including issues in render
result caused false positives (100px default sizes before layout)
and made AI chase nonexistent problems, degrading output quality.
* Clean up AI chat PR: remove debug scaffolding, add stock photo provider adapter
- Delete _deletionLog (stack traces on every node deletion)
- Delete aiOverlayLog and all debug push() calls
- Delete succesful_prompt_diffucult_ux.md (raw debug dump) and tokens.md (empty)
- Remove double-rAF hack from ai-adapter (added 32ms to every tool call)
- Restore simple beforeSnapshot for undo (was matching by tool name)
- Remove 70-line overlay state tracker from ChatPanel.vue
- Remove Math.* regex hack from calc (expr-eval has builtins)
- Refactor stock-photo into provider adapter: Pexels + Unsplash
- Add Unsplash key management to settings UI and use-chat composable
---------
Co-authored-by: Danila Poyarkov <dev@dannote.net>
182 lines
5.9 KiB
TypeScript
182 lines
5.9 KiB
TypeScript
import { describe, test, expect, mock } from 'bun:test'
|
|
import { renderText } from '../../packages/core/src/renderer/scene'
|
|
import type { SceneNode } from '../../packages/core/src/scene-graph'
|
|
import type { SkiaRenderer } from '../../packages/core/src/renderer/renderer'
|
|
import { initCanvasKit } from '../../packages/cli/src/headless'
|
|
import { SceneGraph, SkiaRenderer as SkiaRendererClass } from '@open-pencil/core'
|
|
import { initFontService, setCJKFallbackFamily, markFontLoaded } from '../../packages/core/src/fonts'
|
|
|
|
function createMockCanvas() {
|
|
return {
|
|
drawParagraph: mock(() => {}),
|
|
drawPicture: mock(() => {}),
|
|
drawText: mock(() => {}),
|
|
save: mock(() => {}),
|
|
restore: mock(() => {}),
|
|
clipRect: mock(() => {})
|
|
}
|
|
}
|
|
|
|
function createMockParagraph() {
|
|
return { delete: mock(() => {}) }
|
|
}
|
|
|
|
function createMockPicture() {
|
|
return { delete: mock(() => {}) }
|
|
}
|
|
|
|
function createMockRenderer(overrides: Partial<Record<string, unknown>> = {}) {
|
|
const paragraph = createMockParagraph()
|
|
return {
|
|
fontsLoaded: true,
|
|
fontProvider: {},
|
|
textFont: {},
|
|
fillPaint: { getColor: () => new Float32Array([0, 0, 0, 1]) },
|
|
ck: {
|
|
MakePicture: mock(() => createMockPicture()),
|
|
LTRBRect: mock((...args: number[]) => args),
|
|
ClipOp: { Intersect: 0 }
|
|
},
|
|
DEFAULT_FONT_SIZE: 14,
|
|
isNodeFontLoaded: mock(() => true),
|
|
buildParagraph: mock(() => paragraph),
|
|
_paragraph: paragraph,
|
|
...overrides
|
|
} as unknown as SkiaRenderer & { _paragraph: ReturnType<typeof createMockParagraph> }
|
|
}
|
|
|
|
function textNode(overrides: Partial<SceneNode> = {}): SceneNode {
|
|
return {
|
|
text: 'Hello 你好',
|
|
fontSize: 16,
|
|
fontFamily: 'Arial',
|
|
...overrides
|
|
} as SceneNode
|
|
}
|
|
|
|
describe('renderText', () => {
|
|
test('uses buildParagraph when fonts are loaded and node font is available', () => {
|
|
const r = createMockRenderer()
|
|
const canvas = createMockCanvas()
|
|
|
|
renderText(r, canvas as never, textNode())
|
|
|
|
expect(r.buildParagraph).toHaveBeenCalledTimes(1)
|
|
expect(canvas.drawParagraph).toHaveBeenCalledTimes(1)
|
|
expect(canvas.drawText).not.toHaveBeenCalled()
|
|
expect(r._paragraph.delete).toHaveBeenCalledTimes(1)
|
|
})
|
|
|
|
test('uses paragraph even when node font is NOT available (fallback to default)', () => {
|
|
const r = createMockRenderer({ isNodeFontLoaded: mock(() => false) })
|
|
const canvas = createMockCanvas()
|
|
|
|
renderText(r, canvas as never, textNode())
|
|
|
|
expect(r.buildParagraph).toHaveBeenCalledTimes(1)
|
|
expect(canvas.drawParagraph).toHaveBeenCalledTimes(1)
|
|
expect(canvas.drawText).not.toHaveBeenCalled()
|
|
})
|
|
|
|
test('prefers textPicture over paragraph', () => {
|
|
const r = createMockRenderer()
|
|
const canvas = createMockCanvas()
|
|
const node = textNode({ textPicture: new Uint8Array([1, 2, 3]) })
|
|
|
|
renderText(r, canvas as never, node)
|
|
|
|
expect(canvas.drawPicture).toHaveBeenCalledTimes(1)
|
|
expect(r.buildParagraph).not.toHaveBeenCalled()
|
|
})
|
|
|
|
test('falls back to drawText only when fonts are NOT loaded', () => {
|
|
const r = createMockRenderer({ fontsLoaded: false, fontProvider: null })
|
|
const canvas = createMockCanvas()
|
|
|
|
renderText(r, canvas as never, textNode())
|
|
|
|
expect(canvas.drawText).toHaveBeenCalledTimes(1)
|
|
expect(r.buildParagraph).not.toHaveBeenCalled()
|
|
})
|
|
|
|
test('does nothing for empty text', () => {
|
|
const r = createMockRenderer()
|
|
const canvas = createMockCanvas()
|
|
|
|
renderText(r, canvas as never, textNode({ text: '' }))
|
|
|
|
expect(r.buildParagraph).not.toHaveBeenCalled()
|
|
expect(canvas.drawText).not.toHaveBeenCalled()
|
|
expect(canvas.drawPicture).not.toHaveBeenCalled()
|
|
})
|
|
})
|
|
|
|
describe('renderText headless visual', () => {
|
|
test('renders CJK text via fallback font through paragraph shaper', async () => {
|
|
const ck = await initCanvasKit()
|
|
const fontProvider = ck.TypefaceFontProvider.Make()
|
|
initFontService(ck, fontProvider)
|
|
|
|
const interData = await Bun.file('public/Inter-Regular.ttf').arrayBuffer()
|
|
fontProvider.registerFont(interData, 'Inter')
|
|
markFontLoaded('Inter', 'Regular', interData)
|
|
|
|
const notoPath = new URL('../../tests/fixtures/fonts/NotoSansSC-Regular.ttf', import.meta.url).pathname
|
|
const notoData = await Bun.file(notoPath).arrayBuffer()
|
|
fontProvider.registerFont(notoData, 'Noto Sans SC')
|
|
setCJKFallbackFamily('Noto Sans SC')
|
|
|
|
const graph = new SceneGraph()
|
|
const page = graph.getPages()[0]
|
|
const node = graph.createNode('TEXT', page.id, {
|
|
text: '你好世界',
|
|
fontFamily: 'Inter',
|
|
fontSize: 32,
|
|
fontWeight: 400,
|
|
width: 200,
|
|
height: 50,
|
|
fills: [{ type: 'SOLID', color: { r: 0, g: 0, b: 0, a: 1 }, opacity: 1, visible: true }]
|
|
})
|
|
|
|
const surface = ck.MakeSurface(200, 50)!
|
|
const renderer = new SkiaRendererClass(ck, surface)
|
|
renderer.viewportWidth = 200
|
|
renderer.viewportHeight = 50
|
|
renderer.dpr = 1
|
|
renderer.fontsLoaded = true
|
|
;(renderer as unknown as Record<string, unknown>).fontProvider = fontProvider
|
|
|
|
const canvas = surface.getCanvas()
|
|
canvas.clear(ck.WHITE)
|
|
renderText(renderer, canvas, graph.getNode(node.id)!)
|
|
surface.flush()
|
|
|
|
const image = surface.makeImageSnapshot()
|
|
const encoded = image.encodeToBytes(ck.ImageFormat.PNG, 100)!
|
|
image.delete()
|
|
surface.delete()
|
|
|
|
expect(encoded.length).toBeGreaterThan(200)
|
|
|
|
const decodedImage = ck.MakeImageFromEncoded(encoded)!
|
|
const pixels = decodedImage.readPixels(0, 0, {
|
|
width: 200,
|
|
height: 50,
|
|
colorType: ck.ColorType.RGBA_8888,
|
|
alphaType: ck.AlphaType.Unpremul,
|
|
colorSpace: ck.ColorSpace.SRGB
|
|
})!
|
|
decodedImage.delete()
|
|
|
|
let darkPixels = 0
|
|
for (let i = 0; i < pixels.length; i += 4) {
|
|
if (pixels[i] < 128 && pixels[i + 1] < 128 && pixels[i + 2] < 128) {
|
|
darkPixels++
|
|
}
|
|
}
|
|
// CJK characters are dense — should have many dark pixels if rendering correctly
|
|
// Tofu boxes would have far fewer (just outlines)
|
|
expect(darkPixels).toBeGreaterThan(500)
|
|
})
|
|
})
|