314 lines
13 KiB
Markdown
314 lines
13 KiB
Markdown
# testing Specification
|
|
|
|
## Purpose
|
|
Test infrastructure. Playwright visual regression (E2E with Figma CDP reference), bun:test unit tests, no-chrome test mode, data-ready synchronization, and page reuse optimization.
|
|
## Requirements
|
|
### Requirement: Playwright visual regression testing
|
|
E2E tests SHALL use Playwright to create shapes and compare screenshots against baseline snapshots.
|
|
|
|
#### Scenario: Visual regression test pass
|
|
- **WHEN** `bun run test` is executed
|
|
- **THEN** Playwright tests create shapes, take screenshots, and compare against baseline PNGs
|
|
|
|
### Requirement: Figma CDP reference tests
|
|
A separate Playwright project SHALL connect to Figma via Chrome DevTools Protocol (CDP) to capture reference screenshots for pixel-perfect comparison.
|
|
|
|
#### Scenario: Figma reference capture
|
|
- **WHEN** `bun run test:figma` is executed with Figma running in debug mode
|
|
- **THEN** Playwright connects to Figma via CDP and captures reference screenshots
|
|
|
|
### Requirement: No-chrome test mode
|
|
The editor SHALL support a test mode (activated via URL param or environment) that hides Chrome/UI elements for clean screenshot capture.
|
|
|
|
#### Scenario: Test mode rendering
|
|
- **WHEN** the editor loads in test mode
|
|
- **THEN** only canvas content is rendered, without toolbar, panels, or other UI chrome
|
|
|
|
### Requirement: Data-ready optimization
|
|
The editor SHALL signal readiness via a `data-ready` attribute for E2E test synchronization, replacing unreliable timeouts.
|
|
|
|
#### Scenario: Wait for ready
|
|
- **WHEN** a Playwright test waits for the data-ready attribute
|
|
- **THEN** it proceeds only when the canvas is fully rendered
|
|
|
|
### Requirement: Page reuse for speed
|
|
E2E tests SHALL reuse the browser page across test cases to minimize overhead. Target: all E2E tests complete in <3s.
|
|
|
|
#### Scenario: Fast E2E execution
|
|
- **WHEN** the full E2E test suite runs
|
|
- **THEN** it completes in under 3 seconds by reusing the page
|
|
|
|
### Requirement: bun:test unit tests
|
|
Engine unit tests SHALL use bun:test and complete in <50ms.
|
|
|
|
#### Scenario: Unit test speed
|
|
- **WHEN** `bun test ./tests/engine` is run
|
|
- **THEN** all unit tests pass in under 50ms
|
|
|
|
### Requirement: Fig-import unit tests
|
|
Unit tests SHALL verify the .fig import pipeline: node type mapping, transform extraction, fill/stroke/effect import, gradient stops, image fills, arc data, stroke properties, and nested frame hierarchies.
|
|
|
|
#### Scenario: Run fig-import tests
|
|
- **WHEN** `bun test ./tests/engine/fig-import.test.ts` is executed
|
|
- **THEN** all import pipeline tests pass covering tier 1 rendering features
|
|
|
|
### Requirement: Layout unit tests
|
|
Unit tests SHALL verify Yoga auto-layout computation: direction, gap, padding, justify, align, child sizing (fixed/fill/hug), cross-axis sizing, wrap, and nested layouts.
|
|
|
|
#### Scenario: Run layout tests
|
|
- **WHEN** `bun test ./tests/engine/layout.test.ts` is executed
|
|
- **THEN** all layout computation tests pass
|
|
|
|
### Requirement: Layers panel E2E tests
|
|
E2E tests SHALL verify the layers panel: node visibility in tree, expand/collapse frames, selection sync between canvas and layers panel. Test fixtures match current demo shapes (Components, App Preview).
|
|
|
|
#### Scenario: Run layers panel E2E
|
|
- **WHEN** the layers panel E2E tests run
|
|
- **THEN** all tests pass verifying tree structure, visibility toggles, and selection sync with updated demo shape names
|
|
|
|
### Requirement: Component-instance sync unit tests
|
|
Unit tests SHALL cover the component-instance sync lifecycle: instance creation with `componentId` mapping on children, sync propagation of property changes, override preservation during sync, new child addition to instances, and detach breaking the link.
|
|
|
|
#### Scenario: Instance child mapping test
|
|
- **WHEN** `bun test ./tests/engine/scene-graph.test.ts` runs
|
|
- **THEN** the test verifies instance children have `componentId` pointing to component children
|
|
|
|
#### Scenario: Sync propagation test
|
|
- **WHEN** a component's text child changes and sync is triggered
|
|
- **THEN** the test verifies instance children receive the updated text and fontSize
|
|
|
|
#### Scenario: Override preservation test
|
|
- **WHEN** an instance child has an overridden text value
|
|
- **THEN** the test verifies the override is preserved while non-overridden properties sync
|
|
|
|
#### Scenario: New child addition test
|
|
- **WHEN** a new child is added to a component and sync runs
|
|
- **THEN** the test verifies existing instances gain the new child
|
|
|
|
#### Scenario: Detach test
|
|
- **WHEN** an instance is detached
|
|
- **THEN** the test verifies type becomes FRAME, componentId is null, and the component's instance list no longer includes it
|
|
|
|
### Requirement: Variable system unit tests
|
|
Unit tests SHALL cover the variables system: add and resolve color/number variables, alias chain resolution, circular alias detection (returns undefined), mode switching, bind/unbind to nodes, and variable removal cleanup. Total: 7 variable tests in a "Variables" describe block.
|
|
|
|
#### Scenario: Variable tests pass
|
|
- **WHEN** `bun test ./tests/engine` is run
|
|
- **THEN** all variable tests pass (add/resolve color, resolve number, alias chain, circular alias, mode switching, bind/unbind, removal cleanup)
|
|
|
|
|
|
### Requirement: .fig roundtrip tests
|
|
The test suite SHALL include roundtrip tests for real .fig files: parsing property invariants, encode/decode cycle fidelity. Test fixtures (material3.fig, nuxtui.fig) SHALL be tracked via Git LFS.
|
|
|
|
#### Scenario: Roundtrip encode/decode
|
|
- **WHEN** a .fig file is parsed and re-encoded
|
|
- **THEN** the re-encoded data decodes to the same node properties
|
|
|
|
#### Scenario: Property invariants
|
|
- **WHEN** a real .fig file is parsed
|
|
- **THEN** all nodes have valid types, dimensions ≥ 0, and required fields present
|
|
|
|
### Requirement: .fig import performance
|
|
The .fig import pipeline SHALL avoid O(n²) child resolution. A children index SHALL be built upfront for linear-time lookups.
|
|
|
|
#### Scenario: Large file import speed
|
|
- **WHEN** material3.fig (87K nodes) is imported
|
|
- **THEN** parsing completes in under 2 seconds (was 37s before fix)
|
|
|
|
### Requirement: JSX renderer tests
|
|
The test suite SHALL include tests for the JSX renderer covering all node types (Frame, Text, Rectangle, Ellipse, etc.), layout props, effects, and nesting. Currently 27 tests.
|
|
|
|
#### Scenario: Render Frame with children
|
|
- **WHEN** a JSX tree with Frame containing Rectangle and Text is rendered
|
|
- **THEN** the scene graph contains correct parent-child relationships and properties
|
|
|
|
### Requirement: AI chat Playwright tests
|
|
The test suite SHALL include Playwright e2e tests for the AI chat panel with mock transport (no real LLM calls). Tests cover: sending a message, receiving a response, tool call display. Real LLM tests available via TEST_REAL_LLM=1 flag.
|
|
|
|
#### Scenario: Chat with mock transport
|
|
- **WHEN** a Playwright test sends a message via the chat input
|
|
- **THEN** the mock transport returns a response and the message appears in the chat
|
|
|
|
### Requirement: JSX export tests
|
|
The test suite SHALL include tests for sceneNodeToJsx() covering shapes, text, layout props, effects, and multi-selection. Currently 14 tests.
|
|
|
|
#### Scenario: Export rectangle to JSX
|
|
- **WHEN** sceneNodeToJsx is called on a rectangle with blue fill
|
|
- **THEN** the output includes Rectangle component with bg prop
|
|
|
|
### Requirement: FigmaAPI unit tests
|
|
|
|
The project SHALL provide comprehensive unit tests for the Figma Plugin API in `tests/engine/figma-api.test.ts`.
|
|
|
|
#### Scenario: Node creation tests
|
|
- **WHEN** `bun test tests/engine/figma-api.test.ts` runs
|
|
- **THEN** tests verify `createFrame()`, `createRectangle()`, `createText()`, and other creation methods
|
|
|
|
#### Scenario: Property access tests
|
|
- **WHEN** tests read node properties (x, y, width, height, name, fills, strokes)
|
|
- **THEN** all getters return correct values from SceneNode
|
|
|
|
#### Scenario: Property setting tests
|
|
- **WHEN** tests set node properties (`node.name = "Test"`, `node.x = 100`)
|
|
- **THEN** SceneGraph is updated correctly
|
|
|
|
#### Scenario: Text property tests
|
|
- **WHEN** tests access `textNode.characters`, `fontSize`, `fontName`
|
|
- **THEN** text-specific properties work correctly
|
|
|
|
#### Scenario: Auto-layout tests
|
|
- **WHEN** tests set `layoutMode = "VERTICAL"`, `itemSpacing`, `paddingLeft`
|
|
- **THEN** auto-layout properties are applied
|
|
|
|
#### Scenario: Tree operation tests
|
|
- **WHEN** tests call `appendChild()`, `insertChild()`, `remove()`
|
|
- **THEN** scene graph tree is modified correctly
|
|
|
|
#### Scenario: Traversal tests
|
|
- **WHEN** tests call `findAll()`, `findOne()`, `findAllWithCriteria()`
|
|
- **THEN** correct nodes are returned
|
|
|
|
#### Scenario: Component tests
|
|
- **WHEN** tests create components and instances
|
|
- **THEN** component-instance relationships work
|
|
|
|
#### Scenario: Serialization tests
|
|
- **WHEN** tests call `figma.toJSON(node)`
|
|
- **THEN** JSON output includes all required properties
|
|
|
|
#### Scenario: Frozen array tests
|
|
- **WHEN** tests access `fills`, `strokes`, `children` and try to mutate
|
|
- **THEN** arrays are frozen and throw errors on mutation attempts
|
|
|
|
#### Scenario: 924 LOC of tests
|
|
- **WHEN** FigmaAPI test file is counted
|
|
- **THEN** it contains 924 lines covering 60+ test cases
|
|
|
|
### Requirement: Eval CLI integration tests
|
|
|
|
The project SHALL provide integration tests for eval command in `tests/engine/eval-cli.test.ts`.
|
|
|
|
#### Scenario: Inline code execution test
|
|
- **WHEN** test runs eval with `--code 'return figma.currentPage.id'`
|
|
- **THEN** output matches expected page ID
|
|
|
|
#### Scenario: Node creation test
|
|
- **WHEN** test runs eval creating a frame
|
|
- **THEN** document is modified and frame exists
|
|
|
|
#### Scenario: Write flag test
|
|
- **WHEN** test runs eval with `--write`
|
|
- **THEN** file is modified in-place
|
|
|
|
#### Scenario: Output file test
|
|
- **WHEN** test runs eval with `-o output.fig`
|
|
- **THEN** new file is created with modifications
|
|
|
|
#### Scenario: JSON output test
|
|
- **WHEN** test runs eval with `--json`
|
|
- **THEN** result is formatted as JSON
|
|
|
|
#### Scenario: Error handling test
|
|
- **WHEN** eval code throws error
|
|
- **THEN** stderr contains error message
|
|
|
|
#### Scenario: 202 LOC of integration tests
|
|
- **WHEN** eval CLI test file is counted
|
|
- **THEN** it contains 202 lines covering 17 integration scenarios
|
|
|
|
### Requirement: Tool schema unit tests
|
|
|
|
The project SHALL provide unit tests for unified tool definitions in `tests/engine/tools.test.ts`.
|
|
|
|
#### Scenario: Schema structure tests
|
|
- **WHEN** tests validate tool schemas
|
|
- **THEN** each tool has name, description, parameters, and handler
|
|
|
|
#### Scenario: Tool handler tests
|
|
- **WHEN** tests invoke tool handlers with valid params
|
|
- **THEN** handlers return expected results
|
|
|
|
#### Scenario: Parameter validation tests
|
|
- **WHEN** tests call handlers with invalid params
|
|
- **THEN** errors are raised or handled gracefully
|
|
|
|
#### Scenario: 390 LOC of tool tests
|
|
- **WHEN** tools test file is counted
|
|
- **THEN** it contains 390 lines
|
|
|
|
### Requirement: Tool adapter tests
|
|
|
|
The project SHALL provide adapter tests in `tests/engine/tools-ai-adapter.test.ts` (190 LOC) and `tests/engine/tools-cli.test.ts` (219 LOC).
|
|
|
|
#### Scenario: AI adapter format test
|
|
- **WHEN** AI adapter converts schema to LLM format
|
|
- **THEN** output matches expected structure
|
|
|
|
#### Scenario: CLI adapter test
|
|
- **WHEN** CLI commands use tool schema
|
|
- **THEN** parameters map correctly
|
|
|
|
### Requirement: App menu integration tests
|
|
|
|
The project SHALL provide integration tests for app menu in `tests/e2e/app-menu.spec.ts`.
|
|
|
|
#### Scenario: File menu test
|
|
- **WHEN** test clicks File menu items
|
|
- **THEN** actions (Open, Save, Export) are triggered
|
|
|
|
#### Scenario: Edit menu test
|
|
- **WHEN** test uses Edit menu (Undo, Redo, Copy, Paste)
|
|
- **THEN** operations execute correctly
|
|
|
|
#### Scenario: View menu test
|
|
- **WHEN** test uses View → Zoom in/out
|
|
- **THEN** viewport zoom changes
|
|
|
|
#### Scenario: Object menu test
|
|
- **WHEN** test uses Object → Group/Ungroup
|
|
- **THEN** nodes are grouped/ungrouped
|
|
|
|
#### Scenario: Text menu test
|
|
- **WHEN** test uses Text → Bold/Italic
|
|
- **THEN** text styles are applied
|
|
|
|
#### Scenario: Arrange menu test
|
|
- **WHEN** test uses Arrange → Bring to front
|
|
- **THEN** z-order changes
|
|
|
|
#### Scenario: 131 LOC of app menu tests
|
|
- **WHEN** app menu test file is counted
|
|
- **THEN** it contains 131 lines
|
|
|
|
### Requirement: Autosave integration tests
|
|
|
|
The project SHALL provide integration tests for autosave in `tests/e2e/autosave.spec.ts`.
|
|
|
|
#### Scenario: Autosave trigger test
|
|
- **WHEN** test modifies scene graph
|
|
- **THEN** autosave timer starts
|
|
|
|
#### Scenario: Debounce test
|
|
- **WHEN** test makes rapid changes
|
|
- **THEN** only one save occurs after 3 seconds
|
|
|
|
#### Scenario: Write test
|
|
- **WHEN** autosave timer expires
|
|
- **THEN** file is written
|
|
|
|
#### Scenario: Skip unchanged test
|
|
- **WHEN** sceneVersion matches savedVersion
|
|
- **THEN** write is skipped
|
|
|
|
#### Scenario: 113 LOC of autosave tests
|
|
- **WHEN** autosave test file is counted
|
|
- **THEN** it contains 113 lines
|
|
|
|
### Requirement: Test coverage expansion
|
|
|
|
The test suite SHALL expand from original coverage to include 2571 LOC of new tests across 7 files (figma-api.test.ts, eval-cli.test.ts, tools.test.ts, tools-ai-adapter.test.ts, tools-cli.test.ts, app-menu.spec.ts, autosave.spec.ts).
|
|
|
|
#### Scenario: Total new test lines
|
|
- **WHEN** new test files are counted
|
|
- **THEN** 2571 lines of tests are added
|