Testing Requirements

This document defines testing requirements for the first version of YAAW - Yet Another Agent Wrapper.

The testing strategy is behavior-first. Tests should validate what a user can do and what the app visibly produces. Tests should avoid asserting private functions, implementation details, or framework-specific internals.

Requirements use:

MUST: required for the first shippable version.
SHOULD: expected unless cost or platform constraints make it impractical.
MAY: allowed but not required.

Testing Principles

Tests MUST focus on user-visible behavior, inputs, and outputs.
Tests MUST NOT depend on private functions or internal implementation details.
Tests MUST avoid over-mocking the app.
End-to-end tests MUST be the primary confidence layer.
Unit tests MAY exist, but they MUST validate high-value public behavior or deterministic input/output logic.
Screenshot capture MUST be available for E2E failures and key visual states.

E2E Scope

E2E tests MUST cover the primary user workflows:

Launch the app.
Create a project from a local directory.
Create a codex thread under a project.
Create a claude thread under a project.
Verify new thread creation asks which agent CLI to invoke.
Verify the visible thread name matches the selected CLI session’s reported name, title, or id.
Switch between project threads.
Use the agent CLI session terminal.
Close and reopen a thread and verify the stored agent CLI session identity is resumed.
Toggle the selected-thread bottom terminal with Cmd+J.
Resize major panels.
Collapse and expand the sidebar.
Collapse and expand the right tool panel.
Use the right panel in Files mode.
Search files with fuzzy matching.
Open a supported local preview file in Browser mode.
Navigate to a typed URL in Browser mode.
Open a file in editor mode and verify nvim fallback behavior.
Open Git mode and verify lazygit or git diff.
Paste an image into each supported CLI terminal and verify YAAW uses the terminal-native attachment shortcut without inserting a visible filesystem path.
Cycle right-panel modes with Cmd+Shift+[ and Cmd+Shift+].
Navigate globally with Cmd+[ and Cmd+].
Archive a thread.
Relaunch the app and verify persisted project/thread/layout metadata.

Full User Journey Test

The suite MUST include one high-level no-mock E2E test that navigates through the app like a real user.

This test MUST:

Launch a real app build.
Create or use a real temporary project directory.
Initialize real files in that directory.
Initialize a real Git repository when Git behavior is tested.
Create a project in the app.
Create a thread and choose codex when prompted.
Verify the thread name matches the Codex session’s reported name, title, or id.
Use the agent CLI session terminal.
Close and reopen the thread.
Verify reopening resumes the stored Codex session identity.
Open the right panel in Files mode.
Search for a file.
Open a supported preview file in Browser mode.
Open that file in editor mode.
Switch to Git mode.
Launch lazygit or git diff.
Toggle the selected-thread bottom terminal.
Resize or collapse panels.
Archive the thread.
Quit and relaunch the app.
Verify durable app state is restored where required.

The full user journey test MUST NOT mock app storage, terminal surfaces, file browser behavior, or right-panel mode switching.

The full user journey test MAY skip live lazygit assertions when lazygit is not installed, but it MUST verify fallback to git diff.

The full user journey test MAY use test-safe command doubles or controlled CLI fixtures so session names and resume identities can be asserted deterministically.

Inputs And Outputs

Tests MUST be written around explicit inputs and observable outputs.

Examples of valid inputs:

User clicks.
Keyboard shortcuts.
Text typed into fields.
Agent CLI choice from the available adapter set.
Controlled agent CLI session names and session identities.
Directory selections.
Files created in a temporary project.
Git repository state.
Terminal commands entered by the test.

Examples of valid outputs:

Visible project names.
Visible thread names.
Visible selected agent CLI labels.
Visible agent CLI session names.
Persisted agent CLI session identities.
Visible active right-panel mode.
Visible terminal output.
Visible file search results.
Visible browser preview or URL navigation state.
Visible nvim state.
Visible lazygit or git diff fallback output.
Persisted project/thread records after relaunch.
Screenshots captured by the test harness.

Tests SHOULD assert stable user-visible labels, state, and outputs rather than view hierarchy names or private object identifiers.

Screenshot Requirements

The E2E test harness MUST be able to capture screenshots.

Screenshots MUST be captured:

On every E2E failure.
After the app launches.
After project creation.
In Files mode.
In Browser mode.
In nvim mode.
In Git mode.
With the selected-thread bottom terminal expanded.
After panel resize or collapse behavior.

Screenshots SHOULD be written to a deterministic test artifact directory.

Screenshot filenames SHOULD include:

Test name.
Step name.
Timestamp or stable step number.

Screenshot tests MUST NOT replace behavioral assertions. They are supporting evidence for debugging and review.

Normal E2E Tests

The suite SHOULD include smaller E2E tests in addition to the full user journey.

Recommended tests:

Test	Required behavior
Project creation	A selected directory becomes a named project in the sidebar.
Project sidebar nesting	Project rows expand and collapse to show active and archived thread history.
Project row thread creation	A project-row new-thread action creates the thread under that project.
Project pin and reorder	Pinned projects sort first, and manual project reorder persists.
Codex thread creation	A new Codex thread appears under the selected project and gets an agent CLI session terminal.
Claude thread creation	A new Claude thread appears under the selected project and gets an agent CLI session terminal.
CLI choice prompt	Creating a thread asks which available CLI family to invoke.
Optional thread naming	Creating a thread accepts an optional user-entered name and falls back to CLI-derived naming when blank.
Thread pinning	Pinned threads sort above unpinned project threads and persist after relaunch.
Thread naming	The visible thread name matches the bound CLI session name, title, or id.
Thread resume	Closing and reopening a thread resumes the stored CLI session identity.
Thread switching	Switching threads changes the active agent CLI session terminal and right-panel context.
Thread activity	Agent helper and OSC notifications update the selected thread status, preview, unread state, focus-based read clearing, and persisted latest activity after relaunch.
Right-panel modes	Files, Browser, `nvim`, and Git modes can be selected by icon/tab.
Right-panel shortcuts	`Cmd+Shift+[` and `Cmd+Shift+]` cycle right-panel modes.
Settings key bindings	Settings exposes a searchable key binding list, edits write through to YAML, unbound actions remain unbound, and duplicate active bindings in a scope are reported.
Native shortcut defaults	`Cmd+,`, `Cmd+N`, `Cmd+Shift+N`, `Cmd+1`, `Cmd+2`, `Cmd+3`, and `Cmd+R` route to their documented app or Settings-scope actions.
Global navigation	`Cmd+[` and `Cmd+]` move through app navigation history.
Bottom terminal	`Cmd+J` expands and collapses the selected-thread bottom terminal.
File search	Hidden files are visible and fuzzy search returns expected matches.
Browser preview	Supported local preview files open in Browser mode from the file browser context menu.
Browser URL	A typed URL opens inside the right panel without a separate app window.
nvim open	Opening a file launches `nvim` in the right panel.
External open	The selected project and file external-open actions use detected destinations and configured defaults without depending on real installed editors.
Git open	Git mode launches `lazygit` or falls back to `git diff`.
Persistence	Project/thread/agent CLI session/layout metadata survive app relaunch.

Mocking Policy

E2E tests MUST use a real app process.

E2E tests MUST use real local directories and files.

E2E tests SHOULD use real SQLite storage in an isolated test location.

E2E tests SHOULD use real embedded terminal surfaces when practical.

Mocks MAY be used only for:

OS dialogs that cannot be controlled reliably in automation.
Time or clock values.
Isolated error injection that cannot be produced safely through real inputs.

Mocks MUST NOT replace the full user journey test.

Unit Test Policy

Unit tests are allowed when they are high value and input/output based.

Good unit test targets:

Fuzzy file matching.
Ignore-rule evaluation.
Path normalization.
YAML settings parsing and validation.
YAML settings raw text load/save behavior, including malformed-save no-overwrite behavior.
SQLite migration behavior against a real temporary database.
Public project/thread storage APIs.
Agent CLI session metadata persistence.
Agent CLI resume command construction at a public boundary.
Keyboard shortcut command routing at the public action level.

Poor unit test targets:

Private helper functions.
SwiftUI view internals.
AppKit object wiring.
Terminal wrapper internals.
Exact implementation classes.
Re-testing framework behavior.

Unit tests MUST use public APIs or stable module boundaries.

Unit tests MUST assert outputs for given inputs.

Unit tests SHOULD use real temporary files and databases where that improves confidence without making the test brittle.

Test Data

Tests SHOULD create temporary project directories with deterministic content.

Recommended fixture structure:

sample-project/
  README.md
  .env.example
  package.json
  src/
    auth.ts
    index.ts
  docs/
    guide.md

Git-mode tests SHOULD initialize a real Git repository and create at least one modified file so lazygit has visible state.

Tests MUST clean up temporary files after completion unless artifacts are intentionally retained for failure investigation.

Artifacts

E2E runs MUST produce artifacts for failed tests.

Artifacts SHOULD include:

Screenshots.
App logs.
Test runner logs.
SQLite database copy when safe.
YAML settings copy when safe.

Artifacts MUST NOT include sensitive terminal output unless explicitly enabled for a local debugging run.

Acceptance Criteria

The app has an E2E harness that can launch the native macOS app.
The E2E harness can interact with the app through clicks, typing, and keyboard shortcuts.
The E2E harness can capture screenshots.
The E2E harness covers opening the settings editor, editing YAML, saving, and returning to the workspace.
The suite includes one full no-mock user journey test.
The suite includes focused E2E tests for project/thread creation, agent CLI selection, session naming, session resume, panel behavior, file search, Browser mode, nvim, external-open settings/actions, lazygit, shortcuts, and persistence.
Persistence tests verify agent kind and CLI session identity survive relaunch.
Tests verify visible inputs and outputs instead of private implementation details.
Unit tests are limited to high-value public behavior or deterministic input/output logic.
Test artifacts are saved for failure review.