Exact source used by this build. SHA-256: 9444becde3f3ff8ffc638775b387fc9ed62203c9334a3dee65f9b4d0a0a6800f.
1 # Feel lab
2
3 The feel lab records real inputs against the local game, saves 30 fps action footage and numbered frame strips, and measures the current client. It does not modify game code. Each scenario starts in a fresh local world; setup actions such as hoeing before watering use the same UI as the recorded actions.
4
5 ## Setup and run
6
7 Requirements: Bun, `/opt/homebrew/bin/ffmpeg`, and Playwright's Chromium. Playwright is a development dependency because the lab needs reproducible Chromium viewports, keyboard/touch input, browser measurements, and video capture; it is not shipped with the game.
8
9 ```sh
10 bun install --no-save
11 PLAYWRIGHT_BROWSERS_PATH="$PWD/runs/feel/.cache/browsers" bunx playwright install chromium --only-shell
12
13 bun run feel:record walk
14 bun run feel:record hoe water plant
15 bun run feel:record
16 bun run feel:check
17 bun run check
18 ```
19
20 Run commands from the repository root. No scenario arguments means all starter scenarios. Recording always covers both desktop 1280×720 and iPhone 16 landscape 852×393, DPR 3 with touch enabled. This is Chromium mobile emulation, so it does not measure Safari or native iPhone performance.
21
22 The dependency version is pinned in `package.json`. The existing `bun.lock` was outside this task's allowed scope, so installation uses `--no-save` to preserve it; a frozen-lockfile installation needs a separately authorized lockfile update.
23
24 Capture uses CDP screencast JPEGs with Chromium's compositor timestamps. ffmpeg preserves the irregular source timing, trims to the action interval, encodes a constant 30 fps `video.webm`, and extracts its PNG frames. Video and strips use CSS viewport dimensions; the phone browser still runs at DPR 3.
25
26 The recorder reuses `dev:server` and `dev:client`, allocates available ports, uses disposable local persistence, joins with a test name, and shuts down its browser and owned development processes on completion or failure. A tools-only Vite configuration points the client at that local server. An isolated world name keeps a recording separate from normal local play. Server startup uses local mode and does not require deployment or credentials.
27
28 Browser files default to the ignored `runs/feel/.cache/browsers` directory. Set `PLAYWRIGHT_BROWSERS_PATH` before running either command to use another existing installation. Chromium needs permission to launch normal local subprocesses. A macOS sandbox error containing `MachPortRendezvousServer` / `bootstrap_check_in ... Permission denied` prevents recording; rerun from an environment that permits Chromium. Such a failure provides no motion measurements.
29
30 ## Starter scenarios
31
32 | Name | Recorded action |
33 | --- | --- |
34 | `walk` | Walk six tiles right, then six tiles down. |
35 | `hoe` | Hoe three adjacent reachable tiles. |
36 | `water` | Water the three tiles, after unrecorded UI setup tills them. |
37 | `plant` | Plant the three tiles, after unrecorded farming setup. |
38 | `sleep` | Request sleep and observe the next day. |
39 | `warp` | Walk across a warp into another area. |
40 | `pause` | Toggle the current pause control on and off. |
41 | `joystick` | Drag and release the virtual joystick, including actual touch input on phone. |
42
43 Scenario outcomes are checked against the client state, so a recording that does not perform its intended action fails rather than silently producing a successful result. Current client limitations remain visible: pause is a pause/resume control rather than a full pause menu, and sleep advances the day without a transition animation. Tool effects currently update immediately after the authoritative response; there is no tool-swing animation to measure.
44
45 ## Outputs and measurement meaning
46
47 Each scenario and viewport produces files under `runs/feel/<scenario>/<viewport>/`:
48
49 - `strip.png`: a grid spanning the recorded action. Every cell includes a one-based frame number and action-relative timestamp. Sampling uses a regular every-Nth-frame interval and includes the last frame, with at most 24 cells by default.
50 - `summary.json`: `status: "passed"` after outcome checks, initial/final state, nested `metrics`, recording metadata, and `strip.samples` used by the comparison command.
51 - `video.webm`: the action interval at constant 30 fps.
52 - `frames/frame-000001.png`, etc.: the extracted 30 fps sequence.
53 - `frames/frames.json`: frame filenames and exact action-relative `timeMs` values, starting at zero.
54 - `telemetry.json`: input events, action logs, post-render state observations, and raw compositor frame times. Timestamps use the page's `performance.now()` timebase; subtract its `startMs` to align them to video time zero.
55 - `generated-files.json`: the exact files owned by this recording, used to clean up a previous invocation without deleting unknown files.
56 - `error.json`: the failure reason when an individual recording fails. A failed scenario does not receive a successful summary.
57
58 Lab recordings, browser downloads, process logs, and disposable world state stay under ignored `runs/feel/`. The existing dev scripts can also create their usual ignored Vite/Wrangler caches and logs. Repeated recordings replace only lab-owned outputs; unknown files are preserved and cause an explicit refusal. Raw capture JPEGs are removed after successful video encoding, and each invocation's world state is removed after shutdown. `runs/feel/shutdown.json` records whether every owned process group is gone and its port is closed; if the browser fails before server startup it records `noDevProcessesStarted: true`. Do not add captures or generated strips to Git.
59
60 Normal completion, failures, SIGINT, and SIGTERM enter cleanup. SIGKILL or a host crash cannot execute cleanup: inspect `.runtime/<service>-process.json` and `.recording-lock/owner.json` to identify only this lab's processes before terminating them or removing a stale lock. Never kill unrelated dev servers or remove unknown output files.
61
62 The main `summary.json` measurements are:
63
64 | Field under `metrics` | Meaning |
65 | --- | --- |
66 | `movement.tilesMoved`, `activeDurationMs`, `tilesPerSecond` | Observed travel during actual held-input intervals, excluding waits and coordinate jumps across areas. Individual movements appear in `movement.actions`. |
67 | `movement.firstRenderedMovementLatencyMs` | Time from the actual input event to the first Phaser post-render observation with a changed player position. |
68 | `movement.firstCapturedMovementLatencyMs` | Time from input to the first compositor callback at or after that observed change. |
69 | `tools[].framesToEffect30Fps` | Input-to-observed-tool-effect time, rounded up to 30 fps frames. Millisecond latencies are also available. |
70 | `tools[].animationFrames` | `null`: the current client has no tool animation. |
71 | `frameTimes` | Render interval mean, median, p95, maximum, observed FPS, and estimated dropped opportunities at 60 and 30 fps. |
72 | `capture.rawIntervals` | Compositor callback intervals, independently of game render intervals. |
73 | `capture.slotsWithoutRawFrame`, `slotsWithMultipleRawFrames`, `duplicateOrResampledSlots` | Raw callback occupancy in 30 fps time slots, useful for identifying resampling pressure. These are not exact encoder duplicate counts. |
74
75 Rendered movement latency is an observed state-after-render proxy, not a physical display or pixel-difference measurement. Captured latency identifies a subsequent compositor timestamp; it does not prove the callback contains the changed pixels. Client prediction is included, so these fields are not server round-trip timings. Fields that cannot be measured are `null`.
76
77 Frames per tool use measures input-to-rendered-effect, not the duration of an animation. The exported video is constant 30 fps, so ffmpeg can hold source imagery when no new source frame is available; a 30 fps file does not establish that the game rendered 30 unique frames each second. Inspect both the measured intervals and neighboring video frames when judging a hitch.
78
79 The lab observes the existing world scene using a read-only browser probe and render events. It does not inject player positions, grant inventory, change the clock, or send direct simulation intents. A future explicitly supported read-only hook would make instrumentation less dependent on scene internals; see the recommendations below.
80
81 ## Compare external reference frames
82
83 Keep reference imagery outside this repository. The compare command reads the source files in place and writes both the comparison image and its alignment JSON outside the repository:
84
85 ```sh
86 bun run feel:compare walk /Volumes/Storage/References/walk
87 bun run feel:compare walk /Volumes/Storage/References/walk --fps 60
88 bun run feel:compare walk /Volumes/Storage/References/walk --offset-ms 250
89 bun run feel:compare walk /Volumes/Storage/References/walk --viewport desktop --out /private/tmp/walk-comparison
90 ```
91
92 By default, comparisons go to a unique `/private/tmp/tamarack-feel-comparisons-*` directory, with one `<scenario>-<viewport>.png` and alignment JSON for each recorded viewport. The command prints the output paths. `--viewport` selects a directory name under the scenario's output, such as `desktop` or `iphone16-landscape`. Reference directories, individual reference frames (including manifest paths), and `--out` must be outside the repository; symlinks pointing back into the repository are rejected. Existing output images or JSON files are never overwritten; choose a fresh directory for another comparison. Reference source files are never copied. Because the composite contains reference pixels, the composite must also remain outside the repository.
93
94 Without a manifest, the command naturally sorts PNG/JPEG/WebP filenames and treats them as consecutive frames at `--fps` (default 30), starting at time zero. Use a `frames.json` alongside reference images for keyframes that are not equally spaced:
95
96 ```json
97 {
98 "frames": [
99 { "file": "idle.png", "timeMs": 0 },
100 { "file": "first-step.png", "timeMs": 100 },
101 { "file": "foot-down.png", "timeMs": 233.333 }
102 ]
103 }
104 ```
105
106 Manifest timestamps take precedence over `--fps`. The paired grids are labeled `reference` and `tamarack`. The Tamarack grid uses exactly the frame selections recorded in `summary.strip.samples`; the nearest reference frame is paired with each timestamp. Its timing difference (`Δ`) appears in each caption and alignment JSON. Positive `--offset-ms 250` means Tamarack time 0 corresponds to reference time 250 ms. Reference timestamps must cover the full selected Tamarack interval after that offset; missing coverage causes an explicit error before image creation. The command does not clamp to endpoint frames or silently crop the action. Sparse keyframes can have large timing differences; the command does not interpolate or fabricate frames. Image aspect ratios are preserved inside matched cells.
107
108 ## Add a scenario
109
110 Create a typed data file in `tools/feel/scenarios/`, then import and append it to the list in `tools/feel/scenarios/index.ts`:
111
112 ```ts
113 import type { Scenario } from "./types";
114
115 export const shortWalk: Scenario = {
116 id: "short-walk",
117 description: "A short rightward movement followed by a stop.",
118 steps: [
119 { type: "wait", ms: 250 },
120 { type: "move", direction: "right", tiles: 2 },
121 { type: "wait", ms: 350 },
122 ],
123 outcomes: [{ type: "displacement", x: 2, y: 0, tolerance: 0.25 }],
124 };
125 ```
126
127 Optional `setup` steps run before capture. `steps` is the recorded timeline. The union in `scenarios/types.ts` supports `wait`, `keyDown`, `keyUp`, DOM-control `tap`, measured tile `move`, target-tile `useTile`, and `joystick`. Pair explicit key-down/up steps so inputs are released. Prefer movement targets when exact distances matter; a fixed key-hold duration depends on frame scheduling. Use real control selectors and define an `outcomes` assertion so a changed UI or blocked action causes a useful failure.
128
129 Suggested client hooks, if future scope permits them: an opt-in read-only `window` accessor exposing player position, area, day, pause state, rendered-frame time, and tool-animation phase; stable automation IDs for controls; and lifecycle events marking tool start/end and transition start/end. These should expose observations without providing arbitrary state setters or bypassing normal input.
130