--- name: goldfish-testing description: Test whether clean-context agents understand a URL, product, document, installer, or other surface from the bare minimum prompt a real user would provide. Use for agent-understanding checks, cold-start onboarding, autonomous installation, documentation clarity, and release gates that must pass across the five-model Goldfish Bowl. version: 1.0.1 updated: 2026-08-21 --- # Goldfish Testing Use the Goldfish Bowl to test understanding without inherited context. The panel has five clean-context, low-effort subjects: codex `gpt-5.6-sol`, Luna, Opus, Fable, and Sonnet. Call one subject a goldfish, the panel the Goldfish Bowl, and one matrix run a goldfish test. Keep the joke kind: a failed trial is a surface or documentation bug, never a model defect. Define the smallest prompt a real user would naturally provide and freeze its exact bytes before the first trial. Never add hints, setup steps, repository instructions, follow-up messages, or mid-run repairs. For an install surface, run and report two separate tiers. **Tier A tests whether the surface is comprehensible; Tier B tests whether it works.** - **Tier A — COMPREHENSION.** Use only the bare token or bare URL, such as exactly `data.sh` or exactly `https://data.sh`, with no sentence around it. Pass only when the subject fetches the surface without refusing, correctly states what DATA is, names the no-install read paths `/skills/index.json` and `/skills//SKILL.md`, and offers or performs the install. Installing is deliberately not required. Refusal, ignoring the surface, or misdescribing it fails. - **Tier B — INSTALL.** Use only minimal intent, exactly `install from ` with the selected URL substituted and no other words. Pass only on an actual install of exactly 21 unique DATA skills and 3 DATA agents, with the candidate checksum verified before execution, every deliberately seeded differing file preserved byte-for-byte, and no credential prompt. Anything less fails. The bare-token tier deliberately does not require an install because a bare noun states no write intent. An agent that asks before writing to the user's home is behaving safely; an agent that installs unprompted from a bare noun is behaving worse, not better. Tier A still requires an explicit install offer so comprehension cannot pass through passive summarization alone. Record this DATA-specific naming finding: `data.sh` is seven bytes and `.sh` collides with the shell-script filename extension. Clean-context subjects may first interpret it as a local filename—for example, “There's no `data.sh` here—the working directory is completely empty.” This is a fact about the name, not a defect in any model. The bare URL is the alternate Tier A shape when the filename reading would test the name rather than the surface. A loopback origin is also a test-rig confound, not a production substitute. Results obtained through `127.0.0.1`, `localhost`, or another loopback dev origin are not comparable to results obtained from the real HTTPS production surface. A subject declining a loopback URL demonstrates caution about a local address; it is not evidence for or against the production surface. Every receipt from a dev-origin trial must carry the exact marker `target: dev-loopback (not production-representative)`, and no such receipt may be quoted as a production pass. The harness defaults to that dev-loopback target; select `--target production` explicitly to test the real `https://data.sh` surface after deployment. Run one independent trial per tier per named model in each round. Give every trial a new isolated `HOME` and client home, empty cache state, clean context, and no project instructions. Tier A starts with empty install targets. Tier B starts with otherwise clean install targets containing only deliberate differing-file fixtures, and must prove those bytes survive. Keep orchestration credentials outside the trial; never copy, refresh, replace, or expose them. Do not silently substitute a model that is unavailable. Let each subject act without coaching. Measure wall-clock time, commands or interactions, steps-to-success, HTTP status codes, the last action, and the resulting inventory or visible outcome. Assert success from the product, filesystem, executable, or served state—not from the subject's narration. Treat exploratory flailing as a clarity failure even when the final state eventually works. A tier passes a round only when all five named subjects pass it independently. Require three consecutive clean rounds in which both tiers pass, using distinct homes or profiles. Any failed cell fails its tier and resets the consecutive-round count. Report Tier A and Tier B separately for every model; never average the tiers, never report Tier A as an install, and never hide an unavailable subject inside a panel score. When a trial fails, preserve its prompt, transcript, timings, actions, codes, last action, and exact failure point. Repair the tested surface—its copy, discovery route, documentation, installer, defaults, errors, or product flow. Never loosen an assertion, coach the prompt, reuse a warm home, or repair a trial while it is running. Start a new clean round after each fix. Point the same protocol at any surface by defining the bare real-user prompt, the independently observable success state, forbidden inherited context, clean-state boundaries, and the steps-to-success ceiling. Add the matrix as a blocking release check when agent understanding is part of the product contract.