Writing Scenarios

Scenarios are the core of AXIS testing. Each scenario defines a task for the agent, criteria for judging success, and optional setup and teardown steps. Well-written scenarios produce consistent, meaningful scores.

Anatomy of a Scenario

A scenario is an object with a name, prompt, rubric, and optional lifecycle hooks. The same shape applies whether you author it as JSON, a JavaScript or TypeScript module, or an inline entry in axis.config.{js,ts}. See Authoring Scenarios below for the available formats.

Authoring Scenarios

Scenarios live in the configured scenarios directory or are listed inline in axis.config.{js,ts}. The three formats below are interchangeable for the shape itself; pick the one that matches how you want to manage the scenario.

{
  "name": "Debug and fix a broken script",
  "prompt": "There is a JavaScript file at src/add.js that has a bug. Find it, fix it, and verify the fix by running the test.",
  "rubric": [
    { "check": "Agent identified the bug (subtraction instead of addition)", "weight": 0.3 },
    { "check": "Agent fixed the bug so add(a, b) returns a + b", "weight": 0.4 },
    { "check": "Agent ran the test and it passed", "weight": 0.3 }
  ],
  "setup": [
    { "action": "run_script", "command": "mkdir -p src && echo 'function add(a,b) { return a-b; }\nmodule.exports = { add };' > src/add.js" },
    { "action": "run_script", "command": "mkdir -p test && echo 'const {add} = require(\"../src/add\");\nconsole.log(add(2,3) === 5 ? \"PASS\" : \"FAIL\");' > test/add.test.js" }
  ],
  "teardown": [
    { "action": "run_script", "command": "rm -rf src test" }
  ]
}

// scenarios/refactor-add.ts
import type { ScenarioInput } from "@netlify/axis";

export default {
  name: "Refactor a buggy add() function",
  prompt: "There is a bug in src/add.js. Find it, fix it, and verify the fix.",
  rubric: [
    { check: "Agent identified the bug", weight: 0.3 },
    { check: "Fix is correct", weight: 0.4 },
    { check: "Verification was run", weight: 0.3 },
  ],
} satisfies ScenarioInput;

// scenarios/dynamic.ts
export default async () => ({
  name: "Dynamic scenario",
  prompt: "Run task for " + (process.env.TARGET_ENV ?? "staging"),
  rubric: [{ check: "Task ran cleanly" }],
});

// axis.config.ts
import type { AxisConfig } from "@netlify/axis";
import refactorAdd from "./scenarios/refactor-add.js";

export default {
  scenarios: [
    "./scenarios",          // directory walked for .json/.ts/.js scenarios
    refactorAdd,             // module-imported scenario
    {                        // pure inline literal; key required
      key: "smoke-test",
      name: "Smoke test",
      prompt: "Verify the build runs cleanly.",
      rubric: [{ check: "Build succeeded" }],
    },
  ],
  agents: ["claude-code"],
} satisfies AxisConfig;

Writing Effective Prompts

The prompt is what the agent sees as its task. The quality of your prompt directly affects the consistency and usefulness of your scores.

• Be specific about what to do. "Fix the bug" is vague. "Find and fix the bug in src/add.js where subtraction is used instead of addition" gives the agent a clear target. Specific prompts produce more consistent results across runs.
• Specify the expected output. If you want a file created, say what it should be called and what it should contain. If you want a test to pass, say which test command to run. Leaving the end state implicit forces the judge to guess what "success" means.
• Scope appropriately. A prompt that asks the agent to "set up a full CI/CD pipeline" tests many things at once and makes it hard to isolate what went wrong. Smaller, focused scenarios produce more actionable scores.
• Avoid giving the agent the answer. The point of testing is to observe how the agent discovers and solves the problem. If you tell the agent exactly which line to change, you are testing its ability to follow instructions, not its ability to debug.

Designing Rubrics

The rubric defines what "success" means. A judge LLM reads the agent's full transcript and evaluates each check on a 0 to 10 scale. Well-designed rubrics produce scores that reflect real quality differences.

Start simple: string rubrics

The simplest rubric is a plain string. The judge reads the transcript and gives a single 0 to 10 score for how well the agent met the description.

{
  "name": "Create an Express server",
  "prompt": "Create a working Express server that listens on port 3000.",
  "rubric": "The agent should create a working Express server on port 3000"
}

String rubrics work well for simple scenarios where you just want a holistic pass/fail judgment. The downside is that you get a single score with no visibility into what went right or wrong.

Add structure: check arrays

For more granular scoring, use an array of checks. Each check is evaluated independently, so you can see exactly which criteria the agent met and which it missed.

"rubric": [
  { "check": "Server starts on port 3000" },
  { "check": "GET / returns a 200 response" },
  { "check": "Server has error handling middleware" }
]

When no weight is specified, AXIS distributes weight equally across all checks. In this example, each check is worth one-third of the Goal Achievement score.

Control importance: weighted checks

Add weight to each check to control how much it contributes to the score. This is the recommended approach for most scenarios because it lets you express which outcomes matter most.

"rubric": [
  { "check": "Server starts on port 3000", "weight": 0.4 },
  { "check": "GET / returns a 200 response", "weight": 0.3 },
  { "check": "Server has error handling middleware", "weight": 0.3 }
]

In this example, the server starting is weighted highest because it is the core outcome. You can also mix weighted and unweighted checks -AXIS distributes the remaining weight equally across any checks without an explicit weight.

Writing good checks

• Make checks observable. The judge evaluates checks by reading the transcript and examining the workspace. "Agent understood the problem" is hard to verify. "Agent modified src/add.js to use addition" is concrete and observable.
• One assertion per check. "Agent fixed the bug and ran the test" is two things. If the agent fixes the bug but skips the test, it is unclear how to score this check. Split compound assertions into separate checks with their own weights.
• Weight by importance. The core outcome (did it work?) should carry more weight than peripheral concerns (did it clean up after itself?). If every check has equal weight, a cosmetic failure impacts the score as much as a functional failure.

Setup and Teardown

Setup actions run before the agent starts. Use them to create the starting state that the scenario depends on: files to edit, databases to seed, servers to start.

Teardown actions run after scoring completes. Use them to clean up resources that should not persist between runs.

"setup": [
  { "action": "run_script", "command": "cp -R \"$AXIS_CONFIG_DIR/scenario-fixtures/my-project/.\" ." }
],
"teardown": [
  { "action": "run_script", "command": "rm -rf /tmp/my-project-artifacts" }
]

Lifecycle environment variables

Setup and teardown scripts run with the workspace as their working directory. The following environment variables are available inside run_script commands:

The AXIS_CONFIG_DIR variable enables the recommended pattern of keeping fixture data in a versioned scenario-fixtures/ directory next to axis.config.json, then copying just the bits a scenario needs into its workspace:

scenario-fixtures/
  my-project/
    package.json
    src/index.js
    .axis/baselines/main.json   # hidden files copied via the trailing /. on cp -R

scenarios/
  my-scenario.json              # setup: cp -R "$AXIS_CONFIG_DIR/scenario-fixtures/my-project/." .

Capturing notes for the report

Setup and teardown each receive their own $AXIS_OUTPUT file path. Anything written to that file is captured as GitHub-flavored markdown and surfaced as a "Setup notes" or "Teardown notes" panel in the agent's expanded detail row in the HTML report. Multiple actions in the same phase share one output file, so each script can append.

"teardown": [
  {
    "action": "run_script",
    "command": "{ echo '## Teardown'; echo; if [ -f summary.md ]; then echo \"- **summary.md**: $(wc -c < summary.md | tr -d ' ') bytes\"; else echo '_no summary.md was written_'; fi; } >> \"$AXIS_OUTPUT\""
  }
]

The notes panel for that run renders the markdown — bullet list with bold filenames and the captured byte count. Useful patterns:

• Snapshot the workspace after the agent finishes (file list, sizes, line counts).
• Probe an external resource the agent created (HTTP status, DB row counts).
• Record diagnostic context (commit SHAs, env versions) that scoring should not see but a human reviewer will want.

Common patterns

• Create test fixtures: Use setup to write files that the agent will need to read, edit, or debug.
• Seed data: Populate a database or create configuration files the agent should work with.
• Initialize a project: Clone a repo, install dependencies, or set up a specific project state.
• Clean up: Remove temp directories, stop background processes, or reset state in teardown.

Artifacts

Use artifacts to capture files the agent produced during the run. Each entry is a glob pattern relative to the scenario's workspace; matching files are copied into the report after teardown so you can inspect them later without keeping the workspace around.

{
  "name": "Generate API summary",
  "prompt": "Fetch the docs and write summary.md.",
  "rubric": "summary.md exists and covers the docs.",
  "artifacts": [
    "summary.md",
    "*.log",
    "out/**",
    "screenshots/*.png"
  ]
}

Captures run after teardown, so any cleanup scripts you write don't need to dance around the files you want to keep. You can also set an artifacts array at the top level of axis.config.json to capture the same patterns from every scenario; the two lists are merged.

Glob syntax

• * matches any characters within a single path segment.
• ** matches any number of path segments (including zero).
• ? matches a single character within a segment.
• [abc] / [a-z] match character classes; [!abc] negates.

Where captured files live

Each captured file is copied to disk under the report directory, preserving its path relative to the workspace:

.axis/reports/<id>/scenarios/<scenario-key>/<agent>/artifacts/...

The same files are also embedded (base64) into the report manifest so the HTML report can preview and download them entirely client-side — no local server needed, even when the HTML report is opened directly from disk.

Viewing artifacts in the HTML report

Each scenario row gets an Artifacts panel when files were captured. Click Show artifacts to reveal a collapsible file tree, the eye icon to open a modal preview (text-like files render as text, images as images), or the download arrow for a single file. Download all (.zip) bundles every captured file into one archive in the browser.

Scenario Organization

The filename (without its extension) becomes the scenario key used in reports, CLI commands, and baseline comparisons. Nested directories create namespaced keys.

• scenarios/hello-world.json → key hello-world
• scenarios/cms/create-post.ts → key cms/create-post
• scenarios/api/auth/login.js → key api/auth/login

Use directories to group related scenarios. Agents can be configured to run only specific groups using glob patterns in the agent configuration:

{
  "agent": "claude-code",
  "scenarios": ["cms/*", "api/*"]
}

Agent-specific scenarios

Use the agents field in a scenario to restrict which agents run it. This is useful when a scenario depends on capabilities specific to one agent, or when you want to test different agents on different tasks.

{
  "name": "Use Claude Code MCP integration",
  "prompt": "Use the filesystem MCP server to list files in /tmp",
  "rubric": "Agent successfully used the MCP filesystem tool",
  "agents": ["claude-code"]
}

Variants

Variants let you run the same scenario under different configurations -different skills, MCP servers, prompts, or agent restrictions -without duplicating the entire scenario file. When variants is defined, the base scenario becomes a template: only the variants execute, each inheriting all fields from the parent. To also run the unmodified scenario as a control, add a variant with no overrides (the baseline pattern).

{
  "name": "Create a blog post",
  "prompt": "Create a new blog post titled 'Hello World' on the CMS.",
  "rubric": [
    { "check": "Blog post was created successfully", "weight": 0.5 },
    { "check": "Title matches 'Hello World'", "weight": 0.5 }
  ],
  "variants": [
    {
      "name": "with-netlify-mcp",
      "mcp_servers": {
        "netlify": { "type": "http", "url": "https://mcp.netlify.com" }
      }
    },
    {
      "name": "with-custom-skill",
      "skills": ["./skills/blog-helper"]
    },
    {
      "name": "alt-prompt",
      "prompt": "Create a draft blog post titled 'Hello World' without publishing."
    }
  ]
}

This produces three scenario keys: create-post@with-netlify-mcp, create-post@with-custom-skill, and create-post@alt-prompt. Each variant inherits the parent's rubric, setup, and other fields, then applies its own overrides. The variant name is appended to the scenario key with an @ separator.

Variant fields

Only name is required on a variant. All other fields are optional and inherit from the parent when omitted.

Example Scenarios

{
  "name": "Create a README",
  "prompt": "Create a README.md file for a Node.js project called 'my-api'. Include a title, description, install instructions, and usage example.",
  "rubric": [
    { "check": "README.md exists" },
    { "check": "Contains a project title and description" },
    { "check": "Contains npm install instructions" },
    { "check": "Contains a usage or getting started example" }
  ]
}

{
  "name": "Fix failing test",
  "prompt": "The test in test/math.test.js is failing. Find the bug in src/math.js, fix it, and verify the test passes.",
  "rubric": [
    { "check": "Agent identified the root cause", "weight": 0.2 },
    { "check": "Agent fixed src/math.js correctly", "weight": 0.4 },
    { "check": "Agent ran the test and it passed", "weight": 0.4 }
  ],
  "setup": [
    { "action": "run_script", "command": "mkdir -p src test" },
    { "action": "run_script", "command": "echo 'exports.multiply = (a, b) => a + b;' > src/math.js" },
    { "action": "run_script", "command": "echo 'const {multiply} = require(\"../src/math\"); console.assert(multiply(3,4) === 12, \"Expected 12\"); console.log(\"PASS\");' > test/math.test.js" }
  ]
}

{
  "name": "Add API endpoint with tests",
  "prompt": "Add a GET /api/health endpoint to the Express app in src/app.js. It should return { status: 'ok', uptime: process.uptime() }. Write a test in test/health.test.js.",
  "rubric": [
    { "check": "GET /api/health endpoint exists", "weight": 0.3 },
    { "check": "Returns JSON with status and uptime", "weight": 0.3 },
    { "check": "Test file exists and covers the endpoint", "weight": 0.2 },
    { "check": "All tests pass", "weight": 0.2 }
  ],
  "setup": [
    { "action": "run_script", "command": "npm init -y && npm install express" },
    { "action": "run_script", "command": "mkdir -p src test" }
  ],
  "teardown": [
    { "action": "run_script", "command": "rm -rf node_modules package.json src test" }
  ]
}

{
  "name": "Deploy a site",
  "prompt": "Deploy the project in the current directory to production.",
  "rubric": [
    { "check": "Site was deployed successfully", "weight": 0.5 },
    { "check": "Agent confirmed the deploy URL", "weight": 0.3 },
    { "check": "No errors in the deployment log", "weight": 0.2 }
  ],
  "setup": [
    { "action": "run_script", "command": "npm init -y && echo '<h1>Hello</h1>' > index.html" }
  ],
  "variants": [
    {
      "name": "baseline"
    },
    {
      "name": "with-mcp",
      "mcp_servers": {
        "netlify": { "type": "http", "url": "https://mcp.netlify.com" }
      }
    },
    {
      "name": "with-deploy-skill",
      "skills": ["./skills/deploy"]
    }
  ]
}