The Test Authoring Agent takes plain-English test steps and turns them into a framework-compliant GitHub PR, complete with real browser validation, compilable Java code, and Maven-verified test execution. Engineers describe what they want to test. The agent handles everything else.
Writing automation code is one of the most time-consuming things a QA engineer does. Not because it's intellectually difficult; the logic is usually straightforward. It's slow because of the overhead: understanding the framework conventions, looking up the right class patterns, getting the right selectors, wiring everything together correctly, debugging compilation errors, running Maven, fixing test failures.
The Test Authoring Agent saves about 70% of the time spent on automation effort: a run takes about 15 minutes, and the engineer never touches Java.
The Big Idea
The agent is triggered in one of two ways: from AI Test Studio's Tests → Automation Code page, or from the CLI by putting a .txt file in the agent's queue and running ./scripts/run-authoring-agent.sh <module>. Either way, the input is the same: a module name, a platform type (Web, API, or Both), and plain-English test steps describing the scenario.
From that input, a five-step pipeline runs entirely without human involvement. By the time it's done, a git branch exists in the Jarvis repository with complete, framework-compliant Java files, Maven has successfully run the tests, and a GitHub PR has been raised with a description of everything that was generated. A Slack notification goes out. The engineer reviews the diff and merges.
The engineer never touches Java. By the time a PR appears in GitHub, the code has been written, validated against a real browser, compiled, tested, and fixed, all without human involvement in any of those steps.
The Test Authoring Agent in AI Test Studio: write plain-English steps with a module and platform (Web / API / Both), reuse a saved draft, or pull cases marked "Pending Automation" straight from TestRail.
The 5-Step Pipeline
Every agent run executes exactly five steps, in sequence. Each step's output feeds into the next. The live console in the UI streams every action in real time: you can watch Claude navigate the browser, see the Java files being written, and follow the Maven output line by line.
Claude reads the plain-English test steps alongside Jarvis/CLAUDE.md, a plain-text conventions file that defines every naming rule, class pattern, and explicit DO/DON'T for the Java framework. This is not a generic code generation prompt. CLAUDE.md tells Claude exactly what package structure to use, how to name every class, what each class type must extend, which helpers are available and how to use them.
The output of Step 01 is a structured JSON generation plan that specifies everything the subsequent steps need: the module name, test type (API / Web / Both), every Java class that needs to be created (Data POJOs, Builders, Helpers, API enum classes, Page Objects, Test classes), their fields and methods, the API endpoints to cover, the UI pages that will be interacted with and their expected element descriptions. No code is written yet, just a precise specification of what will be written.
Before writing a single line of Java, every step is checked against the real application. API steps are validated with plain HTTP calls against the real endpoints, no model call. For UI steps, Claude takes control of a real headless Chromium browser via Playwright MCP, the same browser automation tool used by the Jarvis test framework itself.
Claude navigates to the actual staging environment URL from the generation plan. Then it clicks through the feature flow step by step, exactly as described in the plain-English input. For each step, it locates the relevant DOM element, records a selector that matches exactly one visible element, and marks the step as either STEP_PASSED or STEP_FAILED.
Only confirmed selectors, those that Claude successfully found and interacted with in the real browser, make it into the generation step. This eliminates an entire class of first-run compilation failures: guessed or hallucinated selectors that look plausible but don't exist in the actual DOM. The extra minutes spent here save significantly more time in the debug loop.
With the JSON plan and confirmed selectors in hand, Claude generates the Java files. But it doesn't do this from a blank prompt. Claude reads three inputs simultaneously:
- The JSON generation plan from Step 01
- The confirmed selectors from Step 02
- Nine reference Java files from the Jarvis repository: one real module end to end (SauceDemo's Data, Builder, Helper, API enum, two page objects, and its API and web tests) plus
ApiHelper. SauceDemo is a demo app, but the module is complete and working, so it shows Claude the exact patterns being used, the exact method signatures, the exact annotations.
Claude then writes complete, compilable Java files directly into the Jarvis repository, not as strings to be reviewed, but as actual .java files in the correct package directories. For a Web + API test, this typically includes: a Data POJO, a Builder, a Helper class extending ApiHelper or BasePage, an API enum implementing the ApiDetails interface, Page Object classes extending BasePage, and Test classes extending TestBase with @DataProvider-parameterised test methods. URLs and credentials go into the repo's properties files as keys, never as literals, and a compile gate runs before the step ends.
Maven runs the specific generated test method(s) using a targeted command: mvn test -Dtest=ClassName#methodName. Results are read from test-output/report.json, a structured JSON file written by Jarvis's JsonTestReporter.java after every test run. This gives the agent machine-readable pass/fail status with full failure details, rather than having to parse Maven's console output.
If any test fails (whether from a compilation error, a runtime exception, or an actual assertion failure), the full error output is fed back into Claude's next prompt. Claude now knows exactly what went wrong: the specific line, the specific error message, the specific test that failed. It corrects the code and Maven runs again, up to AUTHORING_FIX_RETRY_COUNT times, and stops early once another attempt can bring nothing new, rather than burning retries on the same error.
The assertions are frozen before the first fix. A fix may change how the test gets there, never what it checks, so the easiest way to make a failing test pass, weakening its assertion, is off the table.
When retries run out, it often signals a structural issue with the plain-English input; the agent still raises the PR, with a NEEDS-REVIEW verdict and the full error context attached, so the engineer can resolve it in minutes rather than starting from scratch. APPROVED is reserved for a test that actually ran and passed.
A git branch is created with a structured name: authoring/<module>-<timestamp>. All new Java files are committed with a descriptive commit message, pushed to the remote GitHub repository, and gh pr create raises a pull request. The PR description includes a summary of what was generated: which classes, what they cover, and whether the tests passed on the first or second attempt.
A Slack notification goes to #qa-reports with a link to the PR. The engineer opens it, reviews the diff, and merges. The entire flow from "click Run Agent" to "PR is open and waiting for review" typically takes about 15 minutes.
The Live Run console: every step streams in real time, with the session's cost, tokens and model calls on top, and a History of past runs you can replay.
The Secret: CLAUDE.md
The entire agent network's code quality depends on one file: Jarvis/CLAUDE.md. Understanding why this file exists, and what it contains, is essential to understanding why the generated code is framework-compliant rather than generic boilerplate.
Without CLAUDE.md, Claude would generate Java that looks plausible but doesn't fit the framework. Classes with the wrong names. Wrong inheritance hierarchies. Made-up helper methods. Annotations in the wrong places. Code that compiles but doesn't integrate with TestNG's DataProvider setup, or doesn't wire up to the BrowserStack configuration, or doesn't follow the Builder pattern the rest of the codebase uses.
CLAUDE.md solves this by being a complete, plain-text specification of the framework, written for an AI reader, not a human one. It defines:
- Package structure: exactly where every class type lives in the directory tree
- Class naming conventions: module-prefix rules for Data, Builder, Helper, API enum, Page Object, and Test class names
- Inheritance rules: which base classes must be extended and when
- Available helpers: what methods exist in ApiHelper, BasePage, and TestBase, and how to use them correctly
- Explicit DO rules: use
@DataProviderfor parameterised tests, use the Builder pattern for all test data construction, extend TestBase in all test classes - Explicit DO NOT rules: do not use
Thread.sleep, do not hardcode URLs, do not create new helper methods when existing ones serve the purpose
The key leverage point
When the framework changes (a new base class is introduced, a helper method is renamed, a new annotation becomes required), updating CLAUDE.md automatically teaches all agents the new pattern on the next run. No prompt changes are needed. No Python code is updated. The conventions live in the Jarvis repository alongside the Java code, version-controlled and reviewable by the whole team as part of normal code review.
This means the framework itself controls the agent's output. The team that owns the Java framework owns the agent's behaviour, directly, through the same files they already maintain.
Audit Trail
Every agent run creates a full audit trail. A session folder is created at agents/test-authoring-agent/audit/<SESSION_ID>/ the moment a run starts, and each step writes its output there as the pipeline progresses: a JSON file for machines, a Markdown report for people, the full prompt and response of every Claude call, and the run's time and cost.
02-validate-api.json.This audit trail is not just a log file; it is exposed via the HTTP server. The UI's History tab can replay any past session: you can see the JSON plan from Step 01, watch the browser validation steps, and review exactly what was generated and why. This is particularly useful when a NEEDS-REVIEW verdict needs investigation: everything Claude tried, and every error it encountered, is preserved in structured form.
Why structured audit files matter
Debugging an AI agent run should not require reading raw log files. By writing each step's output as structured JSON, the system makes every decision inspectable: you can see exactly what plan Claude created, exactly which selectors were confirmed, exactly what errors caused a retry, and exactly what corrections were made. When something goes wrong (and occasionally something does), the audit trail lets you find it in seconds rather than minutes.
Step Cache (For Iteration)
Developing new scenarios, debugging a specific failure, or iterating on the CLAUDE.md conventions means running the agent repeatedly. Steps 01 and 02, the LLM parse and the browser validation, are the most expensive parts of the pipeline: they take about 8–10 minutes combined and incur API costs on every run.
The step cache solves this. With CACHE_STEPS=true (the default), the agent caches the output of Steps 01 and 02 to disk on the first run. When the same input runs again for the same module, it restores from cache, skipping the Claude parse and the Playwright browser session entirely. Execution jumps straight to Step 03 (Generate), which means you can iterate on code generation and the run/fix loop without paying the cost of the first two steps each time.
# The cache is on by default (CACHE_STEPS=true in config/.env) # Run the agent: Steps 01 and 02 restore from cache ./scripts/run-authoring-agent.sh <module-name> # This run only: parse and validate afresh CACHE_STEPS=false ./scripts/run-authoring-agent.sh <module-name> # Or clear one module's cache rm -rf agents/test-authoring-agent/cache/<user>/<module-name>/
The cache is keyed by user and module, and is only reused while the input file is unchanged: edit the steps and Steps 01 and 02 run fresh. You can have multiple modules cached simultaneously. Clearing the cache for one module triggers a fresh Step 01 and Step 02 on the next run for that module, without affecting cached state for others.
It pays off most while iterating: build the agent, run it, see what it generates, update CLAUDE.md or the input, run it again, fast. Because the cache is only reused while the input is unchanged, an edited scenario always gets a fresh parse and a fresh browser check; CACHE_STEPS=false forces one regardless.
Next: The Test Triaging Agent
The Authoring Agent writes the code. The Triaging Agent keeps your CI signal clean, classifying every failure as a product bug or automation issue automatically, with an adversarial review step to catch misclassifications.