Skip to content

M2.3 — The built-in tests

What you'll learn

  • The ready-made Gauntlet tests you can run without writing any C#, and what each verifies.
  • Which built-in test to reach for given a QA goal.
  • Why the built-ins double as both useful coverage and worked examples for your own tests.

How it applies (QA)

The built-ins are the fastest path to real Gauntlet coverage and the lowest-risk way to validate your pipeline (build → install → launch → report) independent of any custom test logic. If UE.BootTest is red, the problem is the build or the plumbing — proven before you write a line of your own.

Concepts

Gauntlet ships several pre-implemented test nodes that drive common workflows. You select them by name with -test=.

Built-in test What it does Reach for it when…
UE.BootTest Launches a packaged/target build and confirms it boots and reaches a running state without crashing. You want a smoke gate on the shipping artifact for a platform.
UE.EditorBootTest Same idea, but boots the editor rather than a game build. You want to catch editor-startup breakage (a bad default asset, a plugin that won't load).
UE.EditorAutomation Launches the editor and runs the in-engine automation tests inside it. You have in-engine automation/functional tests and want Gauntlet to run them in CI and report.
UE.TargetAutomation Launches a target/packaged build and runs the in-engine automation tests inside it. You need those same tests validated on the cooked build, not just in-editor.
UE.Networking Exercises a networked (client/server) configuration. You want a baseline multiplayer connectivity check.
UE.ErrorTest Deliberately produces errors/failures. You're validating that your reporting correctly catches and surfaces failures (a test of the tests).
UE.PLMTest Drives [PLM] transitions — suspend/resume/terminate. You're checking console/mobile lifecycle handling.

Boot vs. EditorBoot — the cheapest signals

UE.BootTest and UE.EditorBootTest are the two you'll run first and most. They are shallow by design — "does it start" — but that shallowness is the point: they isolate the plumbing. A green boot test proves the build exists, installs, launches, and survives startup on the target. Only then is a failing deeper test plausibly about your logic rather than the pipeline.

The two Automation runners are launchers

UE.EditorAutomation and UE.TargetAutomation are the wiring from M1.2 made concrete: they don't contain test assertions themselves — they run the in-engine automation tests and let Gauntlet collect and report the results. You typically pass a filter to choose which in-engine tests run (e.g. a test-name prefix or group), so you can scope a CI job to a subset.

ErrorTest — validate your safety net

UE.ErrorTest exists to fail on purpose. Its value is meta: it confirms your pipeline actually notices a failure — that a red test turns into a red CI job, a surfaced log, a notification. A reporting path that silently swallows failures is worse than no test; UE.ErrorTest is how you prove yours doesn't.

Pitfall: a green boot test is necessary, not sufficient

"Boot test passes" means started without crashing — not playable, not correct, not content-complete. Treating boot-test green as "the build is good" is a classic over-read. It gates the obvious; it doesn't certify quality.

Worked example

A QA owner is standing up nightly Gauntlet on a new console. Sensible escalation using only built-ins:

  1. UE.EditorBootTest (Win64) — does the editor even start on this CL? Cheapest signal, catches asset/plugin breakage early.
  2. UE.BootTest (console, Development) — does the cooked console build boot on a devkit?
  3. UE.ErrorTest — once green, prove the pipeline reports red correctly before trusting it.
  4. UE.TargetAutomation (console) — now run the real in-engine suite on the target build.
  5. UE.PLMTest (console) — add lifecycle coverage specific to the platform.

Each step adds one variable. When step 4 first goes red, steps 1–3 having passed tells you it's the suite, not the build or the reporting.

Exercise 1 — Pick the built-in

Name the single best built-in for each goal:

  1. Catch a plugin that fails to load on editor startup.
  2. Prove your CI turns a failing test into a failing job.
  3. Run your existing in-engine functional tests on the cooked PS5 build.
  4. Confirm the Switch build survives a suspend/resume.
  5. Baseline that a client can connect to a dedicated server.

Exercise 2 — Order the bring-up

You're given a brand-new platform with an unproven pipeline. Put these in the order you'd enable them and justify the first and last: UE.TargetAutomation, UE.BootTest, UE.ErrorTest, UE.PLMTest.

Self-check — answers

Exercise 1: 1 UE.EditorBootTest, 2 UE.ErrorTest, 3 UE.TargetAutomation, 4 UE.PLMTest, 5 UE.Networking.

Exercise 2: UE.BootTest first — it isolates the build/plumbing with the fewest moving parts; nothing deeper is trustworthy until boot is green. UE.ErrorTest next — prove the reporting catches red before relying on it. Then UE.TargetAutomation (real suite on target), and UE.PLMTest last — lifecycle coverage is meaningful only once the build reliably boots and runs. (Reasonable to swap the middle two; the first and last are the load-bearing claims.)

Done when

  • [ ] You can name each built-in and the one thing it verifies.
  • [ ] You can pick the right built-in for a given QA goal.
  • [ ] You can explain why UE.EditorAutomation/UE.TargetAutomation are launchers, not assertion sets.
  • [ ] You can justify UE.ErrorTest as a test of your reporting path.
  • [ ] You can articulate why "boot green" ≠ "build good."

Next: M2.4 — Reading results.