SSSahil Sharma
Email
← Back to Thoughts
Sun, 28 Dec, 2025

Do AI Test Agents Really Belong in CI Pipelines? An Honest Take

A practical take on where AI Test Agents fit in CI/CD pipelines, based on real-world usage with Playwright Planner, Generator, and Healer.

The rush toward AI in CI

I see many teams rushing to plug AI Test Agents directly into CI/CD and expecting magic.

Here’s my honest opinion after actually using Playwright Planner, Generator, and Healer in production.

The core truth

Planner and Generator are instruction-driven tools.

They operate based on:

  • Explicit instructions
  • Repository context

They generate or update specs only when guided — not continuously or blindly.

If that’s the case, why put them in CI at all?
Why not run them locally, generate specs once, and let CI do its job?

Visual overview

AI Test Agents usage in CI pipelines with controlled stages

The short answer

Yes — for many teams, local-only agent usage is enough.

A clean and valid setup looks like this:

  • Generate specs locally
  • Review them
  • Commit them
  • Let CI run deterministic tests

This setup is simple, predictable, and works well for small to medium systems.

So when does CI integration actually make sense?

CI integration is not about necessity.

It’s about:

  • Scale
  • Consistency
  • Control

I use agents in CI only for controlled spec generation, never for release execution.

How I structure AI agents in CI

Agents never run inside release-critical execution paths.

Instead, I separate responsibilities into three clear stages:

stages: [generate, test, heal]

generate:
  script:
    - npx playwright test --agent=planner
    - npx playwright test --agent=generator
  rules:
    - if: $CI_MERGE_REQUEST_ID

test:
  script:
    - npx playwright test

heal:
  when: manual
  allow_failure: true
  script:
    - npx playwright test --agent=healer

What this structure enforces

  • Planner & Generator only create or update spec files
  • Generated specs go through PRs and human review
  • All specs (manual + AI-generated) run via npx playwright test
  • Healer exists, but only as a manual, non-gating job

Guardrails I never break

❌ AI can draft YAML or pipeline changes, but it can never auto-merge
❌ AI can propose test code, but humans always approve
❌ Healer gets one attempt, then escalation to engineers
❌ No self-healing in production or release pipelines

AI suggests. CI enforces. Humans decide.

The real insight

CI doesn’t need AI.
Teams do.

Where AI test agents actually help

AI Test Agents are excellent accelerators for:

  • Boilerplate specs
  • CRUD flows
  • Selector drift after UI refactors

But they are terrible autopilots for:

  • Business logic
  • Authentication & payments
  • Security-sensitive flows
  • Release decisions

Bottom line

If your product is small and stable, local-only agent usage is enough.

CI integration makes sense only when you need:

  • Standardized generation across teams
  • Traceability via PRs
  • Controlled and repeatable workflows

This isn’t about hype or replacing engineers.
It’s about using AI deliberately — without breaking the guarantees CI/CD exists to provide.