All Posts
Case Studies

What We Learned Running Live Products With an AI Agent Fleet

Practical lessons from running Dayze, CFA Mock Exam, and Coincess with an AI agent fleet in production—GitHub metrics measured only on 2026-10-24 case studies.

6 min read

Phi Digital does not only sell AI agents. We run live products with an AI agent fleet and publish what we can measure. This post summarizes lessons from that operation—using only GitHub counts measured 2026-10-24 on the public case studies—and what those lessons mean if you are considering agents for your own backlog.

Primary sources:

No client logos, no invented testimonials, no unpublished metrics.

The measured numbers (2026-10-24)

ProductWindowMerged PRsAgent signal
DayzeJul 4 – Oct 24, 20263,568At least 2,151 (60%+) from agent-named branches
CFA Mock ExamAug 22 – Oct 24, 2026124290 of 412 commits on main carry an AI agent co-author
CoincessSep 11 – Oct 23, 20263013 from agent-named branches

These are production products Phi owns end to end—not lab demos. Counts come from GitHub (gh pr list --state merged and co-author trailers), measured 2026-10-24. Agent-authored PRs are a floor: agent work committed under human-named branches is not always counted (fleet measurement notes).

Lesson 1 — Throughput needs a queue, not a chat window

The fleet pattern is boring on purpose:

  1. Work is filed as issues with a brief and acceptance checks
  2. Agents pick up issues within per-machine capacity limits on isolated branches
  3. They open pull requests linked to the issue
  4. CI runs; failures become dedicated fix runs
  5. Risky changes wait for human approval

That is how you get thousands of merged PRs on Dayze without turning production into an unreviewed paste bin. If your team only “asks ChatGPT in Slack,” you do not have a fleet—you have suggestions.

Lesson 2 — CI and ownership are the real product

Merging is not deploying blindly. On CFA Mock Exam, main is protected (PR, passing CI, owner approval) as of Oct 14, 2026. Dayze merges to main ship production because Phi owns product, infrastructure, and self-hosted CI runners. Agents accelerate the middle of the loop; humans still own the gates.

For a client engagement, the equivalent is: your issue tracker, your repos, your CI, your approvers. The fleet case study frames exactly that handoff.

Lesson 3 — Measure floors, publish sources

Dayze’s 60%+ figure is “from agent-named branches,” explicitly a lower bound. CFA’s 290/412 is co-author trailers on main. Coincess shows 13/30 from agent-named branches in a younger repo. Publishing method beats publishing vibes. Prospects evaluating vendors should ask: What did you count, when, and what did you exclude?

Lesson 4 — Agents catch failures—and create work that needs review

Case studies document real fixes, not only happy paths:

  • Dayze PR #488: get_events ignored query/date filters
  • Dayze PR #3325: duplicate tool calls could skip audit rows
  • CFA PR #24: checkout_paid analytics missed on Stripe return
  • Coincess PR #41: never select the builder-fee wallet as the user account
  • Coincess PR #19: AI coach broke when a Gemini model was retired

Security reviews also closed access gaps before exploitation (described at category level on the case studies). The point for buyers: budget for evaluation harnesses, monitoring, and human review—see AI Agent vs Chatbot on action risk.

Lesson 5 — Different products, same pattern, different maturity

Dayze is a personal context layer with MCP OAuth, multi-model routing, and large test surface. CFA Mock Exam is a paid exam product with Stripe and server-side grading. Coincess is a crypto PWA with swaps, perps, and push monitoring. Volume differs; the fleet pattern—issues, branches, CI, approval—repeats.

You do not need Dayze-scale PR counts to benefit. You need one workflow with clear acceptance checks. Pricing for that path starts on services: Assessment $2,500 (credited), Single Agent from $5,000, Suite from $12,000, retainer from $2,000/month.

What this is not claiming

  • Not that every business should merge thousands of PRs per quarter
  • Not that agents replace engineering judgment
  • Not that your ROI equals our PR counts (use ROI math for labor savings)
  • Not unpublished customer case studies—these are Phi’s own products

How to apply this tomorrow

  1. Pick one backlog lane with crisp acceptance tests
  2. Require PR + CI for agent output
  3. Gate money, privacy, and infra changes for humans
  4. Count merges and failures the same way every week
  5. If you want help designing the lane, start with an assessment or contact

Small businesses that only need site Q&A should look at the Website Agent instead of a coding fleet.

FAQ

Are these numbers still updating live?

The figures in this article are the measured 2026-10-24 snapshots from the case study pages. The agent fleet board also shows live operational counters; treat those as separate, board-sourced metrics.

Do you sell the same fleet setup to clients?

Yes—as a pattern: your tracker, agents in your repos, your CI, your approvals. Scope and price come from assessment and quote, not from copying Dayze’s PR volume.

Why publish engineering metrics at all?

Because production AI agent consulting should be inspectable. Counts with dates beat adjectives.

Bottom line

An AI agent fleet is a disciplined delivery system—issues, isolated branches, CI, and human gates—applied at volume. Phi’s own products (Dayze 3,568 merged PRs; CFA Mock Exam 124; Coincess 30; measured 2026-10-24) show what that looks like when you own the stack. Start smaller, measure honestly, and buy scope—not hype.

How the fleet compares to a single client agent

Client projects rarely need three Macs of parallel coding agents on day one. They need the discipline the fleet encodes: a queue, isolated changes, automated checks, and a human gate for risk. A Single Agent build (from $5,000) that triages support mail with a preview queue is closer to CFA Mock Exam’s server-side grading change—one clear workflow with tests—than to Dayze’s multi-thousand PR cadence.

When buyers ask for “what you do for yourselves,” translate:

  • Issue brief → acceptance criteria in the SOW
  • Agent branch → isolated integration environment
  • CI → eval suite on golden tickets or sample documents
  • Human approval → your ops lead for refunds, deploys, or PII exports

That translation is what the AI Assessment is for: map your backlog and systems onto a gated delivery plan, then credit the fee if you build.

Guardrails we refuse to skip

The fleet overview stresses metadata-only boards, spend caps, stop flags, and second-agent read-only reviews on risky diffs. Client builds inherit the same posture even at smaller scale. If a vendor cannot explain how writes are gated, treat the demo as entertainment.

For conversation-only needs on a marketing site, skip the fleet conversation entirely and use the Website Agent.

Topics

ai agent fleetai agents in productionsoftware development with ai agents

Related insights

Get AI insights delivered to your inbox

Weekly perspectives on AI agents, automation, and building systems that work in production.

No spam. Unsubscribe anytime.