Phase two, on the same install
What it takes to make AI work here is what it takes to train AI on how you work.
Every serious attempt at AI of your own runs into the same wall, and it is never the model.
The order is forced
Nothing new gets installed between one phase and the next.
The reason to care about the order is money. Treat phase two as a separate programme and you buy the capture, the governance and the environment twice, and the second purchase never gets approved.
The same five pieces
Nothing new gets installed to do the second job.
Read each row across. The piece on the left is running for you today, and the piece on the right is the same one doing the training job later. Everyone else in this market builds a copy of your business and learns in the copy, even when the copy runs in your own cloud.
How an attempt is run safely
An agent rehearses against your real systems, and the write it would have made is caught.
An evaluation run reads from your live systems, so the state is genuinely today's. Its writes are intercepted and recorded rather than committed, so what gets scored is the action the agent would actually have taken.
Recorded replay
mock_all
Replays a recorded run with canned responses.
Real reads
real_reads
Real tools against real data with writes refused outright.
Real reads, captured writes
live_reads_captured_writes
The same real execution, except each intended write persists to the result instead of happening.
Write refusal during a training run is a single enforced seam rather than a convention, including a floor that refuses any write on execution paths where the capture intercept does not exist.
What gets scored
The trajectory, including the write, rather than the paragraph at the end.
Evaluation tools score an output against an expected string, which tells you almost nothing about an agent that can reach a defensible answer through a sequence you would never have allowed.
Tool selection
Did it reach for the right tools for this task, or improvise with whatever was nearest.
0.25
Write actions
The captured write, compared against what should have been written. The score most people never think to build.
0.20
Parameter accuracy
Right tool, right arguments. An almost correct amount or the wrong account is a failure here rather than a rounding error.
0.20
Output quality
A model judging the answer, pinned to a continuous scale when the run is gating a promotion, because a binary judge at a decision boundary is noise.
0.20
Tool order
Sequence matters when one step authorises the next. Checking eligibility after issuing the credit is a different act.
0.15
Around the scores sit two things worth naming.
The two questions anyone with an ML background asks
Both are fair, and the honest answers are better than confident ones.
How many episodes will you actually have?
Far fewer than a lab, and we do not need lab volume, because the unit here is not a rollout.
What nobody doing real enterprise work will ever have is the volume for training from scratch, because every episode is a real business action with a real cost.
What supplies the reward?
Today it is human.
Tying a decision to what the business measured afterwards is the next layer, and we will say plainly that it is a causal inference problem rather than a plumbing one: delayed, confounded, and cleanly identifiable only where a business will tolerate a holdout.
Stated plainly
Three things this is not, so nobody discovers them in month four.
Start with the phase that pays for itself.
The useful first conversation is about the work in front of your team this quarter.