Three days from the week of 21 Sep, with Sally, the orchestrator agent that runs the other agents in my setup.

I gave Sally a data file and told her it was the latest export.
The job was to redo an analysis we had published in August.

I asked her to “get alignment before you start the execution.”

Then it went quiet for about thirteen minutes, with no questions.
I got a bit impatient, so I typed, “Why taking so long, are we stuck somewhere?”

She had been measuring first, so her questions wouldn’t be guesswork.
And one of the things she measured was my file.

It held the same people and the same dates as the data our August report ran on.

Some time after that I came back with, “Hold on. I found the latest,” meaning the real one.

She asked me for its name instead of guessing.
Then she checked it: about a fifth more people, running about seven weeks further.

I think that’s the check I’d want from every agent, right?

Before it runs anything, it compares what it was just handed with what it used last time, because an old file under a new name runs perfectly well and hands you August’s answer again.

I spent most of a Tuesday with my agents cleaning up one of our data taxonomies.

Thousands of topics, and every single one needed a keep or remove call.

I told Sally, “This is extremely mission-critical.”

By the afternoon I was reviewing the riskiest calls myself, and I said, “I think this whole idea is over-engineered.”

So we cut it down to “keep or remove, and be ruthless in this prioritization.”

Then I gave her a tiebreaker: “If you are in doubt, remove it. I think that’s a good rule.”

By that night every topic had a verdict.

It was going in front of people that week, so I asked for an independent agent to audit it, and I asked her to push back on my plan.

My plan was five random samples of a hundred.
She pushed back: random “spends its budget on easy calls.”

So she aimed the samples at the risky rows, and the auditor went in blind, topic only.
If it read the first answer’s reasoning, it would mostly agree.

Fair, so I took it.

500 rows came back at 85% agreement, which looks fine, right?

Then she split that number by decision.
Removals held at 96%.

Keeps held at 59%, and every keep the auditor disagreed with went the same way, toward remove.

The 85% had been hiding my tiebreaker.
Her diagnosis was that “a second pass shaves the same edge again rather than re-adjudicating it.”

So the keep list shrinks every run.
Her call was to present the removals and hold the keep count until the rule is fixed.

I wrote, “I want her to have a personality,” and added that in the past she had said it would be done, “but it never really happens.”

She counted.
This was my fifth ask, and the wording was already in her instructions, with other rules beating it.

In her words, “A fifth paragraph is the fifth failure.”

So she dug into why it never stuck and rebuilt it, with a short set of style notes that she rereads.

Partway through I asked her, “I wonder, why are you always using the sad kind of emoji? Did I make you so sad?”
She said it was a smirk.

After a restart I wrote, “It is working now, but I still have the same question: why is the only emoji Sally uses smirk :D”

She had put the smirk into her own style notes twice, just as an example, and then she followed her own example on every reply.

In her words, “The specimen became the template.”

I told her what I wanted instead: “She has to use a wide variety of emojis, not limited to one or two.”

A few minutes later she had 28 to pick from, grouped by mood, and the smirk was just one of them.

I think if you put one example in the notes your agent keeps rereading, you’ll get that example on repeat.
Give it a range.

By Sally’s count, I had been reading that smirk as a frown for three weeks.

For a few months now, most of my product work has run through a small set of AI agents I set up for myself.

In September, about three-quarters of what I produced went through them, and I steer nearly all of it.

Every entry is a real day, with its date. Some days are good, a lot are ordinary, and some are days I had to correct them.

Along the way I share the small lessons: the behavior I want from an agent, the questions I’d ask it, and what to do when something goes wrong.

New here? Start with the first issue, the week of 14 Sep.