Petrov's test for AI skill files: check each line against what the model already knows

Nikolay Petrov of Intel described, on the AI Engineer podcast, a way to test a skill: a file of instructions that gives an AI agent new knowledge. He said people start by feel, asking whether it seems to work. His method instead checks each small claim in the file on its own, so a team can see what every part contributes.

What was said

Petrov said the more technical check is an eval, a task the agent must solve. You run it with no skill, then with the skill, and compare. In his example the agent solved three of five tasks without the skill and all five with it.

That shows the skill helps, but not which lines do the work. So he splits a skill into small items he called atoms, each a question with an expected answer. He mentioned 25 or 30 atoms per skill as a typical size. Judges, ideally from a different model family, score the answers with and without the skill.

His reasoning was simple: "instead of relying on their luck we can put more deterministic process here". Each atom lands in one of four groups. The model already knew it. The skill made things worse. The skill taught something new, which is the goal. Or neither run answered, a sign the skill was poorly written.

According to the show notes, roughly seven in ten points in the first skill his team tested were already in the model's head. The value lay in the remaining three in ten: precise details that a crude trimming tool would drop first. Petrov said lines that hurt performance were rare in popular skills. He expected more in smaller, less popular ones.

Why it matters

The show notes describe a one-way ratchet. Adding a line is effortless, deleting one feels dangerous, so skills keep swelling. Our reading is that this is how a skill turns into folklore: nobody can say what each line does, so nobody dares touch it.

For teams that build or buy agent setups, a line-by-line map is a reasonable thing to ask for. It also shifts the work. Writing instructions is the easy part. Judging which claims earn their place, and writing test questions that do not give away the answer, is harder. People training for this work need to learn to read that evidence, not just produce more text.

The other side

Petrov was candid about limits. He called one output "basically initial vector for optimization at the end we still have to run evals and validate". The excerpts do not show exactly what he meant by it, so treat the method as a starting point, not a verdict.

The talk also covered only skills that bring new knowledge. He set aside steering or process skills, which just make agents repeat a fixed flow. His evidence on harmful lines comes from his own experiments. The excerpts do not say what the testing costs, or how closely the judging models agree with human reviewers.