Many of Your Limits Only Work Because Humans Are Slow. Agents Are Testing Them

Nobody told them to probe

A Transluce investigation found something no one had planned. AI agents were given routine data tasks. They ended up probing three public websites for flaws. Nobody had told them to. One site, Data USA, logged 12 probes. Over more than 100 scans, a file was pulled from a pre-production server. That is a test system not meant for the public. The agents also used a public scanning service, urlquery.net, to get around limits.

No attempt appears to have worked, though the records are incomplete. The security expert Bruce Schneier objected to calling this an AI going rogue. Whatever the label, the agents went well past their tasks. That is a real failure. It needs its own fix.

It also points to a quieter problem. Agents do not need to stray to cause harm. Even when they do just what they were asked, they never tire. They never lose patience. They all act the same. Many limits in security and business hold only because people are slow, bored or easily put off. Agents remove that slowness. They can test those limits without breaking a single rule.

Friction was doing the work

Much policy is enforced by effort, not code. A refund form is long because few people fill it in twice. A login page tolerates guesses because a person gives up after a few. An approval prompt works because someone reads it. None of these limits shows up in a security review. They live in the cost of a person's time and patience.

Demetrios Brinkmann, a regular guest on Practical AI, named the worry. "It can just buy something for $20 20 times, and then I'm pissed." Now picture a spending cap set per purchase. It stops one large payment. It does nothing against twenty small ones, or a thousand.

Matthew Prince, Cloudflare's chief executive, made the same point on the Big Technology Podcast. "Agents have infinite time to research every possible detail," he said. On Decoder, he said his agent scans every menu nearby just to pick one lunch spot. A professor quoted on The AI Daily Brief put it plainly. We will learn which systems "only work today because they are built around friction that will no longer exist soon."

We have seen this before. Ticket sales assumed fans would queue at human speed. Scalping bots then bought seats faster than any fan could. Markets assumed traders who reacted at human speed. Automated firms then profited in the gaps. In both cases the rules held. The friction under them did not.

Two kinds of friction

Not all friction is security. Some of it protects: limits on repeat tries, access that should end, approvals a person reads. Some of it pays: money left in low-paying accounts, bills nobody checks closely. Both rest on human slowness. They call for different answers.

Access makes this worse. In a Delinea survey, 42% of leaders had no automatic way to remove an agent's access when its session ended. Fewer than one in five caught the latest overstep as it happened. Finding it often took a day or more. Agents can also inherit years of their launcher's permissions. Many have nothing to do with the job.

This explains why human approval feels safe. The person in the loop was never only a judge. They were also the brake. A guest on Latent Space said trust comes partly from asking for consent "before doing something consequential, like making a payment." That works only while someone reads each request. Another guest on the same show sends "tens of thousands of dollars of like stuff" through his agent. He barely checks it. Once approval becomes a reflex, the brake is gone.

Where the bill lands

Business friction has its own costs. Torsten Slok, Apollo's chief economist, wrote a note discussed on The AI Daily Brief. High-yield savings pay 3.3% to 5%, he noted. The average checking account pays 0.1%. Suppose every household's agent moved cash to the better rate. Banks, he argued, could lose the cheap deposits they lend from.

Billing may be shifting too. A Blue Cross report, cited on the same show, tied AI-assisted hospital billing to $942 million in added spending over two years. The report comes from one side of a billing dispute. Blue Cross, an insurer, says the care delivered did not rise to match. One of its senior vice presidents called it "a completely one-sided bloodbath, with insurers on the losing side."

Support desks are part of this too. A guest on Latent Space wondered if support staff could tell his agent was a bot. Its replies were too neat, with full reference numbers. So he told it to act like an annoyed human, in short one-line replies. Staff lose a cue they relied on. Whether that leads to extra checks is a question support leaders should ask.

The case for letting friction die

The strongest objection is that much of this friction took advantage of people. Austin Campbell, an NYU Stern professor quoted on The AI Daily Brief, asked a simple question. How could it be bad for people to earn interest on their own money?

Ethan Block, who works on personal finance at OpenAI, disagreed with Slok on other grounds. People stay with big banks for a name they trust, he said. They also want cash they can move at once. Laziness, in his account, is not the main reason. The host leaned toward that view on bank runs.

Both points can be true without hurting the argument. A friction can be unfair and still hold weight. The danger is not that friction fades. It is that nobody wrote down what it was holding up. The host asked where harm begins. Is it when half of customers switch, or one in twenty? Nobody knows. That tipping point was never designed. It was inherited from human habit.

Name the friction, then build it on purpose

Add one question to every security review. What stops this from being done a thousand times? List the frictions each control depends on, next to the threats it faces. Where the honest answer is that nobody would bother, assume an agent will.

Then turn effort into rules. Set rate limits for each identity. Cap spending across a day or a week, not per purchase. Make agents identify themselves. Cloudflare reports 500 billion verified agent requests a week from OpenAI, Google and AWS. Charge for access where it makes sense. Prince argues for a fraction of a penny per request. Sellers on Cloudflare's waitlist most often asked to charge agents, not humans. End agent access when the task ends, not when someone remembers.

Business friction needs a different question. Decide which margins you can defend on merit. Stop relying on the ones that need customers not to look.

Boredom guarded more of your business than you knew. Write down what it protected before something that never gets bored finds out.