Filtering after the search can return too few documents

An AI analyst that checks permissions can fail without any leak. It can simply return too little. Guests from Bridgewater Associates and turbopuffer described this problem on the AI Engineer show.

What was said

Aaron Linsky, head of product engineering at Bridgewater, and Nikhil Benesch, CTO of turbopuffer, discussed the firm's Vault. It holds about 100 million documents. A speaker on the show said "Every email, every meeting note, every report might have a completely different set of permissions attached to it". A speaker also said Bridgewater's first setup, built without turbopuffer, seemed fine, then fell over; the excerpts do not say how.

Search finds documents by similarity, not by who may read them. The simple fix is post-filtering. The system fetches the closest matches, for example 10,000, then throws away those the user cannot see.

A speaker explained why this disappoints. The search runs out of time before it finds enough allowed documents. One example: finding maybe 40 when the user wanted the top 100. Users may know a document exists, the speaker said, and "you don't turn it up. So you have very poor recall." Recall means how many of the right documents come back.

Filtering first is also slow. If a user can see 50 million of the 100 million documents, scanning them all takes too long. The speakers' answer, which they call native filtering, skips whole clusters of similar documents that the user cannot see. They reported a P90 latency of about 120 milliseconds, meaning nine in ten searches finish within that time, across roughly 400 million vectors.

Why it matters

Our reading: a leak is loud, and a miss can be silent. A document that never appears may leave no trace.

The cost falls on the analyst who acts on a thinner picture and may not realise it. For managers buying or building these tools, that suggests testing both sides of a permission filter. Check that forbidden documents stay hidden. Also check that allowed documents actually come back, user by user.

The other side

The speakers did not claim that silent misses cost Bridgewater anything specific. The excerpts give no count of missed documents and no example of a bad decision. The talk was mostly about speed and recall.

The unhappy user in the speaker's example already knew the document existed. That user notices. Whether others miss documents and never find out is our worry, not something the speakers showed.

This is also a vendor describing its own fix. Cluster sizes are still being tuned, a speaker said, and the settings may change.