- Microsoft Research describes a synthetic-data evaluation platform for Copilot, built with no customer data at any stage of its pipeline.
- Microsoft reports a 39% gain in its own data-quality score over its first approach. The post publishes no Copilot accuracy results.
- Buyers should ask vendors who builds the test, what it measures, how answers are scored, and whether outsiders can check it.
The test now matters as much as the model
An AI assistant summarizes your board meeting. Who decides if the summary is good? A test does. Whoever writes the test sets the standard.
The lesson here is that trust in business AI rests on the exam, not only on the model. A buyer who cannot see how the exam works is taking the grade on faith.
Microsoft Research has described one evaluation system for Microsoft Copilot. The post is worth reading for what it shows and for what it leaves out.
What Microsoft built
Good testing needs data that looks like a real company. Real company data is sensitive. Microsoft says it "cannot be used for evaluation under any circumstances."
So Microsoft makes up companies. It calls each one a "tenant". That means an isolated workspace with its own users, data and settings. Microsoft Research says each one contains hundreds of simulated users with complete communication histories.
There are three. One is a drug maker, with regulatory filings and clinical trial talks. One is a capital markets firm, with deal pipelines and compliance messages. One is a software company, with sprint planning and code reviews.
Microsoft describes the privacy rule as a fixed design principle. Customer information never enters the pipeline, from the org charts to the emails to the meeting transcripts. The company says that makes the datasets safe to share across teams.
How the invented data is made
Microsoft built these companies from public sources. It names three: occupational data from the U.S. Department of Labor's O*NET database, online job postings, and SEC filings. These shape job roles, reporting lines and company size. Microsoft says its legal and policy teams reviewed and approved each source.
The data also moves through time. Meetings, project plans and email threads build up over weeks. That matters because many Copilot questions depend on timing, such as "What did the team decide last Thursday?" A single snapshot cannot test that.
A second pipeline writes the test questions. It starts with a library of common work tasks. It picks the ones that fit one industry and adapts the wording. It then generates questions tied to the fake data and rewrites them to sound human. Last, it checks each one for accuracy and uniqueness.
What Microsoft reports
Microsoft Research says its composite quality score for the fake data rose 39% over its first approach. Some industry environments scored above 0.9 out of 1.0 on Microsoft's internal scale. The score measures how closely the data mirrors the patterns of real company environments.
Microsoft also reports that building a dataset used to take days of manual work. Dataset creation now runs as automated pipeline runs. Microsoft estimates savings of about $100,000 per dataset and $300,000 across the project. The post does not explain how it reached those estimates.
The more useful claim for buyers is repeatability. Microsoft says a fixed configuration now lets any team rebuild a dataset and land on the same output. Manual work, it says, gave slightly different results each time. That made it hard to compare one release with the next.
What the post does not show
Every figure here is Microsoft's own, and the quality scores come from Microsoft's own scoring system. On Microsoft's own measure, the 39% is a gain in a realism score over its first approach. In our reading, that is not a measure of realism itself. It also does not say how accurate Copilot is. The post gives no Copilot accuracy results.
Microsoft calls this the first standardized evaluation baseline for Copilot. It was published to an internal shared catalog. That describes internal process. It is not an industry benchmark. The post does not say whether outsiders can inspect the datasets or the method.
A reasonable concern is that invented data can only test the situations its designers imagined. Failures that appear in messy real data might not show up. That is our argument, not a finding in the post. Microsoft says the goal is to find issues that generic benchmarks miss. The post gives no examples. It also does not say how Copilot's answers are scored.
Questions to put to your AI vendors
Ask who builds the test data and who checks it. Ask what the vendor measures: the accuracy of answers, or the realism of the test setup. Ask how answers are scored. Ask whether results can be repeated from a fixed configuration, as Microsoft says its can.
Ask whether any outside party has reviewed the method. Ask how the vendor tests agents that take actions, not only ones that write text. Microsoft lists that as an open question it is still working on.
A vendor's confidence is only as strong as its exam. Ask to see the exam, and ask who has already looked.
Produced by the WebPulse Newsroom with AI assistance from the original reporting credited below, and checked against that source by our editorial review. How we use AI.
Original reporting: Microsoft Research.
