- GitHub Engineering replaced a list of stargazers with an endpoint of daily star counts, cutting react/react's history from about 2,511 requests to about 24.
- Returning fewer results did not make queries cheaper. The database had to filter by date before it counted, and the user names turned out to be unnecessary.
- Ask which fields your own APIs expose only because of how data happens to be stored, and which queries count everything just to return a page.
The extra data nobody asked for
Most systems hand out more than their users need. Sooner or later, someone finds a use for the extra. GitHub Engineering has just described how that played out with a feature many people never think about: star history.
Maintainers use star history to see whether a launch or release changed a project's path. To draw that curve, tools had to download lists of people who starred a repository, along with the date of each star. They kept the dates and discarded the names.
The names stayed available for anyone to collect in bulk. According to GitHub, spammers and phishers started harvesting the public lists for their campaigns. The same lists also put heavy load on GitHub's database.
Why returning less did not make it cheaper
The old endpoint was built to answer one question: who starred this repository? It returned pages of up to 100 people. It was never meant to serve as a time series.
The underlying star record holds only the stars that exist today. A removed star simply disappears from it. So a tool had to read every current star to rebuild the curve. For react/react, which has about 251,000 stars, the full pull meant about 2,511 requests. GitHub adds that many tools gave up partway, either stopping at rate limits or reading only a sample of pages.
A bug made it worse. To work out how many pages existed, every request counted all current stars, though the caller received only one page. In the week before access was restricted, nearly 37,000 of these counts were slow enough to be logged. They averaged nearly 3 seconds each. More than 1,000 were cut off by database safeguards, and that added to request timeouts.
The fix that looked obvious, and failed
GitHub's first fix was to limit stargazer and watcher lists to administrators and collaborators. That worked. The database then served about 60% fewer stargazer requests. The total time spent on the counting query dropped by roughly four-fifths.
It also broke tools such as Star History, which needed only counts and dates. GitHub talked with maintainers and tool builders and built a purpose-made endpoint.
The natural design was to group stars into daily or weekly buckets and return a page of buckets. But in a plain query, the database groups and sorts every star first. Only then does it apply the page limit. A small response still meant a large amount of work.
GitHub's estimate for react/react is that this design would have examined roughly 6 million rows. Each of 24 requests would have reprocessed all 251,000 stars.
The shipped design reverses the order. It turns the page number into a date range on an indexed column, keeps only those weeks, and then counts per day. One request returns up to 30 weeks of daily counts, with no identities. The database now reads each star about once across the full history.
What this shows
The lesson here is that a smaller answer and a cheaper answer are different things. Cost depends on how much data the system touches, not on how much it returns. GitHub's engineers put it this way: the fix came from examining the sequence in which the database did its work.
The privacy half carries a second lesson. Because each timestamp came paired with a user, the old response made it seem that identity was a required part of the data. Tool builders never needed it. The people exposed were not the maintainers or the tool authors. They were the developers who had starred a repository.
A third point concerns dependency. Star History users had to rally after the access change broke the tool. Any product built on someone else's API carries that exposure.
Limits GitHub states
These are GitHub's own figures about its own system. The company also names remaining limits. Each request is bounded by a time range, not by the number of stars. If one week holds an unusually large number of stars, that request can still be slow or costly.
Calendar boundaries and removed stars also affect what the endpoint can return. The curve is an approximate signal of interest, which is how the tool builders GitHub spoke to used it.
Questions for your team
First, which fields do our APIs and exports return only because of how the data is stored? Ask who uses each one. Names, emails and IDs often ride along with data that someone wanted only to total.
Second, which of our slow queries count everything just to build a page? GitHub found the same pattern on its stargazers page and in other endpoints once it looked.
Third, which of our tools depend on a third-party API that could restrict access? Do we know what they truly need from it?
Returning less is a privacy decision. Doing less work is an engineering one. GitHub's fix needed both.
Produced by the WebPulse Newsroom with AI assistance from the original reporting credited below, and checked against that source by our editorial review. How we use AI.
Original reporting: GitHub Engineering.
