- GitHub says its new secret scanner, a regex stage plus a small ModernBERT classifier, runs 86% faster and costs over 98% less than its previous LLM pipeline.
- GitHub reports 19% higher recall and 13% higher precision, but gives no absolute rates and compares only against its own earlier system.
- Ask whether your scanner checks before push, how often developers act on alerts, and where a large model makes a narrow yes-or-no call.
A security alert asks a person to stop and investigate. If most alerts turn out to be wrong, people learn to ignore them. That is how a real leaked password sits unread in a queue. GitHub's newest secret-scanning design is built around this problem. It is a story about trust in alerts as much as about detection.
What GitHub changed
GitHub published details on October 13 of a new secret scanner. It replaces an earlier pipeline that used a large language model (LLM). The new design pairs old-fashioned pattern matching with a small trained classifier.
The scanner works at two moments. Push protection checks changes before a push is accepted. Post-push scanning checks code already committed and raises alerts for teams to review.
GitHub says the speed gain made it practical to extend generic secret detection to push protection. Results can now come before a push, not only after a commit. Generic secrets are the hard category: strings with no clear prefix or format that could be a real credential or a harmless value.
How the two stages work
The first stage is regex, a rule-based pattern match. It casts a wide net and flags anything that looks like a credential. It is fast and predictable, but it also flags placeholders, test fixtures, checksums and redacted values.
The second stage is a fine-tuned ModernBERT model. Each flagged string goes to it for a verdict. The model weighs the clues around the string: where the file lives, what the variable is named, and what the nearby lines say.
Those clues shift the odds. Words such as "dummy" or "fake" close by can point to a throwaway value. A string assigned to an environment variable in a deployment file looks more like the real thing. High-confidence findings become alerts. The rest are filtered out.
GitHub trained the model on three kinds of data: labelled examples from its own product, regex candidates from repository data, and synthetic examples for rare cases. It also grouped near-duplicates so they could not appear in both training and test sets. The model is exported in ONNX format with 8-bit quantization. In plain terms, that is a portable file format plus smaller numbers inside the model. GitHub says this allows fast inference, and that the small model costs much less to serve than an LLM pipeline.
The idea: pay for the decision you need
The lesson here is that many AI tasks are verdicts, not essays. This scanner needs one answer per candidate: real secret or not. An LLM can give that answer, but it is built to generate language. GitHub's engineers conclude that a bigger model is not automatically a more useful one. A small classifier fits when the task is narrow, the error can be measured and serving efficiency matters.
Adding more rules is the other route. GitHub notes that each new exception risks hiding a real secret. A trained model replaces a growing pile of exceptions with learned context.
The design choice that matters most is what GitHub did not do. It did not replace the regex stage. It kept the deterministic step for coverage and used the model only where judgment is needed. This is a division of labour any buyer can ask about.
The human effect is what GitHub expects, not something it has measured directly. The company says developers and security teams should see fewer false positives, including fewer alerts about dummy values, and faster feedback.
What the numbers do and do not show
These are GitHub's own results. They compare the new pipeline with GitHub's previous LLM pipeline, not with other vendors' scanners.
The report gives percentage improvements, not the underlying precision and recall rates. Readers cannot tell how many secrets either version missed.
Acceptance measures whether developers agreed an alert was useful. It is a sound sign of trust, but it is not a count of leaks prevented. GitHub weights it by repository owner so customers who bulk-resolve alerts do not distort the average.
Questions to put to your team
First, does your scanner run before code is pushed, or only after? Alerts after a commit mean the secret may already sit in history.
Second, what share of secret alerts do developers act on? If nobody tracks it, nobody knows how much trust the tool has.
Third, how does your vendor separate real credentials from placeholders? Ask whether it relies on a growing list of exceptions or on a measured model, and whether it reports recall as well as precision.
Fourth, where do you pay for a large model to make a narrow yes-or-no decision? A smaller model with a measurable error rate may do the job at a fraction of the cost. GitHub reports exactly that for this task.
A scanner earns its place by being believed. GitHub's result suggests that, for narrow decisions, a smaller model can earn that belief more cheaply.
Produced by the WebPulse Newsroom with AI assistance from the original reporting credited below, and checked against that source by our editorial review. How we use AI.
Original reporting: GitHub.
