GitHub has announced a significant evolution in its secret scanning infrastructure, introducing a new, context-aware classification system that dramatically improves detection accuracy while slashing operational costs. By shifting from a heavy reliance on remote Large Language Model (LLM) calls to a compact, fine-tuned model, the platform has achieved an 11.5% increase in alert precision and a 98% reduction in serving costs, all while cutting median scan execution times by 86%.
In the modern software development lifecycle, the speed of code deployment is matched only by the complexity of the security risks involved. As developers increasingly rely on AI-accelerated workflows to write and ship code faster than ever, the risk of accidentally committing sensitive credentials—such as API keys, database passwords, and private tokens—has grown proportionally. GitHub’s latest engineering breakthrough addresses this tension between the need for rapid security feedback and the technical challenges of identifying genuine secrets within vast repositories.
The Challenge of the "Hard Negative"
At the heart of the challenge is the difficulty of distinguishing a legitimate credential from a harmless lookalike. Security scanners have traditionally relied on pattern-matching, often using regular expressions (regex) to identify strings that share the length, alphabet, or prefix structures of known secrets. While these deterministic methods are exceptionally fast, they suffer from a fundamental limitation: they lack the ability to understand context.
In many development environments, strings that mimic credentials are abundant. Documentation placeholders, test fixtures in sample code, redacted values, build identifiers, and non-sensitive hashes often trigger regex alerts. These are what security engineers call "hard negatives"—strings that appear to be secrets to a rule-based system but are, in fact, benign.
Prior to this update, GitHub employed an LLM-based pipeline to add a layer of intelligence to the scanning process. However, while effective at discerning context, the latency and high computational cost of querying remote LLMs made it difficult to scale this protection effectively across the entire platform. The challenge was to create a system that could "understand" the code as well as an LLM, but operate with the speed and efficiency required for real-time, high-volume scanning.
A Two-Stage Approach to Intelligent Scanning
To resolve this, GitHub’s engineering and applied science teams designed a two-stage, hybrid pipeline. The first stage retains the speed and reliability of deterministic regex scanning. By casting a wide net, this initial phase ensures that no potential candidate is missed, acting as a high-recall filter that captures anything that remotely resembles a credential.
Once a candidate is flagged, it enters the second, more sophisticated stage: a fine-tuned ModernBERT classifier. This model evaluates the candidate not in isolation, but within its immediate ecosystem. By analyzing the surrounding code, the file path, variable names, and adjacent text, the classifier assigns a probability score to the finding. If the model determines that the string is a high-confidence match, an alert is generated. If it identifies the string as a likely false positive—such as a "dummy" variable in a test file or a "replace-me" placeholder in a config guide—it suppresses the alert before it ever reaches the developer.
This design choice—prioritizing deterministic scanning for breadth and machine learning for nuance—is central to the system’s success. The ModernBERT model provides a level of contextual understanding comparable to an LLM but is specifically optimized for code-adjacent text. Because the model is compact and efficient, it can be run locally or within a highly optimized serving stack, eliminating the overhead of external API calls.
Training for Precision and Scale
Developing the model required a rigorous training and evaluation strategy. The team focused on the "hard negative" problem, feeding the model examples of strings that are syntactically similar to real secrets but semantically different. By including candidates that often confuse rule-based systems, the model learned to distinguish between an actual production credential and a harmless example string.
The training data was meticulously curated from three sources: real-world secrets, realistic false positives, and synthetic data that passed through deterministic validation and rigorous quality reviews. To prevent data leakage and ensure the model was truly learning generalized patterns, the team grouped training rows by repository and candidate value, ensuring that no near-duplicates could appear in both training and evaluation sets.
The resulting model is exported to an ONNX format with 8-bit quantization, which enables extremely fast inference. This efficiency is what allows GitHub to extend its "push protection" capabilities. By returning results before a developer even pushes their changes to the server, the system provides immediate, actionable feedback, effectively stopping leaks at the moment of creation.
Tangible Results in Production
The real-world impact of this transition has been validated through both internal production testing and customer-facing A/B experiments. The 86% reduction in median post-push scan execution time is perhaps the most critical metric for developer experience, as it ensures that security does not become a bottleneck in the development process.
The 11.5% increase in owner-weighted alert precision confirms that the system is successfully reducing the "noise" that often leads to alert fatigue. When developers receive fewer false alarms, their trust in the security tools increases, making them more likely to heed warnings when a genuine, critical vulnerability is detected. Furthermore, the 98% reduction in serving costs compared to the previous LLM pipeline provides a sustainable path for GitHub to continue scaling its security coverage as the platform grows.
Looking Toward the Future
The success of this project highlights a broader trend in machine learning for security: the realization that bigger is not always better. While general-purpose LLMs are powerful tools, a smaller, fine-tuned classifier can be the superior choice when the task is narrow, the decision boundary is measurable, and efficiency is paramount. By focusing on the specific product tradeoff—maintaining recall while minimizing false positives and latency—GitHub has created a system that is both more accurate and more scalable.
As the industry moves toward more sophisticated AI-driven development, the need for equally sophisticated security infrastructure will only increase. GitHub’s focus remains on refining these models and expanding their capabilities. The team is currently working on further improvements to the classifier and investigating new ways to integrate context-aware security into the broader development workflow.
The engineering efforts behind this pipeline—led by teams across Microsoft and GitHub—represent a significant step forward in making security a seamless, rather than disruptive, part of the development cycle. By combining the transparency of traditional rules with the intelligence of modern machine learning, GitHub is providing developers with the protection they need to build with confidence in an increasingly complex digital landscape. As the platform continues to evolve, these advancements in secret scanning will remain a cornerstone of its commitment to developer productivity and software integrity.
