OpenAI shared the GPT-5.6 system card and shipped three new models that are now available for use which have cleared the “High” cybersecurity capability threshold under OpenAI’s Preparedness Framework. This is a formal classification indicating that the model removes bottlenecks to scaling cybersecurity operations such as automating end-to-end attacks and also automating the discovery of vulnerabilities. This remains a step below “Critical” which would indicate functional zero-days against hardened systems with no human in the loop.
The three models as in the past differentiate how complex the reasoning tasks map: the flagship model (Sol), a mid-tier option (Terra), and a cheaper / faster (Luna) option. All three reasoning levels notably have cybersecurity capabilities now rather than publishing a separate cyber model. We observed a regression in precision (the share of reported findings that are real vulnerabilities, ie. reports higher false positives) but also showed a greater than 2x improvement in recall on true positives.
Luna as a cost per true positive was a standout result with a better ROI, 6x less expensive while only sacrificing a marginal result in the F1 harmonic mean between recall and precision in our latest benchmarks.
The Model is Not the Whole Story
OpenAI’s CTF evaluation doesn’t test the model in isolation, so while the 96.7% reported by OpenAI is impressive; if you strip the harness and hand the model a pre-configured prompt that number doesn’t hold. Another benchmark, CVE-Bench has shown the models to be doing “slightly better” than GPT-5.5, an improvement but not exponential. Many benchmarks use common open source repositories with known vulnerabilities which can dilute how validation is performed. The saturation of the benchmarks being used to validate not showing significance between models any longer could be explained by training data intermixing with test data.
We iterate on our benchmarks with a different problem statement to assess the use cases we believe our customers care about the most rather than hyper-focusing on what the models are training on. This often means we see different results from our benchmarks, and compare both native LLM use alone with Semgrep Multimodal. This continues to support our hypothesis that what you wrap around the overall security workflow including the harness, tools, and scaffolding continues to outperform and improve in lock step with model improvements.
IDOR Detection Results for GPT-5.6
The image below demonstrates a snapshot of the IDOR Detection use case. We often lead with this over other vulnerability classes we test like SSRF, etc. because it provides a strong contrast to traditional SAST data flow analysis. For all of our benchmarks, we’ve judged true positives after rigorous security researcher review on representative production applications over ten runs.
This should be no surprise, but GPT-5.6 across all reasoning levels outperforms GPT-5.5 on the same benchmark. This suggests the model has remained stable where it needed to and has improved as it has expanded cybersecurity capabilities.
Whether running the models alone with a guided prompt or with our complete harness, both scenarios demonstrate improvements.
We judge performance based on the ability to correctly identify true positives (recall), correctly avoid false negatives (precision), and the harmonic mean (f1) between the two when the outcome may otherwise be mutually exclusive.
Luna May be the Best Fit Model Today
There is a parallel conversation about whether to move off of frontier models onto open-weight ones, but that’s not the only framing to consider. Our CTO made a point recently, it isn’t risky to use an open source model, it’s what happens when teams are moving fast with AI-generated code today. Code validation, security checking, etc. cannot easily keep pace without adequate tooling so that gap between code generation and code review where vulnerabilities slip into production poses the biggest risk for the team. Organizations that skip security steps are taking on outsized risk whether using an open source model or a commercially available one.
This is what makes the collaboration of OpenAI and the cybersecurity industry welcomed and seems to keep both offensive and defensive measures on an even playing field. Our benchmarks recently showed remarkable results from GLM-5.2 but the frontier model release has pulled ahead again. We regularly evaluate alternatives and can get better and more consistent results from frontier commercial models like GPT-5.6, Opus 4.6, Gemini 3.x, etc.
Similar to using open-source Semgrep, the savings that comes from using open-source over a commercial offering can be offset by increased costs for fine tuning, validation, infrastructure, and engineering attention needed to run them well. The cost effectively moves across the balance sheet from capital to operating expense. Sometimes, the more rational choice is to take the commercial offering.
For high volume, narrowly scoped tasks, a small model can perform just as well, such as some of our observations around benchmarks with GLM-5.2. We continue to experiment with various models, but our engineering team primarily uses Claude Code and OpenAI Codex for development, having the enterprise capabilities we need and the ease of putting security guardrails like Semgrep Guardian in place is important to us (yes, we rely on and trust our own product to secure our software).
Putting the Latest GPT-5.6 Release in Context
GPT-5.6 introduces more predictable prompt caching, explicit cache breakpoints, adjustments to pricing, and cybersecurity capabilities across the full reasoning levels. Luna stood out in our results on a cost per true positive, a north star for many security teams, but all three models performed very well across a variety of benchmarks both with a guided prompt or a more complete security workflow.
For more about our benchmarking approach, we'll detail how representative we think the process is in a separate blog post coming soon. As new models become available from Anthropic, OpenAI, Google, etc. as well as open weight and new players, we look at them through a cybersecurity lens to leverage the models in an effective manner that delivers the most reliable, consistent, and cost effective results we can find.
Today, that means GPT-5.6 is better than GPT-5.5, it is better suited for cybersecurity use cases like ours, the harness we built leveraging insights from tools like Semgrep Pro makes a substantial difference over using the model on raw code alone, and as each model improves we do too in lock step and can optimize where it makes sense on the fastest, cheapest, and top scoring model available.
.jpg)
