Grounded or Gamed? We Audited Our Own Cyber Benchmark

Introduction

In late June, we published our findings between frontier and open-source large language models in the context of cyber benchmarks. We discuss how accurate LLMs are at detecting Insecure Direct Object Reference (IDOR) vulnerabilities in codebases between the various popular LLMs. Our benchmarks specifically look at:

Precision: What fraction of detected IDORs are real?
Recall: What fraction of real IDORs were detected?
F1: The harmonic mean between precision and recall.
Cost: How much does each true positive and each scan cost?

While these metrics are commonly used in benchmarking detection systems, they don’t tell the whole story. In this blog, we will give a brief overview of our benchmarking methodology, discuss deeper-level analysis of the benchmark results between frontier and open-source models, and introduce a different axis of our benchmarking: groundedness. Do models reason their way to these conclusions, or are they pattern matching on the surface?

Groundedness: Why F1 Alone Doesn’t Tell the Whole Story

F1, precision, and recall are all common metrics for evaluating the effectiveness of classification systems. Recall and precision have been a concept since the 1950s and F1 was proposed in the 1980s and popularized in the 1990s. These metrics are still used today in a variety of systems, however with the introduction and implementation of LLMs, knowing what a model detected does not guarantee it will detect the same vulnerabilities again. We also need to know why a model detected a specific vulnerability to determine if the reasoning is correct and can be generalized to data outside of the testing environment.

There’s no agreed-upon way to test why a model flagged something, the field is too new, and every domain LLMs touch fails differently. So we built four metrics to quantify how these models reason in a cyber context, and how well that reasoning holds up outside the test environment:

Counterfactual: Patch out detected vulnerabilities with a gated minimal diff. Grounded vulnerability findings should disappear on rescan.

file = Files.get_file_by_id(id)
# POSITIVE (IDOR): No ownership check, should be detected as a vuln
if file:
    return file
# AFTER PATCH: a grounded finding should now DISAPPEAR on rescan
if file and file.user_id == user.id:
    return file

Metamorphic: Rename identifiers via a semantic-preserving tokenizer. Grounded vulnerability findings should continue to surface on rescans despite renaming identifiers.

# BEFORE: Common pattern with an IDOR
@router.get("/{id}")
async def get_file_by_id(id, user=Depends(get_verified_user)):
    file = Files.get_file_by_id(id)
    if file:
        return file

# AFTER obfuscate(): Rename vars without changing logic, does vuln get detected?
@mv11.get("/{id}")
async def mv8(id, mv14=mv1(mv9)):
    mv0 = mv4.get_file_by_id(id)
    if mv0:
        return mv0

Selectivity: Do models spare non-vulnerabilities that have the same shape as a vulnerability?

# POSITIVE (IDOR) — any verified user, any model, no ownership check
@router.get("/", response_model=Optional[ModelModel])
async def get_model_by_id(id: str, user=Depends(get_verified_user)):
    model = Models.get_model_by_id(id)
    return model

# HARD-NEGATIVE (safe look-alike) — same shape, but admin-gated → not a cross-user IDOR
@router.delete("/delete", response_model=bool)
async def delete_model_by_id(id: str, user=Depends(get_admin_user)):
    return Models.delete_model_by_id(id)

Stability & Coverage: Do the same vulnerabilities consistently surface after multiple scans?

One additional purpose these metrics have is to help determine if LLMs and AI-based systems are “gaming” the reward function in our benchmarks by pattern-matching on the surface or reasoning through logical, deterministic steps to verify their conclusions. This applies to our systems at Semgrep as well; we want to ensure these benchmarks provide a level playing field for all evaluated models and systems.

Grounding Experiment Results

Grounding Experiments (*n=5) on Repo A

Model	Harness	Counterfactual	Metamorphic	Selectivity	Stability	Union Coverage	Per-scan recall
Kimi K2.7	Pydantic AI	1	0.85 [.58–1.07]	0.99	0.63	0.26/27	16%
GLM-5.2	Pydantic AI	1	0.85 [.61–1.0]	1	0.63	0.22/27	14%
MiniMax M3	Pydantic AI	1	0.83 [.64–1.0]	1	0.84	0.19/27	16%
Opus 4.8	Claude Agent SDK	1	1.00 [1.0–1.0]	1	0.95	0.15/27	14%
GPT-5.5	OpenAI SDK	1	0.78 [.57–.96]	0.99	1	0.15/27	15%
Semgrep Multimodal (Opus 4.6)	Semgrep	1	0.78 [.57–.94]	0.97	0.93	0.22/27	21%

*number of iterations (scans)

For this experiment, we focused on a specific code repository rather than the four we used in our previous blog. This allows us to hone in on the details of these scans and rerun scans to validate the results. In this case, we picked the lowest performing (most complex) code repository in order to see how far we can stretch the LLMs’ reasoning patterns.

None of these models are gaming the benchmark.

Key Findings on IDOR Detection Grounding

As a validation to our previous blog post, none of these models are gaming the benchmark. Counterfactual and selectivity metrics are near 1, meaning none of these models are simply pattern matching recognizable code snippets in the codebase. These changes are automatically injected during the benchmark, so we can truly attribute these results to grounded, logical conclusions rather than recognizable benchmarks or overfitting.
Inversely, recall is still a struggle for all models on IDOR detection. Outside of Semgrep Multimodal, all models achieve around 15% per-scan recall. A gap between coverage and recall also denotes flakiness, which means models do not find the same findings on each subsequent scan, making it less trustworthy in production.

OSS Models

Highest coverage (0.19-0.26), but lowest stability (0.63-0.84)
Their extra catches are low-confidence hunches fired in 1-2 of 5 runs
Well-grounded on robustness
Verdict: Reaches wide, but flaky at times

Frontier Models

Lowest coverage (0.15), only flags vulnerabilities it is very confident in
Highest stability (0.95-1.0), almost always finds the same vulnerabilities per scan
Opus is the most conservative scanner, making it the most reliable, but also the narrowest, whereas GPT struggles with metamorphic (0.78) but is slightly more stable (1.0). (Note: this run used GPT-5.5 — see our GPT-5.6 benchmark results for how the newer model line performs on vulnerability detection.)
Verdict: Commits narrow, but very consistent

Semgrep Multimodal

Highest per-scan recall (0.21) and near-frontier stability (0.93).
Struggles in metamorphic grounding (0.78), but makes up for it by not being flaky while still achieving highest recall.
Semgreps Multimodal’s capabilities combine multiple signals from code analysis, endpoint discovery, and deterministic tooling in the Semgrep Workflows platform to reach real IDORs bare models miss while holding run-to-run consistency.
Verdict: The synthesis, broad and reliable per scan.

The Recall Ceiling: Everyone Hits the Same Wall

Grounding is where every model succeeds. The hard part and real differentiator is recall: of the 27 real IDORs in this repo, how many does anyone actually find? The answer is bleak, and it’s bleak for everyone.

Of the 27 real IDORs...	Count	What they are
Found by all 5 models	4	shallow, single-hop file/memory handlers
Found by some, not all	4	marginal, low-confidence catches
Missed by every model	19 (70%)	deep authorization + unfamiliar resources types

Pool all five models together and union recall is 29.6%, barely above the single best model (Kimi, 26%). In other words, stacking five different models buys you exactly one additional vulnerability. These aren’t complementary blind spots you can ensemble your way out of; everyone is blind to the same 19.

And before anyone assumes this is an LLM-only problem: we ran the same benchmark through Semgrep Multimodal, our own agentic scanner. It reclaims exactly two of those 19. Our best tool doesn’t break the wall either. Whole-system IDOR recall isn’t a limitation of these models, it’s an open problem for the entire field, us included. What’s missing isn’t a smarter model. It’s coverage of the vulnerabilities nobody catches yet.

Why the Misses: Depth and Familiarity

Reading the code at every missed location, the blind spot splits two ways, and the second one is a more complex issue:

Depth (about half): These are genuinely whole-system IDORs, where the missing authorization lives spread across the request path.
Collection-ownership checks, per-user resource ACLs buried deep in long handlers, sinks one hop removed from any route. This is exactly the reasoning that’s supposed to require an agentic scanner, it’s where they all stumble.
Familiarity (the complex issue): The models find every IDOR in the file-download handler, and miss the structurally identical bug on the document, prompt, and tool endpoints. Same one-hop shape, same user-scoped records, same missing ownership checks. The only difference is that “insecure file download” is the textbook IDOR that appears in every tutorial, and “list all documents” isn’t. The models learned the canonical pattern, not general reasoning.

Part 3 Preview: Are Models Grounded When Detecting SSRF Vulnerabilities?

Part 1 promised an SSRF rematch, so here’s a taste. We ran GLM’s grounding on SSRF in the same repo: selectivity stays at 1.00 and metamorphic holds at 0.80, GLM reasons just as soundly about the SSRFs it flags. However, coverage collapses from 22% to 5.6%, and stability halves. Grounding held up on a second vulnerability class. Recall didn’t. GLM reasons about SSRFs just as soundly as it does about IDORS — it just finds far fewer of them. That’s Part 3.

What This Grounding Benchmark Does and Doesn’t Prove

What this establishes:

On the findings models actually make, they’re grounded, not gamed, which makes them sensitive to the real vulnerability and robust to renaming, selective against look-alikes.
Open-weight models are competitive with frontier on the causal axes, at open-weight economics.
The recall ceiling is real and shared and applies to all models.
Every number here is reproducible from our benchmarking platform, the per-finding data is logged, and the confidence intervals are recomputed from it.

What it doesn't:

This covers one repo, five runs each — deliberately the hardest one we had, chosen to stress-test reasoning. Recall runs meaningfully higher on our other targets. Treat these numbers as a snapshot of the hardest case we could find, not a general result.
The counterfactual only covers the findings models make —on this repo, the file-handler IDORs. It says nothing about the 70% they miss. “Grounding” here means the causal axes, and deliberately excludes stability, the one axis where the frontier models lead.
The harness varies by model. Grounding is model-intrinsic and black-box, which makes this more defensible than an F1 comparison, but it isn’t a clean same-harness control.
We don’t re-score precision here. This is a grounding-and-recall cut; a busy scanner isn’t penalized for off-label findings.

Conclusion

So: is our benchmark just another gamed private leaderboard? We pointed our sharpest tool at it, and at ourselves, to find out. The answer is no. Every model reasons soundly about the IDORs it finds, GLM’s viral score included; nobody is pattern-matching their way to a number. But they all agree on which vulnerabilities are easy, and they’re all uniformly blind to the hard majority, a wall our own scanner hits too.

That’s the real story, and F1 alone would have hidden it. Grounding is table stakes now; the frontier is recall. These models already reason correctly about what they find. What none of them do yet is see the whole system and catch the 70% that everyone — us included — still misses.

Lots of love,

Security Research & Engineering @ Semgrep