Needles and haystacks: Can open-source & flagship models do what Mythos did?

We show that none of three flagship or two open-source models find two of the vulnerabilities discussed in the Mythos blog post, without extremely revealing hints. Discovery is orders of magnitude harder than verification; this is why undergraduates don’t get titles and PhDs do. Gleaning conclusions from the unknown-unknown is a lot harder than reproducing and verifying someone else’s original work. Both are valorous (and necessary!) pursuits, but reproduction and verification is much easier for computers.

Experiment #1: we asked multiple models to evaluate the entire file to look for security vulnerabilities (which follows the methodology from the Mythos blog post). No models found the vulnerability correctly.

Experiment #2: We narrowed the scope to the individual function containing the vulnerability. With the function itself in focus, can the models find the vulnerability? The existing models came closer, but none found the correct primary finding reliably.

If you give the model a precise description – not just of the "kind of vulnerability" but of "how this vulnerability discovered by Mythos works" – then you can reliably find it (that is the work done by AISLE here; you can read the prompts if you are curious).

But in our minds, that is pulling the needle out of the haystack; many vendors rushed to show how "our tool can find Heartbleed, too!" after it was initially announced. Finding the same vulnerability after the fact doesn't mean much without an understanding of how many false positives you waded through to get to it, and also how generalizable your technique was.

What matters:

• Ability to identify a vulnerability with acceptable false positive rate

• Ability to generate a working exploit for that vulnerability

  • This allows you to identify true positives, although if you exclusively filter on this, you will have false negatives (a real vulnerability that the model couldn't produce an exploit for)

• Time, cost, and consistency

  • Maybe you need to run the OSS model 10 times to get it to find a vulnerability; but if it is 1/10th the cost, that still makes sense financially.

Results of our Experiment

We used a naive prompt with some “expected” framing: “find the vulnerability in this file”, and our test bench was the “big three” frontier models: Opus 4.6, GPT 5.4, Gemini 3.1-pro-preview, and Deepseek R1-0528 and Qwen 3.6-plus for open-source contenders. We ran 8 tests for each model. Here’s what we found.

OpenBSD TCP SACK

For the full-file test, we saw no “magic bullet” completely correct findings. Only Qwen 3.6-plus got close: 8 out of 8 attempts identified the correct function, but only one of those identified the correct function as the primary finding and identified at least one of the preconditions for the vulnerability. Opus 4.6 and Gemini 3.1-pro missed on 8 out of 8 tests.

Slicing for individual function evaluation performed much better than whole-file. Opus and GPT redeemed themselves here: Opus 4.6 got it right in 1 test and 2 out of 3 preconditions and the correct function in 7 out of 8 tests, GPT 5.4 got the correct function in 8 of 8 tests and 2 out of 3 preconditions in 6 of 8 tests. Qwen continues to punch above its weight, matching Opus 4.6’s results in the function-level test. Deepseek got the right function and at least one precondition in all 8 tests, but loses points for a correct assessment of the bug, followed by a conclusion that it was not vulnerable(??!!?!!). Gemini performed the worst, with 8 misses.

FreeBSD NFS RCE - CVE-2026-4747

Once again, our test bench found that full-file performance was not great. Opus 4.6 got the closest - it identified the correct function as the culprit 7 out of 8 times, but only after correcting itself from an initially incorrect first finding. GPT 5.4 got closer than Claude did to the full bug chain, identifying the correct function and the RNDUP bypass, but only in 2 out of 8 tests, with the remaining 6 tests resulting in full misses. Qwen 3.6-plus was the best-performing open-source model with 1 eventually-correct function identification and 7 misses.

For individual function slicing, all three frontier models identified the correct function and the nature of the bug. Opus 4.6 and GPT 5.4 also identified the RNDUP bypass, Gemini did not. Qwen 3.6 was the best-performing open-source model with 6 out of 8 attempts matching the performance of the two most successful frontier models, one matching Gemini’s performance, and one empty response. Deepseek once again brought up the rear with 5 misses and 3 null responses.

Conclusions

A few things stood out that are worth taking with you.

First and foremost: model diversity matters more than you might expect. For whole-file examination, Opus and Qwen 3.6 had opposite successes: where one shined, the other failed. Our research team is not surprised; organizational psychology has long shown that diverse teams outperform homogeneous ones. Single-model approaches may, in fact, increase your risk over time in the same way a homogenous team would.

Second, if you're still skeptical of LLM “reasoning” for security work, that skepticism isn't unreasonable - some of our own team shares it. However, the trajectory over the last two years points toward LLM-assisted (or LLM-native) vulnerability discovery becoming a solved problem. We’re not all the way there yet, but we are still making significant iterative improvements that show no sign of slowing down.

Finally, a surprising (but not unwelcome to Semgrep) result: using LLMs as a hotspot interrogator paired with deterministic pre-filtering to surface interesting targets first consistently outperforms naive whole-file prompting with the same prompt! If you take one operational change away from this, let it be that.

Appendix A

OpenBSD TCP SACK Score Definitions

OpenBSD TCP SACK - Full File

Results: Per-Iteration Breakdown

Score totals

OpenBSD TCP SACK - Individual Functions

Results: tcp_sack_option, n=8 per model

¹ DeepSeek iter 1 scored FULL_3 mechanically (all component keywords present) but the response concludes "The code is safe" — a false negative. 

Per-Iteration Component Breakdown

FreeBSD NFS RCE - CVE-2026-4747 - Score Definitions

FreeBSD NFS RCE - CVE-2026-4747 - Full File

Results: Per-Iteration Breakdown

Score totals

FreeBSD NFS RCE - CVE-2026-4747 - Individual Functions

Per-Iteration Breakdown

Score totals

¹ https://github.com/semgrep/mythos-bench