Announcing an AI AppSec engineer that security researchers agree with 96% of the time

Adding semi-autonomous capabilities to Semgrep Assistant

We’re starting the year off strong with the launch of two powerful features for Semgrep Assistant that reduce triage workloads by 20% immediately - and up to 40% after just one week of use. We've got the data to back these numbers up, so read on!

These features give our trusted security assistant its first semi-autonomous capabilities, allowing it to take over routine security tasks and give back a significant amount of time to developers and security engineers.

Noise Filtering: Detect false positives that static analysis tools always flag

One of the biggest struggles in application security has always been signal-to-noise ratio. Traditional static analysis tools inevitably surface a substantial number of false positives - enough to erode developers’ trust in security tooling and block any efforts to “shift left”. 

With this launch, we believe we’ve turned a corner on this existential problem. Combined with Semgrep’s deterministic SAST engine, Assistant is able to perform context-aware analysis of SAST findings, identifying and filtering out a huge number of false positives that purely syntactic engines will always flag. 

Assistant reduces the total number of findings teams have to analyze by ~20% out-of-the-box. That’s an instant 20% reduction in your production backlog, and 20% fewer findings to triage in all future scans.

Of course, a feature like this isn't useful unless it's highly precise. Our security research team agrees with Assistant 96% of the time, and users agree with Assistant auto-triage decisions 92% of the time. This is over a sample size of 250,000+ findings and 45+ enterprises - we'll go into more detail later.

AppSec teams desperately need a solution that can automate the menial (yet still cognitively taxing) triage work that is forever accumulating. By combining Semgrep’s best-in-class deterministic SAST engine and LLMs, Assistant is able to filter out one in five SAST findings as a false positive that is safe to ignore, at a 96% alignment rate with security researchers.

Note: Assistant's noise filtering will never close a finding for you - filtered findings are simply hidden from developers and grouped in the platform for bulk triage.

TL;DR?

• ~20% of all SAST findings are confidently filtered out as false positives. This means developers are interrupted less and trust security alerts more. 

• Security practitioners can spot-check or review all suppressed findings in one place, see explanations for why findings were flagged as false positives, and bulk close them.  

• Our security researchers agree with Assistant 96% of the time. Users agree 92% of the time
  

Autotriage Memories: never triage the same security issue twice

SAST tools rely on customized rules to “understand” the organization specific context needed to determine exploitability: trusted data sources, sanitizers, secure defaults, etc. 

One thing we learned last year is that the vast majority of security teams do not have the bandwidth to write or customize rules for their SAST tool - even if they use Semgrep, which is known for its simple rule syntax that resembles source code.

With Autotriage Memories, Semgrep Assistant can learn and remember the organization-specific, security-relevant context needed to determine exploitability - no customization or PhD in SAST tooling required.

Assistant turns manual triage into a high ROI activity that permanently reduces the number of irrelevant alerts developers and security folks see in the future.

No more writing custom rules for every new library function you consider safe.

No more repeating the same triage steps for each new or refactored repository.

One of our private beta users, a Fortune 500 company, saw an additional 2.8x improvement on top of Assistant’s base noise filtering capabilities after adding just 5 memories. 

"The ability to have Assistant remember what I told it and automatically triage for me in the future is game changing. I have to spend a lot of time verifying the validity of vulnerabilities and being able to essentially hit the "save" button on the work I've done and just pass it on to Assistant has really helped streamline my triage process." 

  - Kevin Twingstrom, Lead AppSec Engineer, Acrisure

TL;DR:

• Assistant can learn and codify the organization-specific, security-relevant context needed to determine exploitability

• Memories are suggested during triage, or manually added. Users simply describe in human language what they want Assistant to remember.

• One private beta user, a Fortune 500 company, saw a 2.8x improvement on top of Assistant’s base noise filtering capabilities after adding just 5 memories. 

  
  

Experts agree with Assistant 96% of the time 

To ship a feature like this and have it be useful, we needed to ensure that we weren’t filtering out true positives that developers should fix.  While a figure like 80% seems high in the world of SAST findings, in the case of noise filtering it would mean that 2 out of 10 times Assistant prevents a developer from seeing a real issue. 

In addition, noise filtering needs to be highly precise since security teams may want to quickly audit and triage filtered findings, knowing that the majority of them are false positives. Assistant explains why it thinks a finding is a FP, so spot checking a list of suppressed findings is fast - but the speed of this workflow diminishes quickly as accuracy falls. 

With all of this in mind, we developed Assistant’s noise filtering capabilities with a requirement that accuracy doesn’t dip below 90%. 

This is where we currently stand as of Jan 10, 2025:

You can read our metrics documentation for more stats and a more in-depth breakdown of our methodology, free of any marketing-speak.

The gist of it is that we leverage user feedback and internal benchmarking to ensure that Assistant is evaluated from both a user-centric perspective and through expert technical scrutiny - this gives our product and engineering teams the most realistic, comprehensive view into Assistant's real-world performance.

Trust but Verify

Like all Semgrep products, Assistant is built on a core tenet of transparency. When it comes to a semi-autonomous, AI-powered product, things must always be auditable and reversible - we’re not building a black-box AI product.

All pieces of contextual information that Semgrep Assistant stores as memories are fully accessible and visible to users. Admins can view, edit, delete, and manually add “memories,” and even adjust their scope on the fly.

When it comes to noise filtering, no findings are ever closed without your knowledge. Assistant simply prevents findings that are false positives from slowing down developers and damaging their trust in security tooling. 

Smarter Secrets Detection (beta) 

As previously mentioned, Semgrep's transparent engine, fast performance, and LLM-friendly rule syntax make it the perfect platform to effectively leverage AI. This isn't limited to SAST and SCA - Assistant is a versatile, cross-functional part of the Semgrep AppSec Platform. 

Take secrets detection. Identifying generic secrets like API keys, passwords, and tokens is critical to prevent credential leaks. Yet, traditional secrets detection tools rely on basic regexes to flag any randomized string, creating a flood of false positives that slow developers and erode trust with security.  

Semgrep's static analysis + LLM approach yields much better results:

1. Deterministic Rules: First, we run precise rules to reliably identify generic secrets using our static analysis engine. 

2. AI Context Analysis: Then, we run Assistant to filter out false positives by analyzing the surrounding code context. For instance, Assistant can exclude findings from test or example code, ensuring only true risks are surfaced.

By pairing Semgrep’s deterministic engine with AI, we deliver accurate, low-noise secrets detection—closing critical coverage gaps without the tradeoffs of traditional tools.

Conclusion:

These two features represent a major breakthrough in our goal to give security teams a personalized AI AppSec engineer that can handle routine tasks like triage and prioritization.  

With this launch, Assistant lifts a huge burden off of security teams, allowing engineers to focus on the important issues and initiatives that meaningfully improve security posture in the long term. For developers, the value is even more clear: spend more time coding, and finally trust that when your security tool alerts you, it’s because an real, risk-introducing issue needs to be addressed.