TL;DR:
Semgrep Assistant now provides greatly improved remediation guidance on 95% of the true positives it identifies.
As a result, developers have a 15% reduction in median time-to-resolution (MTTR), equating to 20 minutes saved per finding.
Improved guidance is rated as actionable by developers 78% of the time
Improved guidance naturally helps developers up-skill and learn secure coding practices, with no security courses or independent research required.
Autofixes are great, but tailored guidance is often better
Customers found Assistant invaluable immediately after we launched the product over a year ago. Findings that received a true positive rating from Assistant had more than double the fix rate as those that did not.
Interviews with users made it clear that the primary driver for this was autofix, the AI-generated code snippet Assistant would suggest to potentially fix an issue. Even when the code snippet needed some modification, it was usually enough to get developers on the right track.
However, in many cases we realized that incremental, step-by-step fix guidance would be far more helpful to a developer, especially since it’s not always possible for an LLM to generate a perfect code snippet.
To illustrate our point, let's look at an example Assistant autofix (no need to understand the technical details):
The fix for this issue is to add a base64-encoded cryptographic hash (SHA-256) using a tool like OpenSSL. Assistant clearly gets that right, so what's the problem?
Assistant doesn't just get the autofix right - it even provides an example/placeholder hash for the developer. So what's the issue?
Well, while this example hash is definitely helpful for a developer who knows they need to generate their own, it might be misleading or confusing for a junior developer, or a developer with less security knowledge.
If the developer doesn't realize that they need to generate their own hash to substitute in place of the example, they might end up making their application less secure by implementing a fix that only superficially addresses the issue.
There are a number of other reasons code snippets generated by LLMs can be limited in their usefulness / confidence:
Code changes are often outside the scope of the finding (e.g., requiring a change to a variable in a different part of the codebase).
Guidance can recommend things outside of code such as, “store this secret in AWS Secrets Manager” or, “update this setting in Datadog.”
In general, code syntax can be hard to get perfectly right.
Prior to this release, Assistant was able to generate high-confidence AI remediation guidnace for about 70% of findings. The new standard for Assistant is over 95%, and the format of the guidance/advice has been greatly improved as well.
Why developers want remediation guidance in human language
After a year of leveraging LLMs to help our users fix security vulnerabilities, we discovered 2 key insights:
LLMs are excellent at understanding and explaining code in human language.
Customers (and their developers) found a ton of value in the explanation of the code changes needed to fix a security issue
Despite only consisting of short paragraphs at the time, users said the explanations were extremely impactful for developers, especially junior developers or developers who lacked security knowledge.
The guidance also helped customers enforce secure guardrails, since after remediating a finding once developers gained the understanding to fix similar issues independently in the future.
Assistant's updated guidance capabilities
Semgrep Assistant now provides step-by-step remediation instructions in human language, with code snippets illustrating the fix incrementally.
This step-by-step guidance empowers any developer - even those with no security knowledge - to quickly fix a security issue, without ever leaving their existing workflow (PR/MR comments and Jira tickets):
Assistant's new, step-by-step guidance + autofix in a PR comment
For findings that are not automatically surfaced to developers, guidance makes it easier for AppSec folks to parse through scan results and quickly decide what to take action on / what to manually surface to developers.Guidance now appears on over 95% of true positive findings.
Prior to this release, Semgrep Assistant provided remediation suggestions in the form of a code snippet on ~ 70% of findings.
Now, the vast majority of security issues flagged by Semgrep come with detailed, step-by-step remediation guidance + a suggested autofix. With this update, almost every Semgrep finding is enriched with everything developers need to quickly and easily fix vulnerabilities before they hit production.
The impact on developers
Let's return to our example of adding a cryptographic hash.
With Assistant's updated guidance capabilities, Assistant instructs and explains how to actually generate a hash, and gives any user the tools and knowledge needed to fix the issue and adhere to the guardrail that was in violation.
The developer will see these instructions directly in their workflow - this is an example showing guidance in a PR comment:
Developers can also see remediation guidance in the description of any tickets created via Semgrep's Jira integration - here's the same guidance in a Jira ticket:
Guidance is far more effective at up-skilling a developer's knowledge of secure coding compared to learning about the vulnerability from a static resource, or conducting independent research via ChatGPT, StackOverflow, etc.
This means that beyond fixing individual instances of an issue, Assistant guidance will also help prevent developers from making the same mistake again.
More importantly though, remediating a finding by following step-by-step guidance is far less time consuming for a developer, so far more vulnerabilities fall under the category of "will fix right now".
The impact on AppSec engineers:
AppSec engineers can see the step-by-step remediation guidance directly in the Semgrep AppSec platform:
Guidance makes it easier for security engineers to parse through scan results in the Semgrep platform and quickly decide what to take action on and manually surface to developers.
With Assistant, security engineers won't have to write personalized fix guidance / code reviews since Assistant has already done so.
Our updated guidance can turn hours of analyzing findings and enriching them with context into minutes of validating generated guidance and spot checking auto-fixes.
Our beta users expressed an appreciation for how the extra context provided by the AI remediation guidance acted as a tailored code review from the security team. Most of our customers don’t have enough AppSec engineers to provide code reviews on every new finding; AI allows them to give developers personalized advice at scale.
Performance and continuous improvement
To ensure that remediation guidance is of high quality, we leverage our internal security research team to evaluate AI results every week. Our security researchers and a rotation of developers evaluate Assistant's guidance against what they themselves would have suggested.
This task force is rating the guidance as actionable 77.9% of the time.
In addition to getting fixed at 2x the rate as non-Assistant analyzed findings, the median time-to-resolution (MTTR) for findings with remediation guidance is 20 minutes faster than those without.
Let us know what you think!
We've been working super hard on this release, motivated by how much Assistant guidance benefitted our early users.
We're super excited to provide vastly improved guidance on nearly every finding Assistant identifies as a true positive (>95%). The result is a 15% reduction in median-time-to-resolution, leading to an average of 20 minutes saved per finding.
We'd love for you to try it out and let us know what you think!
