Shoulda, Woulda...Coulda

In the world of SAST it’s very easy to spot two types of findings:

• True positives: A confirmed vulnerability

• False positives: Something which is not a vulnerability

This is due to the SAST engine reporting findings to the user, and then manual intervention takes place, where the user reviews the findings and determines if they are either a false positive or true positive. There is one lingering thought in the back of your mind, what about false negatives?

• False negatives: A vulnerability which was not detected

Working with SAST tooling is an iterative process. It can look like:

1. Run a scan, find some vulnerabilities, triage for true or false positives

2. Discover false negatives through manual code review or third-party reports

3. Create new or modify existing rules, and ingest them into your pipeline

4. New scans will identify new variants, repeat

We know this is an important part of your SAST journey, so it’s extremely important for us to not just to understand false positives, but also false negatives.

Introducing Shouldafound

Recently at r2c we introduced a new Semgrep CLI feature called shouldafound which enables customers to report false negatives directly to our security research team, who can triage and either update existing rules, or create new rules to capture those new variants.

Shouldafound is useful for various contexts such as:

• A customer's bug bounty report uncovers a traditional SQL injection vulnerability which should have been caught by Semgrep

• Someone performing a code review and wants Semgrep to find particular issues they found manually

• A security consultant reviewing customer code, and adding anonymized versions of their finding so we can help them find that issue automatically in the future

Once you have discovered a false negative, you can easily report missed vulnerabilities by running:

semgrep shouldafound --email "my@email.com" \
--message "Semgrep shouldafound this vulnerability in my code" \
--start 5 --end 10 path/to/my/file.go

Note: It’s important to anonymize the data you send to us, as it creates playground links which are public to anyone with access to the URL.

In this case, Semgrep extracts lines 5 through 10 from path/to/my/file.go and creates a support ticket for our security research team with email my@email.com. The ticket includes a playground link with the extracted code and corresponding message Semgrep shouldafound this vulnerability in my code. Before creating a ticket, the command will remind you that this ticket is public and you should anonymize the code you include if appropriate. We then review the report and decide to update a rule or create a new one, our security research team follows up on the ticket with updates!

We truly hope this feature helps you improve your false negative rate so you can focus on more important tasks.