Thought experiment: should the use of non-cryptographically secure random() be flagged to developers (examples: Bandit and gosec)? Assume we find the function calls 100% accurately (all true positives, no false positives), and we prevent merging when it’s present.
Academics would say yes — the random() rule always detects what it promises and could be a vulnerability. But the practitioners amongst us are likely to hesitate — does it matter enough to stop engineers’ work? Isn’t random() often used for non-cryptographically sensitive operations?
Herein lies the disconnect between static analysis theory and practice: the world’s most accurate static analysis rule is valueless if developers don’t agree the result has impact, yet as an industry we use the rate of true vs. false positives as the measure of quality.
At r2c we wanted to know how static analysis rules for Semgrep perform in practice. So we set out to create a better metric for Semgrep App users. We call it Fix Rate.
Fix Rate is the percentage of merge-blocking findings that are fixed (i.e., not muted*) in CI. We believe this is a proxy for engineering value. As we run our own developer-focused security programs, we’re obsessing over how to increase the Fix Rate for the rules on our projects.
Observing a bad 0% Fix Rate for random() (with only 7 data points from our projects), we decided to silence the rule for r2c developers but send a Slack notification to the security team using multi-policy projects and CI audit mode. Even in this first version, we’re finding Fix Rate incredibly powerful for evolving our own rules, policies, and development practices.
We shared Fix Rate with the Figma security team early on, which they used for rule development.
With its GitHub integration, Semgrep brings security analysis to where development happens. Figmates get security feedback in their PRs, while rule analytics gives the security team feedback on the effectiveness of our rules and patterns. The simple grep like syntax allows us to extend Semgrep to catch new patterns, going from idea to live in an hour.
— Dev Akhawe, Head of Security, Figma
We’re excited for developer action to determine the efficacy of static analysis rules. For this you need to surface real findings to real engineers writing real code; no pre-existing benchmark will suffice.
To a future of fast, delightful, and confidence-inspiring static analysis.
Luke O’Malley
Head of Product, Co-Founder @ r2c
*Semgrep users can mute a finding with // nosemgrep inline comment or via a project-wide ignore file. See docs.
