Sometimes, it feels like things happen for a reason.
After a series of unexpected events, I now find myself as Head of Security Research at r2c, the company behind the open source static analysis tool Semgrep, and I couldn’t be more excited about my role, the company, and the future of application security (AppSec).
In this post I’m going to discuss why I’m betting on r2c, and where I think AppSec solutions are headed.
An academic wake up call
A Tale as Old as Time: Things are different outside academia
While I was a young, fresh-faced PhD student at the University of California, Davis, my colleagues and I pursued a number of research projects involving static analysis, and I even interned at Fortify, which was an up-and-coming static analysis security testing (SAST) vendor at the time.
I was amazed and enthralled by the power of static analysis to find bugs and improve software security at scale. Thoughts of other advanced topics like symbolic execution, abstract interpretation, and more joyfully danced through my head like a child’s dreams of presents at Christmas.
So when I joined NCC Group as a security consultant, I couldn’t wait to see the tools my colleagues used. After all, these are some of the top security testers in the world, brought in by most of the largest, and biggest name companies to improve the security of their software.
So what secret tools did these legendary security consultants use?
Burp Suite
An editor like Emacs, Vim, or VS Code
Aaaand...
grep(Pro-tip: use ripgrep, it’s much faster)
How could this be? How could such advanced approaches and tools exist and yet so few people use them?
It would take a few years, and off-the-record conversations with security engineers from dozens of companies, but eventually I found some answers.
Static analysis in the real world™
As a security consultant performing penetration tests, I quickly saw why we preferred grep:
Above all, consultants are time limited - they need to find bugs yesterday, and don’t have time to set up tools, run multi hour scans, and triage hundreds of false positives. Any tool needs to a) be useful out of the box, b) be able to get up and running in minutes, and c) provide real value.
Consultants rarely receive source code in buildable form, which is required by many tools. Again, a consultant could in theory work to build the source code, but that could easily become a time sink that wastes valuable testing time.
Most commercial static analysis tools have licensing models that make consultant use infeasible. Companies may charge by lines of code scanned, number of repos or developers, or some other factor that either doesn’t make sense or is cost prohibitive for lean consulting firms.
Most commercial static analysis tools are hard to customize - if there’s a code-base specific anti-pattern or bug class you’d like to find, writing and validating a custom check might take hours. That is, once you’ve already put in hours or days learning how to write custom rules in the first place.
But what was keeping companies, with bigger teams and more time and budget, from embracing static analysis?
Legacy SAST: slow, noisy, and out of touch
During my final few years at NCC Group, I was fortunate to have the opportunity to moderate DevSecOps focused panels all around the world featuring senior security leaders from companies including Netflix, Dropbox, Apple, Slack, Datadog, Etsy, DocuSign, and more.
And as a security consultant, I led a number of projects around helping companies scale their AppSec solutions, embrace automation, and tune and roll out a SAST tool they had purchased.
Here are a few things I learned:
Security teams are tired of tools that deluged them with false positives. They can’t send results directly to developers without damaging their relationships with engineering, but the small AppSec teams don’t have time to triage the results.
Most SAST tools require onboarding and tuning by a domain expert in order to really provide value. Tuning and writing custom rules, despite being high leverage, tend to be complex tasks and require days or weeks of focused time to become competent.
I was willing to put in that time because it was a topic I already had interest and a background in, but for most AppSec teams, they don’t have this in-house expertise and/or there are other more pressing matters-- they can’t dedicate one of their headcount for a few weeks to become proficient in the tool. Thus many SAST installations languish.
Many SAST buyers (e.g., heads of AppSec or CISOs) didn’t actually expect the tool to provide security value, but rather were buying it to tick a compliance checkbox. This (initially) surprised but saddened me.
Due to the price and complexity of customization, even very advanced, forward thinking AppSec teams were rolling their own static analysis solutions - which were often collections of regexes!
Most importantly, AppSec teams were starting to view security differently. More on this below.
Some anecdotes:
A mid-sized, rapidly growing startup said they had paid around $150,000 for a popular SAST tool whose output was 1 medium severity issue in the past year.
I considered asking them to pay me $150K and then I’d manually test until I found 1 medium severity issue and then leave, but unfortunately I did not.
In our off-the-record AppSec USA 2018 panel, my friend Zane Lackey asked the audience, “Who here is happy with their current SAST solution?” In a standing room only conference room with a few hundred people, not a single person raised their hand.
A senior sales professional at one of the big vendors came up to Zane afterwards and had some strong opinions on why their product was useful 😅
The biggest thing that stuck out to me though, after spending hundreds to thousands of hours in this space, both sounds obvious but is perhaps initially unintuitive:
It’s impossible to find every bug, no matter how advanced your tools are. Instead, the key to scaling security is to build secure-by-default libraries and tools that developers can use to prevent entire classes of vulnerabilities by construction, and then make sure developers use them.
This is what forward-thinking security teams at companies like Google, Microsoft, Facebook, Netflix, Dropbox, and more believe and have been investing in for years (see our Global AppSec SF 2020 slides for more details and examples).
This is not to say that bug finding (manually or with tools) doesn’t have a role, because it does, but rather that it’s not the highest leverage area security teams can invest in.
A crisis of faith
At this point I began to question my own goals, personal priorities, and beliefs about security in general - was I really pushing the industry forward, manually finding bugs in different applications?
It was also painful continually seeing companies pay so much for AppSec solutions they were unhappy with.
Enter: r2c
Naively, I started building my own static analysis tool - something simple and lightweight that fits easily into how modern security teams work. However, I soon found that building and maintaining a production quality tool yourself is A Lot of Work™.
In a random stroke of luck, my friend from grad school, Daniel DeFreez, told me about this small, SF-based startup called r2c, who were also building a lightweight static analysis tool.
I grabbed lunch with the team and attended a few of their meetups, and was surprised to find how closely our views of the future of security were aligned.
One day I said to Isaac, one of r2c’s co-founders and CEO, “You know, I think you should hire for a role like <this>,” and proceeded to pitch him my dream role, which was an unusual blend of everything: being a pro user of the product and influencing product direction, working closely with marketing to create and share security research that meaningfully pushes the industry forward, and much more.
To my surprise, he chatted with the team, and they ended up agreeing! 😍
Why I joined r2c
I joined r2c primarily for two reasons:
I believed they were on an ideal trajectory for helping shape the future of AppSec, and
I was impressed by the quality of the team and its culture.
Let’s get into both in a bit more detail.
The future of AppSec: killing bug classes via secure defaults
This topic merits its own post (and maybe blog series), but I wanted to at least touch on it here. For more info, see Isaac and I’s Global AppSec SF 2020 slides.
Historically, the security industry has focused on vulnerability identification, via pen testing, bug bounty, SAST, DAST, internal testing, and more. However, this is reactive and in general doesn’t prevent future vulnerabilities from being introduced.
However, there have been some promising developments - modern web frameworks like Django, Ruby on Rails, and others have a number of secure defaults and built-in guardrails that make potentially dangerous tasks safe by default, including context sensitive output encoding (prevent XSS), tight integration with object relational mappers (prevent SQL injection), and more. In my and many others’ opinions, this is why overall web security has improved, not all of the fancy bug finding tools we’ve built.
Forward thinking security teams at companies like Google, Microsoft, Facebook, Microsoft, Netflix, Dropbox, and many others have gone even further than what’s provided out of the box in common frameworks, creating secure libraries for parsing XML, authentication and authorization, mutual TLS between services, secret management, and many more.
The future of AppSec solutions is a one-two punch of secure defaults + lightweight enforcement of those defaults.
That way developers can do what they do best: rapidly build scalable, complex software that brings business value to your customers, and not have to constantly be wary of all of the subtle nuances that could lead to a vulnerability. In an ideal world, security should be completely transparent to developers.
By being fast, easily customizable, and open source, I think Semgrep is the perfect tool for enforcing secure defaults and blocking anti-patterns in a high signal way.
I have much more to say about this. Stay tuned for future posts!
The team
Technical Strength
A strong driver for me to join r2c was the technical chops of the team.
To be honest, it’s almost like some Hollywood caricature of a Silicon Valley start-up: the founders were roommates and friends at MIT, and the rest of the team is PhDs and degrees from places like MIT, Stanford, and Carnegie Mellon, senior leaders from other successful start-ups, and more.
True story: I was chatting with Drew Dennison, one of the r2c co-founders and CTO, during BSidesSF. I offhandedly mentioned:
Semgrep is pretty cool, but you know what would be super useful? An online IDE where you can write Semgrep rules without installing anything, like regex101 or Golang’s playground.
Drew agreed, and literally two days later, I got a text saying, “Hey, it’s live” and a link to what is now the Semgrep editor. This blew my mind 😱
That was the first of many times that’s happened to me at r2c.
It’s not uncommon to hear something like, “Hey, so I had a few extra hours, so I <built this entirely new feature or workflow.” Semgrep has added support for multiple languages in the few months since I’ve joined, which for most SAST companies, requires person-quarters to person-years for each.
Culture
One thing I didn’t think through as thoroughly as I should have before joining, to be honest, is r2c’s culture. Fortunately, I lucked out!
This is rare to find, but r2c has managed to assemble a team of people who are both brilliant and humble, who are happy to take feedback.
True story: On my first day, I had a one-on-one with Isaac, in which I gave him some pointed feedback about my thoughts on the direction of the company. As I was saying it, I realized that maybe it’s best not to say something critical about the company’s direction roughly 3 hours after starting to the CEO 😅
But Isaac laughed, thanked me for my candor, and we got into a frank, detailed conversation about why r2c was pursuing its current direction, things Isaac thought weren’t going as well as they could be, and open questions he was considering.
And on many occasions, I’ve been a part of conversations with the leadership team in which they’re proactively discussing how to make sure each r2c employee has the right guidance, support, and mentorship to be happy and successful in their role.
Frankly, that level of care and the granularity with which they’re looking out for people is not something I’ve seen many places.
What’s next?
We’ve made some exciting progress so far, but there’s much left to be done.
Here are a few immediate things on our agenda:
We’re going to continue adding more advanced program analysis features to Semgrep (e.g., constant propagation, taint tracking), further democratizing static analysis capabilities that used to only be available for six figure contracts.
We’re going to partner with OWASP, to help automate and scale their security expertise to security teams and developers around the world.
We’re going to get into no-holds barred fisticuffs with certain vulnerability classes, and show that real world companies can, in practice, actually eradicate them within their organization.
Stay tuned! 🚀
Clint Gibler is the Head of Security Research at r2c, and the author of the tl;dr sec newsletter.
