Enhancing developer happiness: The impact of identifying code-specific issues

Introduction

Automated security tools have made it easier to detect software vulnerabilities. However, the widespread use and adoption of these tools seem to have had little impact on the prevalence of vulnerabilities in almost all types of software. Traditional security testing tools are often deployed too late in the software development process, slow down agility and releases, and do not provide specific guidance to remediate the found vulnerabilities. To turn the tide, we need more developer-friendly security tools. Traditional security tools were designed for security professionals, and they are great at finding security problems. However, for developers, the problem is no longer just detecting but also preventing and fixing vulnerabilities – at scale.

Before I became a security researcher at Semgrep, I researched how developers interact, learn, and build trust with security tools. I wrote my Ph.D. thesis about this topic. Now that I work at Semgrep, I use the lessons I learned to help build a better security tool to overcome some of the problems of traditional security tools. I have set three important goals for security tools to become more useful to developers. 

A developer-friendly security tool needs to be: 

• Fast: Developers are used to real-time linters giving feedback; they don’t tolerate scans that take half an hour or longer.

• Relevant to the developer’s work: The best way to build trust with developers is to only show what matters to them and what they can act upon. Be selective about the rules that display findings to your developers. The metric that truly matters is the effective false positive rate.

• Usable and well-integrated into the developer’s workflow: Ideally, feedback is provided during the code review stage or even during the development process itself. The tool should be embedded into existing tools and give feedback in ways that the developers are used to receiving it. A great way to achieve this is to integrate feedback into GitHub PRs or, even better, in the IDE itself.

In this blog post, I want to discuss how rule customization can help each of these. We’ll talk about why you should do this and how tools allow you to do it.

Benefits of customizing rules

While customizing your rules takes effort and time, it generally pays off in several ways. 

First, the most obvious reason for tailoring your rules is that there will be security problems or guidelines for in-house code and libraries that security tools can not detect out of the box. Maybe certain function calls in your in-house library require specific arguments or should not be called with unsanitized user input. Security tools don’t have such project-specific information available unless you supply it.

These checks can go beyond security; they can be helpful for code quality and maintainability as well. When I made my first pull request to Semgrep, which is an open source tool, I used a different YAML parser than was used in most of the codebase. The reviewer caught this, and we added a check to the CI checks to detect this automatically in the future. 

Providing your security tools with project-specific checks and remediation makes it more likely that the feedback sent to your developers actually matters and is accurate. When that happens, you build trust with developers and this greatly improves their interaction with the tool. While researching different tools, I found that a 60% fix rate seems to be a ceiling for what can be achieved without rule customization. Tools that do allow customization and pay close attention to developer interaction easily reach north of 90% fix rate. You can read more about this in Google's Tricorder paper or my Ph.D. thesis.

To go beyond the finding itself, even if the vulnerable code pattern can be detected by your tool out-of-the-box, providing a solution is still often not possible. Just look at the simple example of a SQL injection: the solution can be to parameterize the query, or it can be to use one of several secure ORM libraries. If you want your tool to provide accurate and actionable remediation guidance, you could customize the message or, even better, provide autofix behavior that can apply code transformations to resolve the issue automatically for the developer. During experiments set up with developers completing programming tasks in a web application, providing autofixes like this immensely improved the effective false positive rate. 

Now that we’re on the topic of solutions. It is often much easier and much more efficient to detect the lack of using a solution than it is to detect a vulnerability. To go back to the example of SQL injection, It is perfectly possible that at the time a developer writes a SQL query in a function body, this function is never called with user input. In fact, functions are usually implemented before they are used, so this is very likely. That means at the time of development, traditional security tools cannot detect a SQL injection. It is only when this function is called from a (different) context where it receives user input that this can lead to a vulnerability. If we already know the desired outcome, i.e., a parameterized query or an ORM-based query, then we can detect the lack of this solution at the time of development. Your tool will be able to do this much faster as it requires less complex analyses, and the feedback will be shown to developers during development. This improves both the relevancy and the usability of your tool. But it requires that your tool knows what to check for. Interested in learning more about this approach? Check out my talk at OWASP BeNeLux, and check out these blog posts by Netflix and Semgrep.

Now that you understand why you should be customizing your rules, let’s take a look at how tools allow you to do this, and how to choose a tool that will make this easier for you.

How tools allow customization of rules

Some tools do not allow rule customization at all. Among the tools that do, several approaches exist. I will demonstrate a few approaches to a very simple security problem: the use of the outdated DES algorithm for encryption in Java. The aim will be to detect the following code snippet: 

1import java.lang.Cipher;
2...
3Cipher.getInstance(“DES”);

Application Programming Interface (API)

The first solution for tools to allow customization of rules is through an API. To customize the rules, the rule writer is required to write code that extends the tool’s functionality so that it performs additional analyses. The tool, or sometimes only the extension itself, then needs to be built into a new executable that can be used to analyze software products. In the case of Shipshape, it is required to expose this executable as a service using a docker image. An example of a commonly used tool that uses an API is SpotBugs (formerly FindBugs). In the following code snippet, you can see (part of) a SpotBugs rule that detects the use of DES.

1public class DesUsageDetector extends CipherDetector {
2    ...
3    @Override
4    int getCipherPriority(String cipher) {
5        cipher = cipher.toLowerCase();
6        if (cipher.equals("des") || cipher.startsWith("des/")) {
7            return Priorities.NORMAL_PRIORITY;
8        }
9        return Priorities.IGNORE_PRIORITY;
10    }
11    ...
12}

You can read more about the capabilities of their API in the documentation. The advantages of using code are that it allows the rule-writer to be very expressive and that it is usually a syntax the rule-writer is familiar with. The downside is that learning the API, writing code, and building the executable is much more time-consuming.

Custom language

To make customization of rules easier, some tools provide a custom language that makes abstractions of their API. They still require the rule writer to write code, but they provide a more specific syntax to make the development of rules easier. Code Query Language (CodeQL) is an example of such a query language. Below, you can find the CodeQL rule to detect the running example of DES.

1import java
2from MethodAccess call, Method method
3where
4  call.getMethod() = method and
5  method.hasName("getInstance") and
6  method.getDeclaringType().hasQualifiedName("javax.crypto", "Cipher") and
7  method.getParameter(0).toString().regexpMatch("DES.*")
8select call

Snyk also provides a custom query language for customizing rules. Here is the Snyk rule for detecting the use of DES.

1And<"java.crypto.Cipher", HasArg1<StringLiteral<"DES">>>

If you enjoy writing SQL queries more than writing code, this solution might be something for you. The downside to this solution is that the query quickly becomes more complicated once you start looking for more complex patterns.

Formatting language

Some tools allow the use of an existing formatting language (YAML, JSON, or XML) to customize the rules. The rule-writer will usually already be familiar with this language but will need to learn some specifics as to how it’s used for this tool. Many of these tools also provide a graphical user interface to “build” the rules, which makes learning the syntax even easier. Defining what to look for in code is then usually done using only strings that look like the code itself. Here are a few examples of tools using formatting languages.

SecureAssist uses XML.

1<Match>
2  <QualifiedName>javax.crypto.Cipher</QualifiedName>
3  <Method>getInstance</Method>
4  <Arguments>
5    <Argument>
6      <Index>0<Index>
7      <Value>
8        <ComparatorOperator>equals</ComparatorOperator>
9        <ExpectedValue>DES</ExpectedValue>
10        <ComparatorType>String</ComparatorType>
11      </Value>
12    <Argument>
13  </Arguments>
14</Match>

Sensei uses YAML

1
2search:
3  methodcall:
4    type: javax.crypto.Cipher
5    name: getInstance
6    args:
7      1:
8        type: java.lang.String
9        value: "DES"
10availableFixes:
11- name: "Use AES/GCM/NoPadding"
12   actions:
13   - rewrite:
14         to: '{{qualifier}}.getInstance("AES/GCM/NoPadding")'
15

Semgrep uses YAML:

1- id: des-is-deprecated
2  languages:
3      - java
4
5  message: DES is considered deprecated. Use AES/GCM/NoPadding.
6
7  pattern: (java.lang.Cipher $CIPHER).getInstance("=~/DES/.*/");
8
9  fix: $CIPHER.getInstance("AES/GCM/NoPadding");

Conclusion

If you want developers to trust and use your security process and not ignore its findings, look for tools that are fast, usable, and relevant. Make sure the tools are well-integrated into developer workflows, and use the tools only to show developers findings that they can act upon. The developer fix rate can be greatly improved by customizing your rules and by providing fixes through the tool itself. There are different solutions for tools to allow customization of rules; look for a tool that makes it easy to write rules without losing too much complexity.

If you are interested in writing custom rules, try Semgrep for free. If you are looking for a tool that easily allows you to encode which fixes should be used at your company and measure solution adoption, reach out to me! I am looking for people who want to try Semgrep to measure positive security indicators.