Skip to main content

Contributing code

Semgrep welcomes contributions from anyone. If you have an idea for a feature or notice a bug please open an issue. Creating an issue first is preferable to moving directly to a pull request so that we can ensure you're on the right track without any wasted effort. This is also a great way to contribute to Semgrep even if you're not making changes yourself.

This README gives an overview of the repository. For further information on building, you will be directed to semgrep-core contributing and/or semgrep-cli contributing in Making a Change.

File structure​

Semgrep consists of a Python wrapper (semgrep-cli) around an OCaml engine (semgrep-core) which performs the core parsing/matching work. Within semgrep-core, there are two sources of parsers, pfff, linked as a submodule, and tree-sitter-lang, built using tree-sitter. Additionally, semgrep-core contains a subengine, spacegrep, for generic matching.

You may also be interested in perf, which contains our code for running repositories against specific rulesets.

There are many other files, but the below diagram broadly displays the file structure. 

.
├── cli/  (Python wrapper)
│   └── src/
│       └── semgrep/
│ 
├── semgrep-core/  (OCaml engine for matching)
│   └── src/
│       │── pfff/ [submodule](https://github.com/returntocorp/pfff)
│       │── tree-sitter-lang/
│       └── spacegrep/  (Generic matching)
│           └── src/
│ 
└── perf/  (Performance benchmarking)

Most of Semgrep's logic is in cli/src and semgrep-core/src.

Code relationship​

The semgrep-core binary stands alone. Once built, it is possible to run semgrep-core on a semgrep rule for a given language with a file/directory and receive matches.

For example, say you create the config file unsafe-exec.yaml and the program unsafe-exec.py:

rules:
- id: unsafe-exec
  pattern: exec(...);
  message: Avoid use of exec; it can lead to a remote code execution.
  severity: WARNING
  languages: [python]
exec("ls");

If you run semgrep-core -config unsafe-exec.yaml unsafe-exec.py -lang python, it will output

unsafe-exec.py:1 with rule unsafe-exec
 exec("ls");

If you run semgrep --config unsafe-exec.yaml unsafe-exec.py, it will output

running 1 rules...
unsafe-exec.py
severity:warning rule:unsafe-exec: Avoid use of exec; it can lead to a remote code execution.
1:exec("ls");
ran 1 rules on 1 files: 1 findings

The matched code is the same, but with semgrep-cli the output is more polished and includes the message.

semgrep-cli invokes the semgrep-core binary as a subprocess, with a flag to request JSON output. It reads the semgrep-core output and transforms it appropriately.

Currently, depending on the flags used, spacegrep is invoked both independently by semgrep-cli as a subprocess and by semgrep-core as a subfolder. Therefore, semgrep-cli requires the spacegrep binary, but building semgrep-core will build spacegrep as well.

Making a change​

Semgrep runs on Python versions >= 3.6. If you don't have one of these versions installed, please do so before proceeding.

Because the Python and OCaml development paths are relatively independent, the instructions are divided into Python (semgrep-cli contributing) and OCaml (semgrep-core contributing).

To fully build Semgrep from source, start at semgrep-core contributing. It will direct you to semgrep-cli contributing when appropriate.

Depending on what change you want to make, it might be simpler to build only semgrep-cli or only semgrep-core. For example, if you only want to modify Python code, you can skip installing OCaml by downloading binaries for the OCaml parts. Similarly, if you only want to modify OCaml code, you can work on semgrep-core/spacegrep directly.

If you only want to build semgrep-cli, go straight to semgrep-cli contributing. Otherwise, follow the instructions in semgrep-core contributing.

Below is a guide for what functionality each of semgrep-cli and semgrep-core controls.

Only semgrep-cli​

The python code for Semgrep performs pre and post-processing work. You likely need to touch only semgrep-cli if you want to affect

  • How output is formatted
  • What files are scanned for each language
  • The message that is displayed

Go to semgrep-cli contributing

Only semgrep-core​

The OCaml code for Semgrep performs all the parsing and matching work. You likely need to touch only semgrep-core if you want to

  • Fix a parse error
  • Fix a matching error
  • Improve Semgrep's performance

Go to semgrep-core contributing

Both semgrep and semgrep-core​

There are some features that cross through both OCaml and Python code. You will likely need to touch both semgrep-cli and semgrep-core if you want to

  • Fix an autofix error
  • Add a new language
  • Change error reporting

Go to semgrep-core contributing. It will direct you to semgrep-cli contributing when appropriate.

Development workflow​

Before each commit Semgrep will run pre-commit to ensure files are well-formatted and check for basic linting bugs. If you don't have pre-commit installed the following command will do so for you:

python -m pip install pre-commit

Our pre-commit configuration uses Docker images. Please ensure you have Docker installed before running pre-commit. Install the pre-commit hooks with the following command:

pre-commit install

To ensure pre-commit is working as expected, run the following command:

pre-commit run --all

Once pre-commit is working you may commit code and create pull requests as you would expect. Pull requests require approval of at least one maintainer and CI to be passing.

Troubleshooting pre-commit​

On M1 macs some pre-commit tests may fail.

If those checks are running in docker containers (such as hadolint) and exit with code 137, this means they are running into a memory limit. This is because for running x86_64 images on an M1 mac, docker will utilize an emulation with qemu that can cause higher memory consumption. To fix this, change the memory limit in Docker Desktop in the Resources section of the Preferences, 8.00GB should be sufficient.