Contributing code
Semgrep welcomes contributions from anyone. If you have an idea for a feature or notice a bug please open an issue. Creating an issue first is preferable to moving directly to a pull request so that we can ensure you're on the right track without any wasted effort. This is also a great way to contribute to Semgrep even if you're not making changes yourself.
This README gives an overview of the repository. For further information on building, you will be directed to semgrep-core contributing and/or semgrep-cli contributing in Making a Change.
File structureโ
Semgrep consists of a Python wrapper (semgrep-cli
) around an OCaml engine (semgrep-core
) which performs the core parsing/matching work. Within semgrep-core
, there are two sources of parsers, pfff
, linked as a submodule, and tree-sitter-lang
, built using tree-sitter. Additionally, semgrep-core
contains a subengine, spacegrep
, for generic matching.
You may also be interested in perf
, which contains our code for running repositories against specific rulesets.
There are many other files, but the below diagram broadly displays the file structure.
.
โโโ cli/ (Python wrapper)
โ โโโ src/
โ โโโ semgrep/
โ
โโโ semgrep-core/ (OCaml engine for matching)
โ โโโ src/
โ โโโ pfff/ [submodule](https://github.com/returntocorp/pfff)
โ โโโ tree-sitter-lang/
โ โโโ spacegrep/ (Generic matching)
โ โโโ src/
โ
โโโ perf/ (Performance benchmarking)
Most of Semgrep's logic is in cli/src
and semgrep-core/src
.
Code relationshipโ
The semgrep-core
binary stands alone. Once built, it is possible to run semgrep-core
on a semgrep rule for a given language with a file/directory and receive matches.
For example, say you create the config file unsafe-exec.yaml
and the program unsafe-exec.py
:
rules:
- id: unsafe-exec
pattern: exec(...);
message: Avoid use of exec; it can lead to a remote code execution.
severity: WARNING
languages: [python]
exec("ls");
If you run semgrep-core -config unsafe-exec.yaml unsafe-exec.py -lang python
, it will output
unsafe-exec.py:1 with rule unsafe-exec
exec("ls");
If you run semgrep --config unsafe-exec.yaml unsafe-exec.py
, it will output
running 1 rules...
unsafe-exec.py
severity:warning rule:unsafe-exec: Avoid use of exec; it can lead to a remote code execution.
1:exec("ls");
ran 1 rules on 1 files: 1 findings
The matched code is the same, but with semgrep-cli
the output is more polished and includes the message.
semgrep-cli
invokes the semgrep-core
binary as a subprocess, with a flag to request JSON output. It reads the semgrep-core
output and transforms it appropriately.
Currently, depending on the flags used, spacegrep
is invoked both independently by semgrep-cli
as a subprocess and by semgrep-core
as a subfolder. Therefore, semgrep-cli
requires the spacegrep
binary, but building semgrep-core
will build spacegrep
as well.
Making a changeโ
Semgrep runs on Python versions >= 3.6. If you don't have one of these versions installed, please do so before proceeding.
Because the Python and OCaml development paths are relatively independent, the instructions are divided into Python (semgrep-cli contributing) and OCaml (semgrep-core contributing).
To fully build Semgrep from source, start at semgrep-core contributing. It will direct you to semgrep-cli contributing when appropriate.
Depending on what change you want to make, it might be simpler to build only semgrep-cli
or only semgrep-core
. For example, if you only want to modify Python code, you can skip installing OCaml by downloading binaries for the OCaml parts. Similarly, if you only want to modify OCaml code, you can work on semgrep-core
/spacegrep
directly.
If you only want to build semgrep-cli
, go straight to semgrep-cli contributing. Otherwise, follow the instructions in semgrep-core contributing.
Below is a guide for what functionality each of semgrep-cli
and semgrep-core
controls.
Only semgrep-cli
โ
The python code for Semgrep performs pre and post-processing work. You likely need to touch only semgrep-cli
if you want to affect
- How output is formatted
- What files are scanned for each language
- The message that is displayed
Go to semgrep-cli contributing
Only semgrep-core
โ
The OCaml code for Semgrep performs all the parsing and matching work. You likely need to touch only semgrep-core
if you want to
- Fix a parse error
- Fix a matching error
- Improve Semgrep's performance
Go to semgrep-core contributing
Both semgrep
and semgrep-core
โ
There are some features that cross through both OCaml and Python code. You will likely need to touch both semgrep-cli
and semgrep-core
if you want to
- Fix an autofix error
- Add a new language
- Change error reporting
Go to semgrep-core contributing. It will direct you to semgrep-cli contributing when appropriate.
Development workflowโ
Before each commit Semgrep will run pre-commit
to
ensure files are well-formatted and check for basic linting bugs. If you don't
have pre-commit
installed the following command will do so for you:
python -m pip install pre-commit
Our pre-commit
configuration uses Docker images. Please ensure you have
Docker installed before running
pre-commit
. Install the pre-commit
hooks with the following command:
pre-commit install
To ensure pre-commit
is working as expected, run the following command:
pre-commit run --all
Once pre-commit
is working you may commit code and create pull requests as
you would expect. Pull requests require approval of at least one maintainer and
CI to be passing.
Troubleshooting pre-commitโ
On M1 macs some pre-commit
tests may fail.
If those checks are running in docker containers (such as hadolint
) and exit with code 137, this means they are running into a memory limit.
This is because for running x86_64 images on an M1 mac, docker will utilize an emulation with qemu that can cause higher memory consumption.
To fix this, change the memory limit in Docker Desktop in the Resources section of the Preferences, 8.00GB should be sufficient.