semgrep-cli
contributing
The following explains how to build semgrep-cli
so that you can make and test changes to the Python wrapper.
The semgrep-cli
name refers to the project which exposes the actual semgrep
command.
You may want to read the README first to understand the relationship between semgrep-cli
and semgrep-core
.
Setting up the environmentโ
You will need Python >= 3.6.
Most Python development is done inside the cli
directory:
cd cli
We use pipenv
to manage our virtual environment.
You can install it like this:
python -m pip install pipenv
Next we need to initialize the environment. This command will install dev dependencies such as pytest and will also install semgrep in editable mode in the pipenv.
SEMGREP_SKIP_BIN=true python -m pipenv install --dev
note
SEMGREP_SKIP_BIN` tells the installer that we will bring our own semgrep-core; see below.*
Getting the semgrep-core
binaryโ
Almost all usages of semgrep-cli
require the semgrep-core
binary.
To get this binary,
your safest bet is to follow the instructions in Building semgrep-core
,
which takes around 20 minutes.
Two shortcuts are available as alternatives, where you use a pre-compiled binary. The downsides of using a pre-compiled binary are:
- You will not be able to make edits to
semgrep-core
, for example to fix a parse error. - Semgrep will fail if the interface between
semgrep-cli
andsemgrep-core
has changed since the binary was compiled. This has historically been happening around every two months, but can happen at any time without notice.
With that in mind, the available shortcuts are:
The Homebrew shortcutโ
If you installed Semgrep via Homebrew with brew install semgrep
,
a semgrep-core
binary was bundled within that installation,
but is not made available on your $PATH
by default.
You can add the bundled binary to your $PATH
with this series of commands,
provided you have jq
installed:
export SEMGREP_BREW_INSTALLED_VERSION="$(brew info --json semgrep | jq '.[0].installed[0].version' -r)"
export SEMGREP_BREW_INSTALL_PATH="$(brew --cellar semgrep)/${SEMGREP_BREW_INSTALLED_VERSION}"
export SEMGREP_BREW_PYTHON_PACKAGE_PATH="$(${SEMGREP_BREW_INSTALL_PATH}/libexec/bin/python -m pip list -v | grep '^semgrep\b' | awk '{ print $3 }')"
export SEMGREP_BREW_CORE_BINARY_PATH="${SEMGREP_BREW_PYTHON_PACKAGE_PATH}/semgrep/bin"
export PATH="${SEMGREP_BREW_CORE_BINARY_PATH}:${PATH}"
The manual shortcutโ
Visit the releases page and grab the latest zipfile or tarball for your platform. Extract this archive and inside should be the necessary binaries. You can confirm this by running:
./semgrep-core --help
Copy this file to somewhere in your $PATH
so semgrep-cli
can find them. For
example, you may create a ~/bin/
directory within the repository. Include it in your $PATH
and run the binary from there.
Alternatively, you may include it somewhere like /usr/local/bin/
.
Running semgrep-cli
โ
You will want to be in the pipenv environment whenever you run semgrep. Start a shell with
python -m pipenv shell
Make sure you are in cli/
. Within the shell, run:
python -m semgrep --help
To try a simple analysis, you can run:
echo 'if 1 == 1: pass' | python -m semgrep --lang python --pattern '$X == $X' -
/tmp/...
1:if 1 == 1: pass
Congratulations, you have Semgrep running locally!
Installing semgrep
โ
You can always run semgrep
from cli/
, which will use your latest changes in that directory, but you may also want to install the semgrep
binary. To do this, run
pipenv install --dev
Some people have encountered difficulties with the above. If it fails, you can reach out to the semgrep
team on Slack.
If you have an M1 Mac, this may install the incorrect executable. You can run this instead
brew install semgrep
Now you can run semgrep --help
from anywhere.
If you have installed semgrep-core
from source, there are convenient targets in the root Makefile that let you update all binaries. After you pull, simply run
make rebuild
See the Makefile in cli/
Troubleshootingโ
For a reference build that's known to work, consult the root Dockerfile
to build semgrep inside a container. You can check that it builds with
docker build -t semgrep .
Testingโ
semgrep-cli
uses pytest
for testing.
To run tests, run the following command within the pipenv shell:
pytest
There are some much slower tests which run semgrep on many open source projects. To run these slow tests, run:
pytest tests/qa
If you want to update the tests to match to the current output:
make regenerate-tests
Running a single test file is simple too:
pytest path/to/test.py
Or running an individual test function:
pytest -k test_func_name path/to/test.py
semgrep-cli
also includes pytest-benchmark
to allow for basic benchmarking functionality. This can be run like so:
pytest --benchmark-only