I want to get something straight: you do not need to put a dynamic scanning tool into your CI/CD pipeline in order to do DevSecOps properly. You don’t even necessarily need to use automated dynamic analysis at all to do DevSecOps.
I used to do regular consulting via IANs Research, and quite often, I found myself assuring clients that “Yes, what you are doing makes perfect sense. You are covering all of your bases. In fact, you’re doing a GREAT JOB.”
So why the mystery? Why the uncertainty? Let’s dive a little deeper.
What is Dynamic Analysis?
Dynamic (when referring to a system, not a person) means constant change, activity, or progress. When we perform dynamic testing on a technology system, that means interacting with it as it is running. This could mean using live software that is hosted on a web server or a smart fridge that is turned on, with real food inside of it.
Dynamic testing can be performed from within an application, which is what IAST (Interactive Application Security Testing) products promise: testing your running app from the inside out. More commonly, dynamic analysis is performed from outside the application, often with a web proxy, an automated DAST (Dynamic Application Security Testing) tool, or manually with a web browser or direct calls to the system (such as API calls).
Some of the advantages of dynamic analysis include; Getting to see how it actually works, discovering if some of the behaviors are not what you were planning from a business perspective, and trying to find business logic issues (which are often caused by design flaws), and you can validate whether a vulnerability found by an automated tool is exploitable (or not).
"A DAST (Dynamic Application Security Testing) is generally considered a software product that scans web applications and API for vulnerabilities, (generally) performing both active and passive scans. It works in a completed automated way, such that you do not need weeks, months or years or training to be proficient with it. Sometimes DAST tools are also called a VA (vulnerability assessment) scanner or a ‘web app scanner’. "
- Me
The most obvious disadvantage of dynamic testing is that you can’t see the code. This is often called black box testing, where you don’t get to know the design, the functionality, or anything else about the application before you perform your test. Being able to see how the code works, or a network or architecture diagram, can help someone with a malicious mindset find more vulnerabilities faster.
"A web proxy is a software product that can be used for manual, dynamic, security testing. Penetration testers often use web proxies while testing APIs and web applications. Sometimes products combine both the DAST and Web Proxy functionality into one product, and, unfortunately, those are often called a DAST or Web Proxy, as though the terms are interchanged, which leads to more than a little confusion."
- Also me
Other disadvantages of dynamic testing are that there are a whole bunch of different types, and sometimes it gets confusing. When using an automated DAST scanner, pretty much anyone could operate it (this is an advantage). This means that the bar to entry is very low, and you don’t have to hire an expert, which can be expensive, and it can also be quite difficult to attract that type of talent on a permanent basis. That said, automated dynamic scanners, when operated by someone without very much training, can result in bad data being injected into your database, not testing your entire attack surface, inability to talk directly to APIs, and more. Although it’s wonderful to have this automated functionality scanning legacy apps, and finding lots of old bugs in your ancient code, for more modern apps… Some automated DAST tools leave a lot of untested attack surfaces behind.
It should be noted that each DAST and each web proxy product works differently. Some have great scheduling automation options, some don’t. Some are only able to automate passive scanning, while others can do both active and passive scanning. Not all options discussed in this article are available for all products.
- Me, based on comments from my friend Rick
Penetration testing
This brings me to penetration testing. Penetration testing usually involves a whole bunch of tools, an entire toolbox, if you will. It also generally involves at least one extremely skilled security testing expert. They manually test the application by using a series of tools (some automated, some not), to find as many bugs as possible, then validate each one’s exploitability. They only report what they feel to be legitimate business risks, vulnerabilities, or other issues that they feel could hurt your system, your business, your employees, users, or customers. Just the important stuff!
But there are more types of dynamic testing than just automated DAST and PenTesting, and all of them count under the giant umbrella of the term dynamic. Performance testing, stress testing, DDoS testing. All of those are dynamic; they interact with your application to find out if there are problems you should be aware of.
In addition to the traditional DAST scanners, there are newer fuzzing and dynamic scanners that are created only for APIs (application programming interfaces), and they are looking more and more promising every month. In 2020 and 2021, several new API companies came on the market with amazing new products. A lot of them offer dynamic forms of analysis that are different than anything I had seen before.
One of the examples I saw in San Francisco as part of the RSA festivities was an IDE plugin that would allow the developer to fuzz each one of the fields within their API in an automated fashion. Fuzzing means adding in all sorts of bad input to test the input validation of a system. The person demoing it for me, Isabelle Mauny, showed me how it could look at the API definition file, and then automatically generate tests for you. Holy smokes, nothing like that existed when I was a dev!
I’ve also seen some really amazing functionality involving monitoring for data that is being exfiltrated (watching for potential data breaches). A regular web application firewall can be configured to watch for unusually large HTTP responses (think: a whole heck of a lot of data, way more data than makes sense). And that can be quite helpful. However, some of the web application firewalls made for APIs, another monitoring product for APIs, can watch for when you have made a grave error in your access control. They can check to see if perhaps the request that you created has brought back more records than it should have, or different fields that were not expected to be brought back. With the biggest threats to APIs (according to the OWASP API Security Top Ten) being broken authorization at all levels, that’s some pretty spectacular coverage. Although this is more of a shield, than a dynamic test so to speak, we have to choose what reduces business risk the most, and this might protect you from a myriad of issues that an automated tool would likely miss.
Other nifty dynamic functionality that is made only for APIs include inventory tools. Often, they perform active (sending their own test requests) or passive (checking requests and responses for security problems, but never sending their own requests) dynamic scans at the same time they perform inventory, telling you immediately if they spot something new that might be a problem in your API. They can find all your APIs, including some that you thought were decommissioned months or years ago! I personally find this to be an extraordinary step forward in making sure that you have complete tooling coverage of your application portfolio. When I started in AppSec, I would never have imagined that I could have tools that could stop an error within just a few minutes of it being released into prod!
On top of this is WHERE you can do testing with all of these cool new dynamic tools. You can test directly in the IDE (integrated development environment), AS YOU WRITE YOUR CODE! That’s amazing, and the furthest ‘left’ security could ever push from a tooling standpoint. Some of them can be run nightly or even continuously in your production environment. When I started in AppSec, I had to manually run every single scan for dynamic tests. Now I can ‘set it and forget it’, only receiving reports when it finds new bugs. It’s a security nerd’s dream come true!
This leads me back to the title of this blog post: you truly do not need to run an automated DAST product in your CI/CD to say you’re doing DevSecOps. It is NOT a requirement! You can run all sorts of different types of dynamic tools, in several different places (IDE, against prod and pre-prod, continuously in prod, or CI/CD), and still do a great job and have excellent coverage. The key with DevSecOps is ensuring whatever you do follows the processes of the DevOps folks where you work and that it works within the Three Ways of DevOps (providing fast feedback, optimizing efficiency for the entire system, and aiming for constant improvement and learning).
If the way you run your tools slow down the pipeline for everyone, that’s not a win. If the tools you choose don’t get you good coverage, that’s not a win. If the tools you have report a lot of false positives, that’s never a win. Instead of trying to follow what the vendors and marketing materials tell us, focus on finding what creates the best results for YOUR org. Every dev shop is unique, and thus your security program will be too!
Potential alternatives to running an automated DAST in your CI/CD
Automating a DAST to run monthly, overnight, receiving an email in the morning: set it and forget it!
Focusing almost exclusively on static forms of analysis (SAST, SCA, code review, secret scanning), and then pentesting the important apps once a year (pentesting is a form of dynamic testing)
Move towards a micro service architecture, where the front-end GUI is dumb (no business logic) then, when you’re ready, switching your old toolset for modern API-specific tooling, plus continuing with static and other forms of testing.
Use DAST manually, but only for legacy monolithic apps (think of it as backward compatibility), PenTest the 2-5 most important apps, then use API tools for dynamic testing of APIs, plus (continuous) monitoring and inventory for extra coverage.
Ditch all dynamic testing, and just do static forms of testing (only recommended if you have a limited budget and you only have time and money for one tool, and for some reason do not want to use free DASTs).
Install an IAST tool into all of your apps, and use it in pre-prod environments and/or prod environments. Then you could skip DAST, or just PenTest the important apps, on top of the IAST.
PenTest everything once a year (the most expensive option, and certainly not the best, but I’ve seen it)
PenTest just your 3-5 most important apps then cross your fingers for the rest of them (not recommended, but more popular than you might think!)
None of the above options include non-tooling activities and support you can provide, which I always recommend in addition to tooling!
Training (secure coding, how to use security tools, secure design, threat modeling, etc.)
Security best practice instructions for each type of technology
Architecture and design review
Threat modelling
Security requirements for every project
Security Champion programs
I could go on forever! There are many other ways than just buying tools to support a secure system development life cycle (S-SDLC or SSDLC).
All of this aside, try not to let yourself get too caught up in what you read on the internet (this blog post included) and instead focus on what you and your team feel gets you the best coverage, fits your budget, and works WITH the developers and the processes they use. If you’re really struggling, it might be time for a change.
PS: If you are interested in trying out a SAST product that scans your code within seconds, try out Semgrep!
