Building the Fastly WAF

In keeping with our security team’s vision for defending the modern web, we launched our Web Application Firewall (WAF) to help our customers secure their sites and applications while providing reliable online experiences for their users. In this post, we’ll take a look into the tech behind our WAF, exploring how we built a performant, highly configurable, and comprehensive solution to help you secure your infrastructure.

Varnish-backed + OWASP foundation

At the core of Fastly’s stack is the Varnish cache, and the Varnish Configuration Language (VCL). VCL gives Fastly users (and Fastly as well) a declarative HTTP processing language with operators and features like regular expressions and transformations. We implement WAF using VCL, translated from the ModSecurity project and language using internal code. The WAF executes in pure VCL on every cache node inside of a service configuration, for Fastly-compatible performance and control.

The Fastly WAF also offers better performance as compared to other solutions because it’s not a separate offering; as CEO Artur Bergman put it, “We will always insist that every component of the Fastly platform is fully integrated, as we don't limit features to subsets of our network.” Because we maintain our WAF with the in-house tools and skillsets we’ve cultivated (rather than devoting resources to maintain a totally separate entity), we can ensure it’s fully integrated with our edge cloud platform.

We used the OWASP Core Rule Set (CRS) as a foundation to launch with a large set of rules that give coverage for the OWASP Top Ten web application risks. Similar to the OWASP CRS, Fastly’s CRS implementation uses a variety of detectors, many as regular expressions, to screen requests that look suspicious. From there, a configurable threshold can determine if you wish to permit or reject the request. We complement the CRS with application-specific rules from a commercial partner, Trustwave, as well as our own internally-developed rules, allowing us to cover hundreds of applications if the OWASP CRS fails to detect the attack.

In combination with the Fastly API, our WAF gives you the ability to pick which individual rules to run; many customers enable all of OWASP, minus rules that generate false positives based on their specific traffic patterns, and then set a threshold after a tuning process.

Once you’re set up with WAF (let us know here and we’ll get you set up), the Fastly stack takes that configuration along with the rules you picked and applies a transformation process to generate VCL, which is then injected into the service and runs as part of your entire VCL workflow.

Drawing on OSS for rulebase selection & composition

There are thousands of web application vulnerabilities and tens of classes of attacks; our WAF needed to have a robust ruleset that covered popular applications and well-known vulnerabilities as well as a ruleset that’s been tested and validated. You can do this all in-house, although that’s a significant amount of work; we quickly realized that instead of reinventing the wheel, we’d take a look at what’s come before.

We settled on building on efforts of the ModSecurity team, an open source, established, and community effort, which has benefited from years of insights and testing. While we didn’t use the ModSecurity framework, we chose to extend the rulesets available in it. Marrying the ModSecurity component to Varnish would have been challenging, as it was originally built for Apache, and although it’s been ported to Nginx and IIS, it’s still only a first-class module on Apache. While we could have deployed Apache with ModSecurity, we felt this was an unneeded complexity. Instead, we built our WAF in Varnish using VCL (more on this later).

The Fastly WAF incorporates the OWASP CRS, which covers the OWASP top ten risks, in an application agnostic fashion, enabling protection for new exploits using well-known techniques. OWASP CRS works by applying dozens of rules against a request and scoring them to detect anomalies, such as multiple encodings, tokens often seen in SQL injection attacks, etc. In the ModSecurity model, a request is scored with these rules and finally a user-configured threshold is checked to determine if a request should be dropped or passed to origin. As part of the Fastly WAF, we’ve implemented their anomaly scoring system in VCL. In conjunction with the Fastly team, you’ll set this threshold after a tuning period to ensure accuracy and acceptable decisions on requests.

In building a product on the OWASP CRS, we’ve found multiple bugs and submitted patches to the upstream project. Some of these have emerged in releases like 3.0.1, and some are still forthcoming (but we’ve applied the fixes internally at Fastly).

While the OWASP CRS ruleset is broad, it doesn’t cover everything. To close the gaps for specific application vulnerabilities, we went with Trustwave as our commercial partner — they employ a large team of security analysts who track vulnerabilities reported or detected, along with exploits seen released or used in the wild. Whatever the OWASP CRS fails to stop is then a candidate for an application-specific rule. This is infrequent, a testament to the broad detection afforded by the OWASP CRS ruleset.

Finally, gaps in rulesets still crop up from time to time, which is why we have a third ruleset: Fastly internal rules, which we’ve written ourselves to provide specific coverage that’s not afforded by OWASP CRS or the Trustwave rulesets. VCL is a far more expressive language than even ModSecurity’s syntax, and so we can write complex rules to address issues that we see under open attack but missed by external rulesets, and VCL allows us to react to such exploits quickly. If you’re a Fastly customer, you’re already familiar with VCL and can look at rules, so you don’t have to learn a separate technology.

Remember that the OWASP CRS ruleset tries to determine the shape of a request: a single request can match multiple rules as it tries to determine if it’s malicious. For each of those matches, Fastly logging gives you a debug log entry, allowing you to see what the anomaly score is after each match, enabling diagnostic workflows and setting thresholds. With Fastly WAF APIs, you can disable specific rules and set thresholds as necessary.

Ruleset management

The Fastly WAF API gives you a tremendous amount of control: in an effort to create a totally comprehensive solution to empower you to make decisions, we expose thousands of rules. Obviously managing thousands of rules by ID alone would be too onerous, so we use the tag that the rulesets have on them, enabling rapid coverage for vulnerabilities in PHP, Java, WordPress, Drupal and the like.

Similarly, ruleset tags give you insight into the kinds of applications covered. To give you an idea of how extensive that list is, here’s a tag cloud of thousands of the tags associated with the rules:

Leveraging the data

One of the tools we use in WAF development and research is a small network of honeypots, running applications which appear to be vulnerable that sit behind a Fastly WAF service. We collect the logs and process them just as a customer would, yielding insight into attacks in the wild and the clients sending them, which offers us key insight into attacker tool development.

Our Fastly WAF honeypot service generates the same logs that customers receive, enabling us to evaluate its utility and build applications using that data. One of the internal services we built on this log processing is a tuning report — also used by customers when onboarding with a Fastly WAF — that consumes the same logs customers receive, and we visualize them with an internal visualization tool built on Google Charts and HiCharts.

This first plot, called a “box and whisker” plot, shows the range of anomaly scores for each rule category observed over a 24-hour time period in the honeypot. For each vertical bar, the heavy part shows the second and third quartiles of the observed values, the white line in the middle shows the mean, and the thin bars show the range of the first and fourth quartiles. This allows you to quickly visually inspect the day’s WAF activities and determine what kinds of attacks you’re seeing, what range of anomaly scores are reported, and from there begin to choose what thresholds to set per category and overall.

Similarly, we can plot the client locations based on geolocalization of the IP addresses seen in the honeypot. Using a Google map chart, we can indicate where these clients originate by shading each origin country darker. In the logs from the above sample, Russia dominated, with contributions from the US and the UK, though these values vary from day to day. Typically with the honeypot we’ll see an individual client scan exhaustively with their tools trying to break in, leading to this sort of domination by a single country. From our vantage point it offers insights into emerging exploit activity and tool development.

Going forward

A key aspect of developing a security product is testing its efficacy and accuracy as an integration test; as part of our WAF development process, the Fastly security research team built a new WAF testing tool. Dubbed “FTW,” the Framework for Testing WAFs supports regression testing of rulesets — i.e., lets us know that new changes and features work, and haven’t broken any existing features. Using a YAML-based test specification format, users can specify every aspect of an HTTP request, including severely malformed requests, to test the ruleset efficacy. We released the tool late last year and it’s seen a significant contribution from the OWASP CRS community.

In an upcoming post we’ll dive deeper into how FTW works, but suffice it to say the testing framework has allowed us to identify issues quickly and address them. We’ve even found issues in the CRS rules (and shared pull requests back upstream, some of which made it into dot releases), and continue to use it to capture regressions and track their resolution internally. Using this approach, we can ensure that the WAF works as we intend it — the control APIs, ruleset, and translator that yields the VCL that defines our WAF.

Remember, no security product will detect or prevent all possible attacks or threats.