Running a multi-tenant WAF at the edge

By Reed Morrison, Software Developer

Web Application Firewalls (WAFs) are a critical layer in modern web security, providing a website’s first line of defense against vulnerabilities. WAFs can be used to defend against and notify on attempted exploits, allowing for mitigations faster than organizations can patch vulnerable software. For a global CDN, this functionality must be implemented in a way that is sensitive to performance, providing response times on the order of milliseconds. When we first introduced a WAF engine to the Verizon Digital Media Services stack three years ago, we selected the ModSecurity Rules Engine, which we found to be first-rate for individual WAF use cases. Furthermore, ModSecurity’s support of the OWASP Core Rule Set (CRS), powerful rule language, and API access to the HTTP traffic stream in real time offered significant flexibility.

Enter waflz

However, as the number of customers using the WAF increased, we experienced performance and resource bottlenecks. ModSecurity’s dense ruleset propagated across every customer instance drove memory and CPU utilization up across our network, increasing operational costs. Additionally, testing and deploying new rules was difficult because the rule language was often unwieldy and difficult to write and parse. These issues, along with development complexity with the existing ModSecurity library, led to the development of waflz, an open source WAF engine, published under the Apache 2.0 license.

For Verizon Digital Media Services, waflz is a significant improvement on ModSecurity because:

  • It consumes less memory.
  • Offers better performance.
  • Is API-driven.

Waflz supports a subset of ModSecurity capabilities, the OWASP Core rulesets 2.x and 3.x, and several third-party rulesets.

Designing waflz

Waflz is designed from the ground up to deliver high performance and multitenancy. When necessary, the design traded performance over flexibility. Ultimately, waflz supports a restricted subset of ModSecurity capabilities. For example, for security and performance reasons, some ModSecurity directives like SecRemoteRules and inspectFile were deemed unsuitable for running on the edge.

The engine can be configured with rules in either ModSecurity format or JSON. The entire WAF product was designed to be “API-first.” To this end, waflz provides first-class support for JSON as both inputs and outputs. Waflz uses Google Protocol buffers internally to represent both configuration (including rules) and alert formats. Choosing Protocol buffers allow for interoperability with JSON inputs and outputs, as well as adding strictly typed schemas for both.

Some of the principle engineering challenges in a CDN are dealing with the high concurrency and multitenancy that comes from serving thousands of customers: Every one of our global edge servers must be able to process a request for any of our customers as fast as possible. Furthermore, edge server applications must provide real-time patching and processing for any given customer configuration.

waf201
Fig 1. Customer config patching on an edge server.

Having many WAF rulesets loaded into the memory of the running HTTP application server process on the edge across all of our customers presented a scalability challenge. Waflz overcomes this issue by creating the WAF rulesets only once in memory and sharing read-only references between the customer configurations and rule customizations. Additionally, several potential performance optimizations were identified, which improved request processing times, including space and time savings to some critical internal data structures.

Waflz also has built-in features that make validating engine behavior and testing one-off ModSecurity rules easier, eliminating the need for a complex standalone test harness.

Conclusion

Waflz is part of our CDN technology stack and can operate at massive scale while enabling efficient granular rule testing and customization. Despite the challenges for CDN application, ModSecurity and the new libmodsecurity are fantastic, flexible libraries that are ideal for many use cases. Indeed, in the process of developing waflz, we’ve contributed back to ModSecurity development. The new WAF engine has been running in production globally for more than a year, concurrently supporting and protecting thousands of different client configurations without any loss in performance.

Visit github to view more examples of waflz functionality. Need to improve the security and performance of your website and apps? Learn about our Cloud Security Solution and Website Acceleration Solution now.