May 13, 2021

Real-Time Log Delivery now supports both CDN and security logs

By Mahrou Zia Ghazvini, Product Manager, Verizon Media

Our customers rely on Real-Time Log Delivery (RTLD) to obtain critical insights on the performance of our delivery network, identifying issues and making changes when needed. We are excited to announce that we have launched a new feature of our Real-Time Log Delivery that enables customers to stream WAF events into RTLD. Our RTLD WAF delivers log data for threats identified by our WAF's access rules and managed rules. It currently excludes log data for rate-limited requests, but look for this functionality supported in an upcoming release.

In addition to our new security feature, we have made several other updates to RTLD to help our customers collect, process and analyze the data they need to generate the exact visualizations that will help them make better-informed performance decisions.  Here is a summary of the new functionality:

  • Endpoints: Customers can stream raw CDN performance and security logs in less than 60 seconds to different supported endpoints, including Amazon S3, Splunk, Sumologic, Azure Blob Storage, Datadog, or their own web server via HTTP Post. Upcoming releases will include support for New Relic and Google Cloud Storage.
  • Profiles: RTLD offers broad customization with the ability to create multiple profiles. Each profile operates independently and allows full flexibility to select destination, filtering and output fields per profile. Since each profile is independent, more than one profile may be configured to deliver the same log data set.
  • Filtering: Filtering allows our customers to use RTLD for only specific segments of their overall traffic profiles, which helps reduce costs and resources. By focusing on the data that matters most to them, our customers can make better decisions sooner. Customers can filter CDN logs by edge CNAME, HTTP response codes (2xx, 3xx, 4xx, or 5xx), or user agent.  RTLD WAF logs can be filtered by edge CNAME, country code, security application manager, access rule, custom rule, or managed rule configuration.
  • Downsampling: In addition to filtering, customers also have the option for downsampling logs to 0.1%, 1%, 25%, 50%, or 75% of the set of log entries that will be delivered. For example, downsampling 1 million log entries to 1% would result in 10,000 log entries.
  • APIs: Customers can configure RTLD via APIs to gain efficiencies and interface our network with their preferred configuration method. RTLD configuration APIs give customers an efficient method for setting up and changing RTLD settings. See the architecture diagram below for details.  Access our REST API help center for RTLD here
  • Fields: Both RTLD CDN and RTLD WAF offer several fields to choose from. The fields are categorized into related buckets, making it easy for customers to select the information they want to receive while avoiding data not relevant to their business needs. With RTLD CDN, any info in a custom request or response header can be added as a data field to report on information being captured in headers. Cookie values can also be selected. A complete list of CDN log fields can be found here. View WAF log fields here.
  • Log Formats: For HTTP post, AWS S3, and Azure Blob Storage integrations, customers can choose from JSON, JSON Array, and JSON Lines formats. We will be adding .csv support in an upcoming release.

RTLD Architecture

RTLD supports several use cases, including:

Real-time CDN log monitoring

  • For general delivery of the logs, CDN administrators can monitor in real time how traffic is delivered and catch any performance issues or errors as they occur.

Large live events monitoring 

  • Live sports use case: The Verizon Media Platform supports the streaming and delivery of multiple live sports events. Using Real-Time Log Delivery in conjunction with video solutions, like slicer monitoring and ad data, provides an excellent way to monitor these events and act quickly should issues arise.

Business intelligence

  • Integrate with third-party log delivery endpoints to shine a light on traffic insights and trends to enable informed decision making.
  • Multiple profile use case: Send full log data to S3 for batch processing while separate profile sends non-2xx responses to Splunk to alert you of problems.

Get the intelligence you need to manage your web application and infrastructure in real time. Make better decisions faster with RTLD.  

Contact us today to learn more about this update and RTLD.

Contact us

Sales

Support

Manage your account or get tools and information.

More info