Log streaming

Fastly supports a variety of third party log endpoints as recipients of log data streams emitted from Fastly services. Your services on Fastly emit log events, and the log endpoints receive them. Supported log endpoints include generic protocols (so you can operate your own log receiver), and dedicated proprietary connectors for popular third-party services.

Setting up remote log streaming

Logging endpoints can be configured via the web interface, API, or CLI, and are available to both VCL and Compute services.

  1. VCL
  2. Compute

Creating a log endpoint in a VCL service will normally trigger automatic log generation, meaning you do not have to separately emit logs within your service. For simple use cases, this is a convenient way to set up logging in a single step.

If you would prefer to emit log messages using your own code, first set up the endpoint (setting the placement property to 'none'), then generate log events in your code that target your configured log endpoint.

Follow the guide for setting up remote log streaming for VCL to learn more.

Generic log endpoints

Generic log endpoints can be used to instruct your Fastly service to send logs to a destination of your choice, including your own logging infrastructure.

HTTP endpoint: Challenge request

When sending logs to an HTTP endpoint, Fastly requires proof that you control the domain name specified in the URL field. We verify this by sending a challenge request to /.well-known/fastly/logging/challenge. The response to a challenge request must include a SHA-256 hash (in a hex string format) of your Fastly service ID and it must appear on its own line in the response. If multiple Fastly services are configured to use the same log endpoint, multiple hex(sha256) values can be added to the challenge response. Alternatively, an asterisk (*) can be used on a line to allow any service to post to the HTTP endpoint. For example:

GET /.well-known/fastly/logging/challenge HTTP/1.1
HTTP/1.1 200 OK
Content-Type: text/plain
Content-Length: 132
ef537f25c895bfa782526529a9b63d97aa631564d5d789c2b765448c8635fb6c
06ae6402e02a9dad74edc71aa69c77c5747e553b0840bfc56feb7e65b23f0f61
*

The correct value for your challenge response can be generated on most unix-like systems by running sha256sum

$ echo -n "YOUR_FASTLY_SERVICE_ID" | sha256sum
ef537f25c895bfa782526529a9b63d97aa631564d5d789c2b765448c8635fb6c -

Syslog endpoint: Facility and severity

All log messages delivered to syslog endpoints have a facility of local0 (a user-defined code in the syslog standard) and a severity of info (level 6).

Dedicated integrations

We support a variety of third-party services, which may be configured via the web interface, API or CLI.

IMPORTANT: Third-party services to which you send logs from Fastly may vary substantially in the constraints they impose on log messages. For example, both Datadog and Google's BigQuery require that messages are JSON-formatted, but they have different requirements for the format of date/time values. Other providers (e.g., Amazon S3) accept nearly any kind of log message. Consult your log provider's documentation to understand what format is expected.

The following services are currently supported via dedicated integrations:

NameInstructions
Amazon KinesisWeb interfaceAPICLI
Amazon S3Web interfaceAPICLI
DatadogWeb interfaceAPICLI
DigitalOcean SpacesWeb interfaceAPICLI
ElasticsearchWeb interfaceAPICLI
Google BigQueryWeb interfaceAPICLI
Google Cloud Pub/SubWeb interfaceAPICLI
Google Cloud StorageWeb interfaceAPICLI
Heroku LogplexWeb interfaceAPICLI
HoneycombWeb interfaceAPICLI
LogglyWeb interfaceAPICLI
Microsoft Azure Blob StorageWeb interfaceAPICLI
New RelicWeb interfaceAPICLI
PapertrailWeb interfaceAPICLI
Rackspace Cloud FilesWeb interfaceAPICLI
ScalyrWeb interfaceAPICLI
SplunkWeb interfaceAPICLI
SumoLogicWeb interfaceAPICLI

Consult your preferred provider's documentation for details on how to set up their service. Some details specific to particular providers are included in the web interface guides linked above.

Compatible integrations

Many third-party services not explicitly supported by Fastly can also be used via a generic transport, or a compatible third-party for which we have a dedicated connector.

Generating Logs in VCL

You have a two options to emit logs from your VCL services. Either generate the logs automatically or explicitly generate them through VCL code.

Generating logs automatically in VCL

The most common way to emit logs from VCL services is to use automatic log generation, where Fastly adds logging code to your service automatically, including any specific properties you specify.

When creating a log endpoint for a VCL service, you can optionally provide a log string as the format property, in a Fastly-specific log format, and we will generate and include a log statement in your VCL automatically. This allows you to create operational logging in one single step - creating both a log instruction that generates log events and a log destination to which those events can be sent - so that, for every request, log output will be emitted to the log endpoint.

The generated log statement will normally be placed in the vcl_log subroutine, but this can be changed using the placement property available on all log endpoint types. Setting placement to "none" will make the log endpoint available but will not generate any log instructions. In that case, you will need to write a log statement in your VCL manually.

The format property of log endpoints accepts a proprietary syntax based on the Apache log format which is converted into edge code for VCL services, and inserted into the VCL subroutine identified by the placement property.

IMPORTANT: This section does not apply to Compute services, where Fastly cannot manipulate your code. Each Compute language offers an API within the Fastly SDK that allows data to be written to a log endpoint from your code. This must be done separately to the creation of the log endpoint.

The value of the format property is the literal string to log, with the following placeholders replaced with the appropriate dynamic value:

PlaceholderSource VCL variableDescription
%%A literal % character.
%areq.http.Fastly-Client-IPThe client IP address of the request.
%Aserver.ipThe local IP address.
%Bresp.body_bytes_writtenThe size of response in bytes, excluding HTTP headers.
%bresp.body_bytes_writtenThe size of response in bytes, excluding HTTP headers. In Common Log Format (CLF), that means a "-" rather than a 0 when no bytes are sent.
%{foo}Creq.http.Cookie:fooThe contents of cookie Foobar in the request sent to the server.
%Dtime.elapsed.usecThe time taken to serve the request, in microseconds.
%freq.url.pathThe URL path, e.g. /images/cat.jpg
%hreq.http.Fastly-Client-IPThe client IP address of the request.
%Hreq.protoThe request protocol.
%{foo}ireq.http.fooThe contents of the specified header in the request sent to the server.
%Ireq.bytes_readBytes received, including request headers and body.
%mreq.methodThe request method.
%{foo}oresp.http.fooThe contents of the specified header in the response.
%Oresp.bytes_writtenBytes sent, including headers. Will never be zero.
%pserver.portThe canonical port of the server serving the request. Always returns 80.
%{format}pserver.portThe canonical port of the server serving the request. Valid formats are "canonical", "local", or "remote". Always returns 80 for "canonical" and "local", and always returns "-" for "remote".
%qreq.urlThe query string (prepended with a ? if a query string exists, otherwise an empty string).
%rreq.request, req.url, req.protoThe first line of the request (unquoted).
%sresp.statusThe HTTP status code on the response. For requests that restart, this is the status of the original request. Use %>s for the final status.
%ttime.startThe time the request was received, in Standard English format (e.g., [01/Jan/1970:00:00:00 -0700]). The last number indicates the time zone offset from UTC.
%{format}ttime.start.msecThe time, in the form given by format, which should be in strftime(3) format (potentially localized). If the format starts with begin: (the default) the time is taken at the beginning of the request processing. If it starts with end: it is the time when the log entry gets written, close to the end of the request processing. In addition to the formats supported by strftime(3), the following format tokens are supported: sec (number of seconds since the Epoch), msec (number of milliseconds since the Epoch), usec (number of microseconds since the Epoch), msec_frac (millisecond fraction), and usec_frac (microsecond fraction).
%Ttime.elapsed.secThe time taken to serve the request, in seconds.
%Ureq.url.pathThe URL path requested, not including any query string. Same as %f.
%vreq.http.hostThe value of the Host header on the client request.
%Vreq.http.hostSame as %v.
%{vcl}V{vcl}The literal VCL to include without quoting. This can be used to write VCL variables to your logs (e.g., %{client.geo.country_code}V or %{tls.client.cipher}V). This %-directive is a Fastly extension and is not found in Apache. See useful variables to log for more examples.

Additionally, the following placeholders are recognized but not relevant to Fastly services, and always return constants intended to allow the output to remain parsable by tools intended to work with Apache formats:

  • Always returns "-": %{foo}e, %l, %{foo}n, %P, %{foo}P, %R, %u
  • Always returns 0: %k
  • Always returns +: %X

When using format to generate log instructions, format_version should be set to 2. A long-deprecated, version 1 log format continues to be supported for compatibility but is not recommended for new configuration.

Examples

To replicate Apache's common log format, set format to:

%h - - %t "%r" %>s %b

This will produce a log line such as:

123.1.12.123 - - [10/Oct/2000:13:55:36 -0700] "GET /apache_pb.gif HTTP/1.0" 200 2326

However, in modern usage, structured formats such as JSON are increasingly replacing the venerable space-separated log formats popularized by Apache. Where supported by the system you are sending logs to, we recommend constructing JSON or another format that can be parsed by a standard library. This also permits logging of a much greater variety of valuable data points.

HINT: If you are familiar with VCL, using only the raw-VCL placeholder %{.....}V instead of the Apache tokens may make it easier to understand and maintain your log configurations.

Here is an example of a format that produces JSON-formatted log lines:

{
"timestamp": "%{strftime(\{"%Y-%m-%dT%H:%M:%S%z"\}, time.start)}V",
"client_ip": "%{req.http.Fastly-Client-IP}V",
"geo_country": "%{client.geo.country_name}V",
"url": "%{json.escape(req.url)}V",
"request_referer": "%{json.escape(req.http.referer)}V",
"request_user_agent": "%{json.escape(req.http.User-Agent)}V",
"fastly_is_edge": %{if(fastly.ff.visits_this_service == 0, "true", "false")}V,
"response_state": "%{json.escape(fastly_info.state)}V",
"response_status": %{resp.status}V,
"response_reason": %{if(resp.response, "%22"+json.escape(resp.response)+"%22", "null")}V,
"response_body_size": %{resp.body_bytes_written}V,
"request_method": "%{json.escape(req.method)}V",
"request_protocol": "%{json.escape(req.proto)}V",
"fastly_server": "%{json.escape(server.identity)}V",
"host": "%{if(req.http.Fastly-Orig-Host, req.http.Fastly-Orig-Host, req.http.Host)}V"
}

Fastly does not attempt to parse your log output, so take care to ensure that if you are logging in a standard structured format, it is correctly escaped and valid. If not, your log destination may not understand it.

Explicit logging with VCL code

VCL services also support logging to a named log endpoint with an explicit line of code. In VCL services, data is logged with the log statement (unless you're using automatic log generation).

When a log endpoint is configured with automatic log generation, you can still log to that endpoint manually as well.

VCL log statements are prefixed with syslog, the service ID, and the name of the destination endpoint, followed by a :: delimiter. This prefix, up to and including the delimiter, is removed from the log message before it is dispatched to the log destination.

// |----------------- Header -------------------| |--------------- Content ---------------|
log "syslog " + req.service_id " + my_log_endpoint :: " + req.http.Fastly-Client-IP + " " + req.url;

Log messages must be a single line. Newline characters or null bytes in log messages will terminate the message.

Storing intermediate state in VCL

In VCL services, log statements are typically placed in the vcl_log subroutine, which runs after delivery of the response to the client has completed (see VCL life cycle). This is also where log statements generated by automatic log generation are inserted. Since each VCL subroutine is a separate scope with a different set of defined variables, some data values which are defined in other parts of the lifecycle are not available in vcl_log. It's also possible that other VCL code in your service might have modified the values of some variables, but you wish to log the original state of the request as received by Fastly.

The following code example demonstrates how HTTP headers can be used to sample variables throughout the VCL flow, providing additional insights into backend requests and responses, the shielding process, and more granular timing data:

Useful variables to log in VCL

Fastly exposes a wealth of information for every request, and much of that data is useful to log. The full set of data available in VCL services is described in the VCL reference. Compute services have language-specific SDKs providing access to much of the same data and a set of environment variables which are common across all our supported languages.

Here we have collected a list of data available to Fastly services which are typically the most popular data to include in log messages. Those marked with ⭐ appear in the largest number of customer configurations.

WARNING: Be sure to take into account security, privacy, and compliance requirements when making decisions about the data you intend to include in logs. Some jurisdictions may have data protection regulations.

To use one of these variables in a VCL log message, either write it directly using the log statement or, if using log generation, include it in the format property of a log endpoint, in the form %{VARNAME}V. Where data is indicated as available in the Compute platform, the access syntax will depend on the language SDK in use. Consult the SDK documentation for your preferred language.

Custom log formatter for VCL

HINT: Select any variables in the table below to generate a custom log format in format.
DescriptionVCLCompute?
req.http.fastly-client-ipYes
client.as.numberNo
client.as.nameNo
client.geo.country_codeYes
client.geo.conn_typeYes
client.geo.latitudeYes
client.geo.longitudeYes
tls.client.protocolNo
tls.client.servernameNo
tls.client.cipherNo
tls.client.ciphers_shaNo
tls.client.tlsexts_shaNo
client.socket.congestion_algorithmNo
client.socket.cwndNo
client.socket.tcpi_rcv_spaceNo
client.requestsNo
client.socket.tcpi_rttNo
client.socket.tcpi_rttvarNo
client.socket.tcpi_rcv_rttNo
client.socket.tcpi_rcv_mssNo
req.is_ipv6No
fastly_info.is_h2No
fastly_info.h2.is_pushNo
fastly_info.h2.stream_idNo
req.vclNo
req.vcl.versionNo
server.datacenterYes
server.hostnameYes
fastly.ff.visits_this_serviceNo
req.header_bytes_readNo
req.body_bytes_readNo
req.url.pathYes
req.http.refererYes
req.http.user-agentYes
req.http.acceptYes
req.http.accept-languageYes
req.http.accept-charsetYes
req.http.if-modified-sinceYes
req.http.if-none-matchYes
fastly-info.stateNo
obj.ageNo
obj.hitsNo
obj.ttlNo
resp.statusNo
resp.responseNo
resp.http.cache-controlNo
resp.completedYes
resp.http.content-typeYes
resp.http.expiresYes
resp.http.last-modifiedYes
resp.http.etagYes
resp.header_bytes_writtenNo
resp.body_bytes_writtenNo
resp.bytes_writtenNo
resp.completedNo
client.socket.tcpi_delivery_rateNo
client.socket.plossNo
client.socket.tcpi_delta_retransNo
strftime({"%Y-%m-%dT%H:%M:%S%z"}, time.start)No
time.start.secNo
strftime({"%Y-%m-%dT%H:%M:%S%z"}, time.end)No
client.socket.tcpi_last_data_sentNo
time.elapsed.usecNo

Generating logs in Compute

To explicitly generate logs in Compute, each language SDK offers an equivalent API to log output to a named log endpoint.

  1. Rust
  2. JavaScript
  3. Go

Logging STDOUT and STDERR in Compute

For Compute services, it's also possible to emit data to STDOUT and STDERR (for example, using println! in Rust). As a special case, any log endpoint called stdout or stderr will capture the log output on these interfaces. STDOUT output can also be streamed to a terminal using fastly log-tail. Learn more about tailing logs for Compute services.

Troubleshooting

If your log events are not flowing to your destination log endpoint correctly, the following may help to diagnose the problem:

  • Log endpoint names are case sensitive. Any log events that target a non-existent endpoint will be silently dropped, so check that you have spelled the log endpoint name correctly and that the case matches the name specified when the log endpoint was created.
  • Many log providers apply constraints to the serialization format, length or content of log messages that they accept. Check your provider's documentation to ensure that the format you are sending is correct.

To see what log output your application is generating, swap out your log endpoint for an HTTP log endpoint with the address set to the URL shown below, to deliver your log messages to the viewer below:

Limitations and constraints

There is a limit on the maximum length of each log message which varies by platform. In VCL services it is 16KB; in the Compute platform, it is 64KB.