Fastly-generated 503 errors in VCL
If a fatal error occurs in your service while processing your VCL logic, Fastly will generate an error response with a 503 HTTP status, instead of using a response from an upstream server or cache, and unless intercepted by code running in vcl_error, this generated error may be served to the end user.
Error states can be triggered at any point in the VCL request lifecycle and will immediately terminate processing of the current subroutine and invoke vcl_error with a synthetic obj variable that contains information about the error. The value of obj.status is always 503 and obj.response is set to a descriptive name for the error. You can therefore intercept Fastly-generated errors by writing custom VCL in the vcl_error subroutine:
if (obj.status == 503 && obj.response == "backend read error") { ... }Possible errors
The following table itemizes the possible errors that may be generated by Fastly during VCL execution, their causes and possible troubleshooting actions.
HINT: obj.response is case sensitive and capitalization is somewhat inconsistent between different errors. Starting from HTTP/2 the HTTP response "reason" text (obj.response) is no longer sent along with the response, so if a Fastly generated error response is delivered to the end user unmodified, it is typically not possible to know what the specific error was.
obj.response | Description |
|---|---|
All backends failed or unhealthy | A director typically used for load balancing is failing because either: a) all the backends are unhealthy; or b) no backend could provide a valid response (if the director uses a chash policy, the error is suffixed with the policy name, i.e. "All backends failed or unhealthy (chash)"). Try reviewing backend configuration, checking your origin server is accessible from Fastly, and ensuring that the server is working as expected. |
Backend is unhealthy | Your configured health check is reporting that the selected backend is down. Fastly will not send traffic to backends that are failing health checks. To provide alternative options when backends are down, consider using load balancing or redundancy and failover. If the origin server is in fact healthy, try checking health check settings. |
backend read error | A timeout occurred when Fastly attempted to fetch content from your origin server over an established connection. It can also be due to a variety of transient network issues that might cause a read to fail (e.g., router failovers, packet loss) or an origin overload. Consider benchmarking backend response times to assess the origin server's expected performance, and if necessary increasing backend timeouts. |
backend write error | A timeout occurred when Fastly attempted to write a request body to your origin server over an established connection (typically on POST requests). Typically troubleshooting follows the same steps as for backend read error. |
Backend unavailable, connection timeout | Same as connection timed out. |
Backend unavailable, connection failed | Same as connection timed out. |
Backend unavailable, socket(2) failed | An internal error on a Fastly cache node prevented a socket being allocated for the fetch. |
Backend.max_conn reached | Fastly attempted to make a request to a backend that has reached its maximum number of connections (by default, 200 per Fastly cache node). Consider increasing the concurrent connection limit, or check whether the origin server is serving responses too slowly. |
between bytes timeout | No data was received from an origin server during a response for more than the allowed duration configured on the backend's between_bytes_timeout property. If that behavior is expected, consider increasing backend timeouts. |
certificate has expired | A certificate installed at the origin has expired. To resolve this, renew your certificate or download a new one. |
Certificate verification failed | Rare edge case triggered by a narrow range of expired certificate or domain mismatch errors. |
client read error | A problem occurred between the end user and Fastly. This is often because the user aborts the request by navigating away from the page. |
Connection refused | Fastly attempted to make a connection to your origin and the server refused the connection. It typically appears when the wrong port is specified for the backend. Try reviewing backend configuration or checking your origin server is accessible from Fastly |
connection timed out | A timeout occurred while waiting to establish a TCP connection from Fastly to your origin, or while waiting for your origin to respond to a request. Consider increasing backend timeouts or enabling shielding using a location close to your origin. |
first byte timeout | Fastly established a connection to your origin, but the origin did not start sending the response within the time configured for the backend's first byte timeout, which defaults to 15 seconds. Consider increasing backend timeouts. |
Generic SSL handshake error | Same as SSL handshake error |
hostname doesn't match against certificate | The TLS certificate being served by your origin does not match the certificate hostname specified in your Fastly service's backend settings. The hostname specified in the ssl_cert_hostname property must match either the Common Name (CN) or a Subject Alternate Names (SANs) listed on the certificate. To resolve this error, reconfigure the backend within Fastly. |
Illegal Vary header from backend | A backend returned a malformed Vary header with its response. A well-formed Vary header specifies that the response can only be used with certain qualifying future requests. If the Vary header is malformed Fastly cannot use the response. |
Maximum threads for service reached | This error occurs when Fastly detects that a service has exceeded a safety limit on the number of concurrent requests. Typically this indicates that a service is experiencing an unusually high load, that an origin is slow, or that features like request collapsing are being intentionally avoided. |
Missing ssl_hostname/ssl_cert_hostname to check against | Caused by configuration errors in the TLS certificate, but rarely seen. |
No healthy backends | A director (typically used for load balancing) is unable to make a backend request because there are no healthy backends available in its group. |
No healthy IP available for the backend | Health checks or DNS resolution on the backend are failing. Check that the backend configuration on your Fastly service has the correct hostname or IP address, or review health check settings. |
No stale object available | This error occurs when you configure Fastly to serve stale objects in the event of a backend failure (or your VCL code invokes return (deliver_stale)) but there is no stale object or it has expired. Consider reviewing any stale-serving-related VCL code, checking health check settings, or bringing a failing origin server back online. |
Quorum weight not reached | A director (typically used for load balancing) can't serve traffic based on its configuration because it does not have enough available backends in its group. Consider checking for and resolving any issues with your origin servers, or review the quorum setting. |
Response object too large | The object being fetched from origin is too big for your Fastly service. You can use the Segmented Caching feature to break such large objects down into smaller chunks. |
Server did not provide certificate | Origin TLS certificate is not configured correctly. In most cases this will trigger SSL handshake error, and not this one. |
SSL handshake error | TLS negotiation between Fastly and your origin failed. To fix this error, review and correct your host's TLS configurations. |
unable to get local issuer certificate | A certificate in the certificate chain is missing or invalid. To better determine which of these issues is the cause of the error, consider running an SSL test on your origin server hostname. You may then need to update certificate configuration. |
IMPORTANT: This table describes errors generated during VCL processing, but Fastly may also generate error responses from our routing systems. Learn more.
Since VCL errors trigger vcl_error and run your VCL code, it is possible to intercept these and serve a custom error page instead of allowing the synthetic error to be seen by end users.
Memory overflows
During VCL execution, exceeding some limits will cause the VCL program to terminate immediately, without invoking vcl_error. In these cases a 503 error is always generated and emitted to the client, and cannot be modified by your service configuration. The most common types of overflows are:
- Workspace exhaustion: VCL programs allocate memory greedily and free it when the request ends. Use
workspace.bytes_freeto understand how much workspace is available. - Header overflows: VCL programs may only define around 90 HTTP headers in total. The exact number available to your program depends on the number of headers used by Fastly during your request.
Troubleshooting options
This is a description of some of the most common and effective actions that can help resolve elevated rates of synthetic errors in your VCL service.
WARNING: Purging cache in response to unexplained elevated error rates may make things worse. Purges typically increase traffic to origin, which may exacerbate a problem rather than fixing it. Try to determine the cause of errors and be certain purging is the right solution.
Benchmark origin server response times
In situations where Fastly is emitting errors relating to backend connections, it's often useful to check backend response times independently. Many outside factors cause backends response times to vary. Run the following command to estimate response time for benchmarking purposes:
$ curl -s -w "%{time_total}\n" -o /dev/null http://example.com/path/to/fileThis can help you understand the typical performance characteristics of an origin server and decide on appropriate timeout settings - or reveal issues with an origin server's performance that you may need to debug outside of Fastly.
Increase origin timeouts
Timeouts apply to multiple stages of the request when Fastly makes requests to a backend, and these can be configured as properties of the backend in the API, UI or CLI. In the case of errors like backend read error, backend write error, connection timed out, or first byte timeout, first check the performance of the origin server. Our defaults are generous, and hitting these timeouts generally indicates a very poorly performing origin server. However, if you need to, adjust the timeouts in the backend configuration, notably the first_byte_timeout property (also available in the web interface).
| Timeout type | Description | VCL variable |
|---|---|---|
| Connect | Maximum duration Fastly will wait for a connection to this backend to be established. If exceeded, the connection is aborted and a synthetic 503 response will be presented instead. | bereq.connect_timeout |
| First byte | Maximum duration Fastly will wait for the server response to begin after a TCP connection is established and the request has been sent. If exceeded, the connection is aborted and a synthetic 503 response will be presented instead. | bereq.first_byte_timeout |
| Between bytes | Maximum duration that Fastly will wait while receiving no data on a download from a backend. If exceeded, the response received so far will be considered complete and the fetch will end. | bereq.between_bytes_timeout |
Except for requests flagged for PASS, Fastly enforces a 60 second timeout on requests between one Fastly node and another, which cannot be changed. Therefore, if your request transits more than one Fastly server (usually as a result of shielding or clustering), the maximum connection and first byte timeouts are effectively 60 seconds on any requests that may produce a cacheable response.
If you want a timeout higher than 60s and do not need the response to be cached by Fastly, flag the request as PASS in vcl_recv:
return(pass);If you do want to use the cache, but must have a maximum timeout higher than 60s, it's possible to disable clustering, but this will result in a significant degradation in cache performance because the response will be cached separately on each Fastly server.
If there are any external interfaces in front of the origin server, such as a load balancer or firewall, these may also apply their own timeouts.
Enable shielding
Enabling shielding may help resolve some backend related connection problems:
- Shortening the distance needed to establish a connection, which may help avoid connection timeouts.
- Reducing TCP handshakes resulting from using multiple POPs. This allows the origin to avoid slowdowns and to process only requests on a few connections from the shield POP, and may resolve issues resulting from holding open too many connections.
- Improving cache hit ratio and request collapsing efficiency, which may significantly reduce overall traffic to origin.
Conversely, some issues can be created by enabling shielding, especially if you operate geographically distributed origin servers, so this option should be targeted to appropriate situations.
Review backend configuration
If your Fastly service is failing to connect to your origin server, some properties of the backend configuration are worth checking:
- Port: Is your origin server listening on the port Fastly is configured to connect to?
- Override host: Have you set the origin server's hostname into the "override host" property of the backend?
Ensure Fastly can reach the origin server
Fastly may not be able to connect to your origin server, even if it works from your local machine. Check that you do not have firewall rules that would prevent connections from Fastly's public IP ranges.
Check health check settings
Health checks will mark a backend as 'sick' if the conditions of the health check are not satisfied, which may happen even if the origin server appears to you to be working.
- Check that the path configured in the health check exists on your origin server
- Check that the health check traffic is not being blocked by origin firewall rules
- Check that the health check conditions are reasonable. For example, if the health check's
thresholdis larger than itswindow, the check will never succeed.
Increase maximum concurrent origin connections
Fastly enforces a limit on the number of concurrent connections from a single Fastly edge node to a single origin host. In general, if you encounter this limit, first consider whether you could take action to reduce the number of connections required:
- Consider making your responses more cacheable, allowing Fastly to respond to more requests without forwarding them to an origin server.
- Improve (i.e., reduce) origin server response times, so that backend connections are used more efficiently.
- Enable shielding, to reduce the overall number of requests that need to be forwarded to origin (thanks to improved cache hit ratio and more efficient request collapsing)
These actions should especially be considered if you encounter problems with less than 10,000 non-hit requests per second.
If you have determined that your origin is not the issue, increase the maximum connections limit to your origin or reach out to Fastly support for further help with this issue.
Update certificate configuration
For missing or invalid certificates, download and replace the missing or incorrect certificate. If both the intermediate and root certificates are correct, insert a valid Server Name Indication (SNI) hostname in the origin TLS options of your Fastly service.
