How to Fix "Varnish Crashing Due to Header Size Too Big": Step-by-Step Guide

A constantly up-and-running website is crucial for business success. Your business website's availability and performance can directly impact the satisfaction of your target demographic, as well as the revenue and reputation. However, despite the attempts, numerous challenges can take your website down. 

One such challenge we've observed is the clash between Varnish and too-long CSP headers. The misalignment between the Varnish HTTP header size limit and the increasing CSP header on the website due to changes in the security policy can cause unexpected website outages that can lead to lost opportunities and erode trust in your business.

In the following points, we will discuss the cause of the outage we've observed, the implemented solutions, and the broader importance of the proactive measures you must take to ensure continuous website functionalities.

The Problem: Varnish Header Size Too Big

Varnish Cache is a high-performing caching software that works as a web application accelerator, also known as a caching HTTP reverse proxy. You install it in front of any server that speaks HTTP, and it then intercepts requests to that web server and presents any content requested from the cache, significantly reducing the loading time. Varnish also reduces server load by caching frequently accessed content, offloading a significant portion of the traffic from the web servers, allowing your servers to handle more requests and reduce the strain on the website infrastructure.

Varnish, however, has a default limit for the http_resp_hdr_len parameter, which determines the maximum allowed size for response headers. This default Varnish HTTP header size limit was insufficient to handle large Content Security Policy Headers. The expanding CSP headers due to the addition of new policies clashed with the default header size limit on Varnish, resulting in a website outage.

The Step-by-step Resolution

When faced with such a challenge, swift and structured resolution is critical to getting the website up and running. Here's a complete rundown of the step-by-step resolution we implemented to resolve the website downtime issue due to Varnish cache software.

Step 1: Diagnosing the problem

As the initial step, our task was to identify the root cause of website downtime by analyzing the logs and system behavior. It became evident that Varnish crashing due to header size was caused by the default header size limit being insufficient to handle the expanded CSP headers of the website.

Step 2: Increasing the header size limit

Once we figured out the misalignment between Varnish's header size limitations and the too long CSP headers causing the site downtime, we moved to the second step of the resolution process. In this step, we updated the configuration to increase Varnish header size limit for the response headers by adding the following parameter:

p http_resp_hdr_len= \

By raising the limit, we ensured Varnish could process the larger CSP headers without blocking responses.

Step 3: Testing & validation

After updating the configuration to increase Varnish header size limit, our team tested the new implementation rigorously in a development environment to avoid any new challenges: 

• Simulation: We simulated real-world scenarios with large CSP headers. 
• Monitoring: Key metrics like response times and resource utilization were monitored to ensure the stability of the configuration updates.

Step 4: Implementation in production

After successfully testing the solution in the development environment, we applied the updated configuration to the production environment and restarted the Varnish service. This step was closely monitored for a smooth transition and to ensure the solution eliminated the header size challenge successfully.

Step 5: CSP Header review and optimization

Despite the updated configuration, which increased the CSP header size limitation to a certain point, there was still some potential for future issues. Recognizing that, we reviewed and optimized our CSP header implementation. During this step, we removed unnecessary and redundant entries and streamlined the headers, ensuring they remained efficient and within acceptable size limits.

The Bigger Picture: Proactive Measures to Ensure Always Functioning Website

While this incident was resolved quickly, it served as a reminder of how critical the role of a functional website is in today's digital landscape. Beyond addressing immediate issues, you must adopt some proactive measures to maintain the availability and improve the performance of their website.

• Ensure Scalability: Invest in infrastructure to handle your website's growth and evolving requirements.
• Implement Redundancies: Have failover systems in place to minimize downtime during outages.
• Monitor Continuously: Use monitoring tools to track performance and detect real-time anomalies.
• Regular Maintenance: Update and optimize systems periodically to address emerging challenges.

Conclusion: Turning Challenges into Opportunities 

In the fast-paced digital world, a business website outage isn't just an inconvenience but a direct threat to customer trust and business success. However, we strongly believe that within problems lie opportunities for improvements. The downtime caused by Varnish crashing due to header size was an unexpected challenge, but it presented us with a great opportunity to improve our systems. By increasing the header size limit, optimizing our CSP implementation, and refining our processes, we secured our website's future resilience and reliability.