How the Vary HTTP Header Can Be Bad

Vary is а powerful HTTP header that plays a significant role in how your website cache is working. When this header is set correctly, it ensures that your site visitors see the right content, regardless of the caching applied. However, if this header is set up incorrectly, it can totally overrule even the benefits of the best caching system and cause resource overusage. In this blog post, I want to shed some light on how the Vary header affects your SiteGround hosted site and specifically our caching system, and to show you what’s the recommended use of Vary header for your site.

How Does the Vary Header Work?

The role of the Vary header is to indicate in which cases a different version of your site should be served. If your Vary header has one of the following values: User-Agent, Cookie, Referrer or even * (wildcard) it can significantly reduce the effect of our caching system. This happens because these values indicate that a different version of your pages should be served based on the browser type used (user-agent), the presence of unique cookie or referral URL or, in the worse scenario when * is used, this means that a fresh content will be served for every single visit, which will completely disable our cache.

Vary: User-Agent

Let’s take a look how the most commonly used value, User-Agent, works. It tells our Dynamic caching: “Hey, you should store different caches for different OS and browsers”. This is done in order to avoid serving a cached desktop version to a mobile visitor for example. Note, however, that nowadays most of the sites don’t actually serve different HTML for their mobile versions. It’s the responsive CSS that does all the heavy lifting and shows your site in a different way on mobile and on desktop. So, unless you’re using a plugin like WP Touch for example, or you know for sure you have a difference in the HTML output of your site, based on visitors browsers, your site should not be sending the Vary: User-Agent header.

Without the Vary: User-Agent set, our caching system will cache your site on the first load and then serve the cached version to the following requests unless you update your content or purge the cache manually. With the User-Agent enabled, the system will keep different copies for each combination of OS and browser version visiting your site. This means that you will have a dynamic request for the first person loading your site on a Desktop Safari, one for the first person loading a mobile Chrome, desktop Chrome and so on.

Let’s say you clear your cache roughly every 100 hits. Without Vary: User-Agent sent by your site, your site will have only 1 dynamic request in 100. On the other hand, with the header enabled, depending on your visitors profile, you will have 5-30 dynamic requests for the same 100 visits. So the very same site will use 5-30 times the resources it would need without the header being set.

Bad, Bad Bots

We hate malware traffic and bad bots. It’s been an ongoing effort to fight them in every possible way. I am mentioning bots here, because if you get spammed by them, the Vary: User-Agent header may determine how successful their attack would be and how being crawled by bots will affect your site performance.

You can think of a bot as a browser that does automated actions on your site. It may collect data for their own index, search for vulnerabilities, try to submit forms, etc. Bot’s User-Agent is simply a piece of text added by its creator. Reputable crawling machines like the Google bot have theirs set properly, while others may try to mimic the Bing bot, or the Google bot or even randomly generate a user agent.

If such bot starts crawling your site and you have the Vary: User-Agent header, each request it makes to your site would be a dynamic one and will eat up your resources. On the other hand, if the header is not used, you would serve cached requests to the bots directly from our Dynamic caching.

How to See Your Response Headers

First of all, you should check the headers your site is returning to the visitors. Usually, I use the Network tab of FireBug on my browser but there is a handy online checker (http://www.webconfs.com/http-header-check.php) that you can use too. There are multiple values of the Vary header and user-agent is just one of them. If you do not have a Vary header or it includes only Accept-Encoding value (that helps gZIP compression to work properly) you should not worry. However, if you see User-Agent or any of the other values (Cookie, Referrer or *) we recommend that these are removed from your header.

How to Configure the Vary Header

Generally, the best approach would be to understand which part of your site is generating it and reconfigure it. However, this might require some troubleshooting skills and may not be a very easy task. So if you’re on WordPress, you can check your headers and then replace the response with only what you want by adding the following lines to your functions.php file:

function replace_wp_headers($headers) {
$headers['Vary'] = 'Accept-Encoding';
return $headers;
}
add_filter('wp_headers', 'replace_wp_headers');

Alternatively, you can try using an .htaccess rules to unset the unnecessary Vary headers:

<ifModule mod_headers.c>
Header unset Vary
Header set Vary “Accept-Encoding”
</ifModule>

Doing this will remove all headers passed from your site and leave only the one specified in your code – Accept-Encoding. Of course, you can add more rules but always make sure you really need them and there’s no other way to achieve the result you’re after!