HTTP Caching Explained

Caching is enabled by the origin server sending certain HTTP headers along with each response. It is important for website developers to understand how to properly set these headers as it directly impacts user satisfaction by enhancing the speed and responsiveness of the website. Moreover, it reduces server load and bandwidth costs, leading to a more scalable and cost-efficient operation. It is an essential aspect of website performance optimization. This article will explain each header and tell you how to set them correctly for your website.

HTTP Caching History Lesson

When a web server sends a response back to a web browser, it also sends along some hidden information in the form of HTTP headers. These headers give instructions to the browser on how it may cache that content. For the website's logo, it may tell the browser to store the image for 1 month and always use the cached version. For the website's "current news" page it will probably tell the browser to never store the page and always fetch a fresh copy. Generally the server will want static files to be cacheable for a long time, and dynamic pages to be cached for a short time, or not at all.

When HTTP/1.0 was standardized in the late 1990s, it introduced the "Date", "Last-Modified" and "Expires" headers. Caches could use this information to determine if a cached response was still fresh or if it had become stale and needed to be refetched.

By the early 2000s the upgraded HTTP/1.1 standard took over. It added the "Cache-Control" and "ETag" headers. These new headers allowed for more sophisticated and flexible caching strategies and enabled a method for efficient revalidation of responses to extend a cached response's lifetime.

The HTTP standard has been further upgraded to HTTP/2 and HTTP/3 but these have not changed the behavior of the caching headers.

The browser's cache will be used when the user navigates to a webpage that contains resources that have been download before. There are two scenarios when the caching headers do not directly apply. If the user presses the "reload" button this generally instructs the browser to bypass the cache and fetch fresh copies from the server. If the user presses the "back" or "forward" button to navigate through their browser history, this will generally use cached resources even if they are stale, without fetching a fresh copy.

First, lets define some terms and then we'll go through each header and explain what each one does.

Definitions

Origin Server: The origin server is the service that listens and waits for incoming requests from visitors and responds with content. The origin server could be a physical computer sitting in a data center, or it could be a virtual server hosted on a cloud platform, or it could even be a "serverless" process running in the cloud. Regardless of the underlying technology, the origin server is the original source for a webpage, image or other resource.

Request: A request is the message that is sent from the web browser to the origin server, asking for some piece of content. Typically a webpage, JavaScript file, CSS stylesheet or image.

Response: A response is the data sent from the origin server back to the web browser with the content that the browser requested.

HTTP Cache: Any mechanism that stores a request/response pair and quickly reuses and delivers the response for a recognized request. This shortcut prevents the response from going all the way to the origin server, thus saving time and resources. An HTTP cache is built directly into every browser, which means that requests can often be fulfilled without causing any network traffic at all. In addition to the browser's built-in cache, there could be several additional caches located in the path between the browser and the origin server. Any of these caches could respond to a request and bypass the origin server.

Private Cache: An HTTP Cache that is tied to a single user. Private caches can contain personal information specific to that one user. For example, if you are signed into a website, the webpage that displays your account information may be stored in your web browser's private cache.

Public or Shared Cache: An HTTP Cache that contains non-personalized responses that can be shared between different users. For example, the images associated with an article that you are reading are the same images shown to everyone, so the shared cache can store these and serve them to any user. Public caches cannot generally cache content encrypted with HTTPS connections because they cannot see the headers and understand the response.

Managed Cache: This is an HTTP Cache that is set up by the owners of the origin server. Its purpose is to reduce the load on the origin server by intelligently routing requests to managed services that can respond faster. An example of this would be a Content Delivery Network (CDN) that is designed to return static content (images, JavaScript files, etc) very quickly while passing requests for personalized webpages straight to the origin server. If set up properly as a proxy, a managed cache can understand and serve content encrypted via HTTPS connections.

Age: This is a number that indicates how long ago a particular response was generated. When a response gets too old, then it becomes "stale" and is no longer valid. When a cache sees a request, it should generally not respond with a stale response. It should instead forward the request to the origin server to get a "fresh" response which it can then continue to store.

Date Header

The Date Header is set by the origin server and indicates the date and time when the response was generated. If the request is fulfilled by the origin server then the Date header should be right now. If the request was fulfilled by an HTTP cache, then the Date header could indicate a date in the past, potential far in the past if the resource has been cached for a long time. The difference between the Date header and the current date is the "Age" of the document. A public or managed cache will usually set an "Age" header with this value so the client knows how long the object has been in the cache.

Date: Sun, 28 Apr 2024 05:54:15 GMT

The date should be formatted as a "RFC-7231" timestamp as in the above example.

Last-Modified Header

The Last-Modified Header indicates when the resource was last updated. It may be older (potentially much older) than the Date header.

Last-Modified: Thu, 18 Apr 2024 05:54:15 GMT

If the server only sends the Date and Last-Modified headers, then the browser may resort to using algorithms to guess at the cacheability of a response. For example, if a resource has not been updated for a long time, then the web browser may assume that it will remain unmodified in the near future and may cache it. This is called Heuristic caching, and it is how caches handle things when no other signals are provided by the origin server. It would be better if the origin server explicitly set the Expires or Cache-Control headers as well.

Expires Header

The Expires Header is set by the origin server. It indicates a specific data and time in the future when the response should be considered "stale". When this date arrives, an HTTP cache should forward any new requests to the origin server for a fresh copy. The origin server can intentionally set the Expires header to a date in the past if it wants to indicate that the response should never be cached.

Expires: Wed, 08 May 2024 05:54:15 GMT

This header was the primary way that HTTP caches functioned with HTTP version 1.0. However, now that practically every client understands HTTP version 1.1, the Cache-Control header is the preferred way to manage caching.

Cache-Control Header

The Cache-Control Header is the modern way to influence how content is cached. The transition from the Expires header to the Cache-Control header reflects an evolution in web technologies, offering more granular control over caching policies. It works for all HTTP Version 1.1 clients, which includes all modern web browsers. If the Cache-Control header is included, it is not necessary to include an Expires header as well. The Cache-Control header is a series of directives separated by commas. Some directives stand alone and some have an equals sign after them followed by a number.

Cache-Control: no-cache, private

In the above example, we are telling caches that the data is private and should not be served from a cache without revalidation with the origin server each time.

Cache-Control: max-age=31536000, public

In this example, we are allowing caches to store the response for 1 year and letting the cache know that the response can be shared with other users.

Here is a list of every valid directives:

• max-age=N: This tells a cache that it can store the content for N seconds. For example, "max-age=86400" would allow the cache to serve the content for 1 day. The time that the document was generated (ie. The Date header) is used as the starting point to measure the age of the resource.
• s-maxage=N: This is the same as "max-age" but it only applies to shared caches and will be ignored by private caches.
• no-store: This directive prevents all caches from storing the response for any reason. Note that this will break the back/forward button in browsers, since the browser will not be able to serve the cached content. This directive is not recommended for this reason.
• no-cache: Contrary to it's name, this directive allows caches to store the response, but it instructs them to revalidate it on every request to ensure that the response is always fresh. If the client is offline and unable to contact the origin server then the cached response cannot be revalidated and it will not be served from the cache.
• no-transform: This tells caches that the content should not be transformed in any way. Sometimes CDNs will resize or recompress images to speed up content delivery. With this directive, this will not happen.
• must-revalidate: A cache is normally allowed to serve a stale resource if it cannot connect to the origin server to revalidate it. If the "must-revalidate" directive is used, then this will not be allowed and the stale resource will not be served from the cache.
• proxy-revalidate: This works the same as "must-revalidate" but only applies to shared caches.
• must-understand: This directive tells caches that it should only store the response if it understands how to do so based on the HTTP status code. For example, lets suppose that a request is made and the response contains a "502 Bad Gateway" status code. This response should not be cached because it indicates a likely temporary error. Some caches may not understand some status codes and may incorrectly cache the response. If this directive is used, the "no-store" directive should also be used as a fallback for caches that do not support the "must-understand" directive.
• private: This indicates to caches that the response contains personalized content and should not be stored in a shared cache. It can only be stored in a private cache, typically the user's web browser.
• public: This indicates to caches that the response does not contain any personalized content and can be stored in a shared cache and served to different users.
• immutable: This directive is poorly supported by browsers, but its purpose is to indicate that the resource will absolutely never change. The benefit to using this directive is that browsers that understand it will not refetch the resource when the user clicks the "reload" button.
• stale-while-revalidate=N: This indicates that a cache is allowed to reuse a stale response for N seconds while it revalidates it in the background.
• stale-if-error: This indicates that a cache is allowed to reuse a stale response for N seconds if it encounters an HTTP error status code while trying to revalidate. Most browsers do not support this directive.

If you don't add a Cache-Control header it could cause an unexpected result. Caches are allowed to store responses heuristically, so if you have any requirements on caching, you should always indicate them explicitly in the Cache-Control header.

ETag Header

The ETag Header contains a string that uniquely identifies the response. If the ETag changes then the response has changed. This string is often a hash of the resource's contents but it can be set to anything that the origin server will understand on repeated requests.

ETag: 6f47298ff3d0c974f624a30392e97cc7

This header is used during revalidation. When the cache tries to revalidate a stale response it will send the ETag for its cached response. The origin server will compare this to its copy. If it matches, then the resource has not changed and the origin server can respond with a "304 Not Modified" HTTP status code indicating that it can be reused. If the ETag has changed, then a new fresh response is generated and sent to the client.

Revalidation

When a cached response becomes "stale" and expires, the HTTP cache does not need to throw it away. It can instead try to revalidate the response from the origin server. This is a lightweight way for the cache to ask the server for a time extension. If the server decides that the original response is still fresh, it can quickly respond with a short "304 Not Modified" response. The cache can then update its previously stored response with a new expiration date. This saves bandwidth and server resources because the origin server does not have to transmit the original resource again.

There are two ways that a cache can revalidate a response. For both methods it will send the request to the origin server and set an HTTP Request Header to some value. The origin server will inspect the request header and either respond with fresh content or a "304 Not Modified" response.

Using Modification Dates

If-Modified-Since: Thu, 18 Apr 2024 05:54:15 GMT

Using this method, the client will set the "If-Modified-Since" header and include the value of the "Last-Modified" header from the previously cached response. The origin server will compare the date to the one it has for the same resource. If the dates are the same then the response has not changed and a "304 Not Modified" response can be sent and the cache can keep using its stored result. If the dates don't match, then the origin server will send a fresh response. It may be difficult for some server architectures to parse and track modification dates for resources. For these cases, ETag validation is used.

Using ETags

If-None-Match: 6f47298ff3d0c974f624a30392e97cc7

Using this method, the client will set the "If-None-Match" header and include the value of the "ETag" header from the previously cached response. The origin server will compare this to its copy of the content and respond with either a "304 Not Modified" response or with a fresh copy of the response.

Recommendations

Now that you have read all of that (or if you just want the TL;DR summary), here are our recommendations for setting the caching headers. First, set the Date, Last-Modified and ETag headers accurately. Then set the Cache-Control header to a value that is appropriate for each response.

Cache-Control: no-cache

For web pages that update frequently.

Cache-Control: no-cache, private

For frequently updated web pages that contain personalized content.

Cache-Control: no-store

For web pages that contain very sensitive content.

Cache-Control: max-age=31536000, public, immutable

For images and other static resources that will not change. If you need to change a static resource it is better to give it a new URL so that it will be an entirely new resource. Typically this is done by putting a version number or hash into the filename.

Cache-Control: max-age=3600, private

For API responses that update occasionally.

How To Set Cache Headers

Now that you know which headers to set and what to set them too, you may be wondering how to set them. This will differ depending on the particular server software that you are using. Some headers are generally set automatically and others must be set intentionally. Here are instructions for some common server technologies.

Using Apache

To set the Cache-Control header with Apache, first enable the "mod_headers" module. Then you can set it either in the Apache configuration file or in an .htaccess file within a particular directory. Here is an example that gives certain static resources a long cacheable time period.

<FilesMatch "\.(jpg|jpeg|png|svg|gif|ico|webp|css|js)$">
  Header set Cache-Control "max-age=5184000, public, immutable"
</FilesMatch>

Using Nginx

To set the Cache-Control header with Nginx, you can use the add_header directive within the server configuration file.

server {
  ...
  location ~* \.(jpg|jpeg|png|svg|gif|ico|webp|css|js)$ {
    add_header Cache-Control "max-age=5184000, public, immutable";
  }
  ...
}

Using PHP

You can set headers using PHP on a case by case basis by using the "header" function.

header("Cache-Control: private, no-cache");

Using a basic Node.js HTTP Server

Using Node.js you can inject the header by using the "setHeader" method of the response object.

const server = http.createServer((req, res) => {
  res.setHeader('Cache-Control', 'private, no-cache');
  ...
});

Using Python with Django

If you are using Django, your views can add the header directly to the response object.

def my_view(request):
  response = HttpResponse("Hello, Django!")
  response['Cache-Control'] = 'private, no-cache'
  return response