Common HTTP Implementation Problems

This document is a set of good practices to improve implementations of HTTP and related standards as well as their use. It explains a few basic concepts, points out common mistakes and misbehaviors, and suggests "best practices".

This document does not incriminate any specific product. W3C does not track bugs or errors in implementations. That information is generally tracked by the vendors themselves, or third parties.

Status of this document

Publication status

This document is the first public version of a Note, published on January 28th, 2003, and made available for discussion only by the editor and authors as part of their work as W3C Team participants in the Quality Assurance Activity. Publication of this Note by W3C does not imply endorsement by W3C, including the W3C Team and Membership.

This document may be updated, replaced, or obsoleted by other documents at any time.

Comments

No formal commitment is made by W3C to invest additional resources in topics addressed by this Note. However, comments are welcome and the W3C Quality Assurance Team may publish an amended version should the amount and quality of the received comments prove it worthwhile or necessary.

Translation

Translation of this document is welcome. However, before starting a translation of this document, please be sure to read the information on translations, in our Copyright FAQ, and check the list of existing translations of this document (available at http://www.w3.org/QA/translations#chips). .

Other W3C Technical Reports and publications

A list of current W3C technical reports and publications, including Working Drafts and Notes, can be found at http://www.w3.org/TR/.

Introduction

HTTP and URIs are the basis of the World Wide Web, yet they are often misunderstood, and their implementations and uses are sometimes incomplete or incorrect.

This document tries to improve this situation by providing a set of good practices to improve implementations of HTTP and related standards (Web servers, server-side Web engines), as well as their use.

This document only deals with the server-side aspect of HTTP, people looking for HTTP implementation problems in Web user agents should have a look at the user-agent counterpart of this document : "Common User-Agent Problems" [CUAP].

Scope of this document

This document is a set of known problems and/or good practices for HTTP implementations and their use, aimed at:

Unless specifically mentioned, what is referred throughout this document as "HTTP" is RFC2616, a.k.a. HTTP/1.1 [RFC2616].

Organization of this document : Guidelines, checkpoints

This document is divided into 12 guidelines and associated checkpoints. Each guideline is a general good practice, whereas the associated checkpoints are practical applications of the guideline. Checkpoints are themselves divided in one or more provision(s).

Targets associated with the checkpoints

If a checkpoint is applicable to several or all of these targets, it will have several tags.The target of a checkpoint is the sum of the target of its provisions.

An example of guideline and checkpoint

Here is an example of a guideline, with an associated checkpoint. Note the way they are presented, the multiple tags for the multiple targets of the checkpoint, etc.:

Conformance to this document

This document has no conformance per se, but since it is about implementation and use of normative specifications (such as HTTP/1.1), one should consider following this set of guidelines as a good step toward conformance to these specifications.

This document uses RFC 2119 [RFC2119] keywords (capitalized MUST, MAY, SHOULD etc.) when referring to behaviors clearly defined by a normative specification. When not capitalized, these words should be interpreted as regular language and not as RFC2119 keywords.

Techniques related to this note

As specified in the abstract, This document does not incriminate any specific product. W3C does not generally track bugs or errors in implementations. However, we welcome implementors and advanced users of such technologies to contribute to this document by providing techniques related to this note's applicable guidelines and checkpoints for a specific implementation.

1.Understanding URIs

A common mistake, responsible for many HTTP implementations problems, is to think this is equivalent to a filename within a computer system. This is wrong. URIs have, conceptually, nothing to do with a file system. One should remember that at all times when dealing with the World Wide Web.

To understand properly what a URI is, one has to think of the World Wide Web as a giant warehouse with an enormous amount of merchandise stored in boxes.

In this warehouse, a URI is not "row 12, 42nd box". A URI is not "that big black box over there", nor the content of the box. The URI is, exactly "The toothbrush can be found at row 12, 42nd box".

A URI is, actually, a reference to a resource, with fixed and independent semantics. An interpretation of this definition is that the URI is some sort of serial number for one of the many merchandises in the warehouse. "Fixed semantics" means that we know that in a box referenced by this serial number, there will be a specific product (we'll use a toothbrush for our metaphor). Always. We know neither the color nor the shape of the toothbrush, but we are certain that whenever and however we dereference the URI (which means, whatever way one (whoever) chooses to learn which box is referenced by the URI , the resource will always be a toothbrush.

Note that the URI is not exactly a serial number, since a serial number does not have any specific semantic, and it can be a reference to multiple instances.

Also, if you upgrade from toothbrush to a newer version of the toothbrush ("toothbrush v2"), the serial number may change. However, its definition "our toothbrush" will not change. One may thus think of the URI as being the identification of a specific semantic, and the HTTP ETag ([RFC2616] section 14.19) being the real serial number.

The URI http://www.example.com/products/toothbrush is then a fixed reference to a specific semantic, rather than being a serial number.

Note also that the HTTP Etag can be shared by identical resources that have different URIs. For example, if http://mirror1.example.org/foo and http://mirror2.example.org/foo share the same ETags, you can then deduct that those are equivalent resources.

The fixed semantics of a URI is one of the most important, yet often overlooked, concepts about URIs.

Tim Berners-Lee, creator of the World Wide Web, has written in 1998 an article named "Cool URIs don't change" [COOLURIs] stressing out this point and explaining how to use URIs properly.

Thanks to our warehouse metaphor, it is obvious that URIs should not change: people looking for a resource will have a lot of trouble finding it if the actual references for the resource changes, hence making the original reference pointing to... nothing.

This is all the more important on the Web (more than in our warehouse example) because the Web is built upon hyperlinks, which themselves use URIs. When URIs are broken, following hyperlinks ( or "bookmarks", which are a form of hyperlinks ) does not lead to the expected resource. In other words, from a server point of view, this means that the resource would miss some traffic... Traffic being the final aim of any content provider (as selling toothbrushes is the final goal of the warehouse owner), behaviors resulting in a loss of traffic should therefore be avoided.

As Tim Berners-Lee points out, When you change a URI on your server, you can never completely tell who will have links to the old URI. They might have made links from regular Web pages. They might have bookmarked your page. They might have scrawled the URI in the margin of a letter to a friend.. In other words, as Jacob Nielsen's writes, Persistent URLs Attract Links, Link-rot equals lost business.

We have seen why one should avoid breaking URIs. The following guidelines focus on techniques and strategies to avoid breaking URIs, or to fix them.

Guideline 1: Choose URIs wisely

This section summarizes, paraphrases and extends the section called "So what should I do? Designing URIs" in Cool URIs don't change [COOLURIs].

1.1: Short URIs SS CM

Use short URIs as much as possible SS CM

In order to make URIs easy to type, write down, spell, or remember, they should be short enough.

This checkpoint is not easy to quantify. However, we can take into account the fact that e-mail will be used to send URIs, and e-mail clients (sender or receiver) are supposed to wrap at 70-80 characters : even though they are not supposed to wrap long URIs, some do. As a result 80 characters is a reasonable total length for URIs (including URI scheme, e.g "https://", and host name).

Please note, however, that this length limit is by no mean a technical limitation, but rather, a practical goal to pursue.

1.2: URI case policy SS CM

Choose a case policy CM

URIs are partly case sensitive which means that, for example http://www.example.com/foo and http://www.example.com/FOO are different URIs and may refer to different resources.

Again, in order for the URIs to be easy to spell and remember, their case should not only be good (see following provisions of this checkpoint) but also consistent. It is thus recommended to choose a case policy, and enforce its use.
Avoid URIs in Mixed case SS CM
A case policy should be chosen, and enforced. All policies are, however, not equally preferable. Mixed-case URIs should be avoided.

Example of a URI following a mixed-case policy:
http://example.com/QAfOo/baRRoX
As a case policy choose either "all lowercase" or "first letter uppercase". SS CM

We suggest that either "all lower-case" or "first-letter uppercase" policy be chosen. Among these two, "all lower-case" may be prefered for its simplicity.

Example, "all lower-case":
http://www.example.com/foo/bar-bar

Example, "first-letter uppercase":
http://www.example.com/Foo/Bar-bar

Guideline 2: Allow URI management

As we said in the beginning of this chapter, a URI is not a filename, and you do not need to tie your URI structure to the file system on the Web server. However, chances are the resources served by a Web server will be available on a specific file system, and thus there should be flexible ways to map one onto the other.

2.1: URI mapping SI

Provide mechanisms for File System to URI mapping SI

Content managers should be able to re-organize the file system without modifying the URI structure. Servers should therefore allow the content manager to map the documents to URIs.
Examples::
Here are a few technologies that may be used for this purpose:
- Aliases
- Symbolic links
- Table or database of mappings
- etc.

2.2: Standard redirects SI CM

Allow the use of standard redirects SI

Content manager should be able to change easily the configuration of the server to use the various HTTP/1.1 redirection schemes (section 10.3 of the HTTP/1.1 specification [RFC2616]) :
- 301 Moved Permanently ([RFC2616] section 10.3.2)
- 302 Found (undefined redirect scheme, [RFC2616] Section 10.3.3)
- 307 Temporary Redirect ([RFC2616] Section 10.3.8)
The content manager should be allowed to use these, either by modifying directly the server configuration or by another indirect way of doing it (local configuration modification file, creation of local "redirect"resources, etc.)

Note that even though the current practice is to use the 302 Found status code for temporary redirects, it is best kept for "undefined" redirects, and the 307 Temporary Redirect status code should be preferred for this purpose.
When you change URIs, use standard redirects CM

If for any reason a content manager change the URI referencing to a given resource, standard redirects, as defined above, should be used to avoid link-rot.

Usually, the HTTP 301 Moved Permanently status code ([RFC2616], section 10.3.2) will be used for this purpose.

Guideline 3: Use independent URIs

URIs should be both stable and independent. By independent we mean that a URI should always reference the same resource, regardless of the context (time, location, user, user-agent, etc.)

3.1: Technology-independent URIs SS CM

Serve dynamic content with technology-independent URIs SS CM

A URI should not show the underlying technology (server-side content generation engine, script written in such or such language) used to serve the resource.

Using URIs showing the specific underlying technology means one is dependent on the technology used, which means that the technology cannot be changed without either breaking URIs or going through the hassle of "fixing" them (see Checkpoint 2.2: Standard redirects).

Using a scripting language to create dynamic content does not mean your URI should end with the same extension as the script's filename.

Advertizing one's development environment to the world also imply security issues. One's site may have been crawled and be a known target for a specific architecture once a security flaw is discovered on that architecture. Obscurity is, of course, no replacement for security, but a good design keeps threats away. Read the Web Security FAQ [WSFAQ] for more on web server-side security.

For these reasons, technology-specific extensions should be hidden, using content-negotiation (see Guideline 7: Server-driven content negotiation.), proxying or URI mapping technologies.
Serve static content without file extension CM

The reason why one should serve static content without file extension is similar to the reason stated above : the content manager may, at some point, want to change the document format used to serve a resource, yet the resource would remain "equivalent". For example, switching from an image file format to an equivalent format, or switching from plain text to HTML...

File extensions should therefore be hidden for static content, using content-negotiation (see Guideline 7: Server-driven content negotiation.), proxying or URI mapping technologies.

3.2: Identification and Session mechanisms SS CM

HTTP/1.1 provides a number of mechanisms for identification, authentication and session management. Using these mechanisms instead of user-based or session-based URIs guarantees than the URIs used to serve resources are truly universal (allowing, for example, people to share, send, or copy them).

Use standard identification instead of per-user URIs SS CM

For the reasons stated above, standard identification mechanisms should be prefered over user-dependent URIs.

Standard identification mechanisms for the World Wide Web are described in RFC 2617 : "HTTP Authentication: Basic and Digest Access Authentication" [RFC2617].
Use standard session mechanisms instead of session-based URIs. SS CM

For the reasons stated above, standard session mechanisms should be prefered over session-dependent URIs.

The latter may only be used in very specific cases, when standard mechanisms do not provide the desired features.

Example of an acceptable practice:
A URI may have some modifiers, like "?" used to pass arguments for cgi, or ";" to pass other kind of arguments or context information. Used for information tracking, this is a proper use of session information in URIs.

Example of a bad practice:
Bob tries to visit http://www.example.com/resource, but since it's a rainy Monday morning, he gets redirected to http://www.example.com/rainymondaymorning/resource. The day after, when Bob tries to access the resource he had bookmarked earlier, the server answers that Bob has made a bad request, and serves http://www.example.com/error/thisisnotmondayanymore. Had the server served back http://www.example.com/resource because the Monday session had expired, it would have been, if not acceptable, at least harmless.

Standard session mechanisms include RFC 2109 : "HTTP State Management Mechanism" [RFC2109], also known as "cookies".

Guideline 4: "Cool URIs don't change", but cool content does

One misconception about Cool URIs don't change is that it advocates "frozen" documents, which content cannot change because that would "break things".

This, again, comes from a misunderstanding of the concept of URIs. If we come back to our warehouse metaphor, used in the beginning of this document, things get clearer: we know that the URI is a fixed reference to a resource (a "toothbrush" in our example), and we know that the reference should not change, however it does not mean that the resource itself should not change... On the contrary, the World Wide Web has been designed with evolution in mind, and if the resource is modified over time, this has nothing to do with the fact that Cool URIs don't change.

4.1: Standard redirects for changing content SS CM

Use standard redirects for changing content SS CM

A good example of what is meant here by "changing/moving content" would be a daily article on a Web site. People want to be able to reference either the "latest daily article", or a specific article.

This is made possible and smooth with the use of two different URIs (or, to be precise, one URI referencing the "latest" issue, and one URI per article), as explained in the following example.

Let us consider an imaginary newsletter, issued every day. The (latest issue of the) newsletter is available at http://www.example.org/newsletter and this is the URI people use to access the newsletter every day.

The content manager wants that every newsletter, and not only the latest issue, be available on his server, so he archives every issue, and each of them is accessible on the Web site at a dated URI, e.g: http://www.example.org/2042/02/12-newsletter for the Feb. 12, 2042 issue.

Using a standard redirect ( HTTP 302 Found, or, even better HTTP 307 Temporary Redirect - [RFC2616] section 10.3.3 and 10.3.8), the content manager, when publishing the Feb. 12, 2042 issue, redirects http://www.example.org/newsletter to the dated http://www.example.org/2042/02/12-newsletter

Readers are, therefore, able to refer to (and access) "the newsletter" for the latest issue, or to any specific issue.

If the server properly sends the Content-Location: HTTP/1.1 Header, there is an alternate technique, described in Checkpoint 5.2: Content-Location.

4.2: HTTP 410 Gone CM SI

When removing a resource, use 410 Gone CM

Most of guidelines 1 to 3 aim at avoiding "link rot", documents that have been moved or removed, resulting in a 404 Not Found status code for agents trying to access a resource once refered to by a URI.

This does not mean the web does not allow for documents to be removed or deprecated. Content managers should avoid, when possible, simply removing resources, and should consider instead the correct standard procedure, which is to use the 410 gone status code ([RFC2616] section 10.4.11).

Whereas the 404 Not found status code only means that the server is unable to find the resource, the 410 gone status code means that the resource is intentionally unavailable. For the sake of semantics and caching (a 410 gone is cacheable unless indicated otherwise).
Allow the content-manager to use 410 Gone for removed resources SI

Content managers should be allowed to use the 410 gone status code ([RFC2616] section 10.4.11) to remove or deprecate resources on a server. There should be an easy way to specify that a resource, or an area, has been removed, using the 410 gone status code.

Guideline 5: Provide indexing agents with useful information

This section deals with providing meaningful and clear information to indexing and crawling user-agents (also often referred to as "robots", "spiders", "crawlers"). It has a strong influence on the traffic for a Web site (both the traffic created by the indexing agents, and the traffic attracted by search results) and should be a primary concern for content managers.

Discussing the use of metadata, and the proper structuring of HTML documents in order to help indexing agents in their task is out of scope for this document, we will, rather, focus on the inner mechanics of indexing. Readers interested in metadata may find interesting bits in these two related guidelines: Guideline 8: Provide useful metadata in addition to content negotiation and Guideline 12: Enrich and enhance.

5.1: Indexing policy CM

Define site-wide indexing policy CM

A site-wide policy specifies what the default behavior of indexing or crawling agents should be, and can be refined on a per-document basis through local indexing directives. (see below for details)

Content managers should define such a policy for their site. The most common way of informing indexing agents of this policy is the Robots Exclusion Protocol [ROBOTSPROTO], but one could use other technologies, such as a metadata database giving indexing directives on a document basis.
Define local indexing policy CM

The site-wide indexing policy may be completed by a local (per document) indexing policy, marked up at the document level.

For example, HTML [HTML 4.01] defines a specific META element for this purpose ([HTML 4.01] Section B.4.1).

5.2: Content-Location SI SS CM

Send valid Content-Location: SI SS

The Content-Location: HTTP header [RFC2616] section 14.14) is crucial for indexing agents as well as user agents, as it gives agents information about the actual (current) location of the resource currently served (as opposed to the generic location used to access the resource).

Content-Location: should not be mistaken for a redirection. While agents and caches may assume that a redirected URI may be used for later requests, they should not assume that a URI specified by the Content-Location: header may be used for later requests, if it differs from the requested URI. However, agents may request a URI once specified as Content-Location: if they specifically intend to request this instance of the resource.
Use Content-Location: for changing content CM

As seen previously, The Content-Location: HTTP header ([RFC2616] section 14.14) is used to inform user-agents of the actual (current) location of the requested resource. This can be used as an alternative to the temporary redirect scheme as explained in Checkpoint 4.1: Standard redirects for changing content.

Example of a good practice:
You may remember the example used in Checkpoint 4.1: Standard redirects for changing content, where the content manager uses standards redirect techniques to serve a newsletter with both a "latest" and "dated" URI.

One could achieve an almost similar result by using the Content-Location: HTTP header : serving http://www.example.org/newsletter (the "latest" URI) with a Content-Location: of http://www.example.org/2042/02/12-newsletter (the "dated" URI).

User-agents, as explained in Common User Agent Problems [CUAP] may then bookmark the "latest news" URI, or the URI of the actual dated content, and may later request the "dated" URI.
Allow the content-manager to set the Content-Location: header SI

See above for the rationale. The Content-manager should be allowed to set the Content-Location: header served for a specific resource at a given time.

5.3 Content-Md5 SI SS

Send Content-Md5 for integrity check SI SS

The Content-MD5 HTTP header ([RFC2616] section 14.15) is used to verify the integrity of the transported entity. and may help cache or indexing engines. Even though HTTP does not make it mandatory, it is recommended that servers (or content-generation engines) compute and send it.

Content-MD5 should not be mistaken with ETag ([RFC2616] section 14.19). The former is a check sum of the resource served whereas the latter is a "serial number" identifying a specific instance of resource. However he md5 sum of the content is supposed to be unique, therefore it may be used as the ETag (but may be too resource-consuming for servers that do not cache the metadata). It is, nevertheless, better to send both headers.

Guideline 6: Provide appropriate caching information

This guideline relates to the Caching mechanisms defined by the HTTP/1.1 specification ([RFC2616] section 13).

We will try to point out facts often overlooked or misunderstood about HTTP caching, as well as giving advice on how to serve easily cachable content.

6.1 Cache-related HTTP headers SI SS

Send proper and accurate Date header SI

HTTP/1.1 servers MUST send a Date: header ([RFC2616] section 14.18). It is the base of all caching mechanisms and must be sent both properly and accurately.
Send Last-Modified whenever possible SI SS

HTTP/1.1 ([RFC2616]) states that servers SHOULD send the Last-Modified header ([RFC2616] section 14.29) whenever feasible. This header is very important because of its use as a cache validator: a cache entry is considered to be valid if the entity has not been modified since the Last-Modified value.
Send Cache-Control directives SI

The Cache-Control header ([RFC2616] section 14.9) defines the behavior of cache engines with regards to the resource sent.

Cache-Control should be preferred over Expires: ([RFC2616] section 14.21) because of its richness. Servers may send both, but be aware that agents are supposed to ignore Expires: if the max-age directive of Cache-Control: is properly sent.

6.2 Cache policy SI CM

Define a cache policy CM

A cache / expiration policy is the rationale behind cache control for every resource served by HTTP/1.1 servers.. Content managers should decide, globally and/or locally, what can or can not be cached, how long caches should keep the document before trying to get a new version, etc. These decisions may be made depending on the frequency at which the documents may be updated.
Allow the Content Manager to set up cache control according to a Cache Policy SI

The content manager should be able to set the max-age parameter for any resource served according to a cache policy.

6.3: Caching generated content SS

Provide actual caching information for content generated dynamically SS

Most dynamic content generation systems act as if the documents they generate and serve were "fresh" (i.e as if the resource was last modified at the date it is served), whether the information itself is, or not.

This is a harmful lie for caching engines and should be avoided.

Regardless of the technology used, it should be possible to provide age information by retrieving the actual information from whatever source is used to generate the dynamic content: file,database, etc.

6.4: HTTP HEAD and HTTP GET SI

Send the same answer to HTTP HEAD and HTTP GET requests SI

Servers MUST send back the same information (HTTP headers) when answering a GET and a HEAD request, as required by the HTTP specification [RFC2616] section 9.4. This is critical for many mechanisms, including caching.

2.Serving content appropriately

Guideline 7: Server-driven content negotiation

Content negotiation stands for the server-driven negotiation based on user agent capabilities and user preferences, including those specified in the Accept ([RFC2616] section 14.1) Accept-Charset ([RFC2616] section 14.2), and Accept-Language ([RFC2616] section 14.4) headers, and beyond.

7.1: Format negotiation SI CM

"Format negotiation" here stands for the server-driven negotiation between equivalent instances of a resource in different "formats", either media-type (often called "content-negotiation" erroneously) or character encoding.

Allow the content manager to use and configure content-type negotiation SI

Content-managers should be provided with an easy way to specify that several documents are different instances of the same resource using various "equivalent" media types.

Server should then apply server-driven negotiation algorithms to serve the most appropriate variant based at least on the requested Accept ([RFC2616] section 14.1) header.
Allow the content manager to use and configure character encoding negotiation SI

Content-managers should be provided with an easy way to specify that several documents are different instances of the same resource with different character encoding.

Server should then apply server-driven negotiation algorithms to serve the most appropriate variant based at least on the requested Accept-Charset ([RFC2616] section 14.2) header.
During format negotiation, be cautious with agents accepting anything SI CM

As explained for example in "Common user agent problems" ([CUAP] section "protocols"), some agents are known to misbehave with regard to format negociation, sending an HTTP header of Accept: */* (thus they are supposed to support every and any content type, which they certainly do not).

While servers are not required to cope with this problem in user agents, a wise practice toward agents sending broken Accept: headers or not expressing specific preference on the content type is to send them a version of the resource in a widely supported document format.

This can be done at the server level using the quality factors used in the negotiation process ([RFC2616] section 12).

See also the related guideline : Guideline 11 : Use flexible technology instead of client sniffing/blocking.
Allow the content manager to set the quality factors used during negociation SI

Content-managers should be provided with an easy way to specify which version (either format or language) of the resource they would rather see served, in case the headers sent by the agent do not leave one clear choice.

See related checkpoint 9.1: When negotiation fails.

7.2: Language negotiation SI

Allow the content manager to use and configure language negotiation SI

Content-managers should be provided with an easy way to specify that several documents are different instances of the same resource translated in different languages.

Server should then apply server-driven negotiation algorithms to serve the most appropriate variant based at least on the requested Accept-Language ([RFC2616] section 14.4) header.
Allow the content manager to set the quality factors used during negociation SI

Content-managers should be provided with an easy way to specify which version (either format or language) of the resource they would rather see served, in case the headers sent by the agent do not leave one clear choice.

See related checkpoint 9.1: When negotiation fails.
Use the Content-Language: HTTP header SI

If the resource is served using language-negotiation (actually, even if it is not), servers MAY send a Content-Language: HTTP header specifying the language of the instance of the resource served. This is an interesting information that agents may use to evaluate the result of server-driven negotiation, exactly as they would with the Content-Type header in the case of format negotiation.

Example of HTTP/1.1 transaction using Content-Language:
```
GET /foo/resource HTTP/1.1
Host: www.example.org
Accept-Language: fr, en-gb;q=0.8, de;q=0.1

HTTP/1.1 200 OK
[...]
Content-Location: http://www.example.org/foo/resource.html.fr
Content-Language: fr
[...]
```

Guideline 8: Provide useful metadata in addition to content negotiation

Server-driven negotiation is used to serve the best content available, based on the accept headers received. This mechanism does not, however, specify variants beyond the generic Vary: HTTP header.

This guideline gives hits at going a little further for the sake of ease of navigation through, and indexing of, multiple HTML documents (variants or collection).

8.1: Variants of (X)HTML documents SS CM

Specify variants of HTML documents SS CM

The HTML specification [HTML 4.01], provides mechanisms to specify (language) variants for a given document ([HTML 4.01] appendix B.4) using the link element ([HTML 4.01] section 12.3).

When used with the alternate type, the link element can specify variants of a given resource, either language variants (translations) with the lang attribute or media variants with the media attribute.

Example of HTML markup for language variants:
```
<LINK rel="alternate" 
         type="text/html"
         href="mydoc-fr.html" hreflang="fr"
         lang="fr" title="La vie souterraine">
<LINK rel="alternate" 
         type="text/html"
         href="mydoc-de.html" hreflang="de"
         lang="de" title="Das Leben im Untergrund">
```

Specify variants of XHTML documents SS CM

Note that this technique is is also applicable for XHTML documents.

Example of XHTML 1.0 markup for language variants (same as above, but with lower-case, closed elements...):

<link rel="alternate" 
         type="text/html"
         href="mydoc-fr.html" hreflang="fr"
         lang="fr" title="La vie souterraine" />
<link rel="alternate" 
         type="text/html"
         href="mydoc-de.html" hreflang="de"
         lang="de" title="Das Leben im Untergrund" />

8.2: Navigation among (X)HTML documents CM

Facilitate navigation among collections of HTML documents CM

Again, using the link element ([HTML 4.01] section 12.3) one can specify relations betweens documents in a collection.

The link types which can be used for this purpose, as described in the Data types section of the HTML 4.01 specification [HTML 4.01] are:
- Start
- Next
- Prev
- Contents
- Index
- etc.
examples of use:
- in a photo gallery (using Next, Prev, Index, etc.)
- for a periodical newsletter (using Next, Prev, Copyright, etc)
- in a compound document (using Contents, Chapter, Section, Subsection, Appendix, Glossary, etc.)

Guideline 9: Provide default and fall-back solutions

HTTP [RFC2616] is about serving content in the most appropriate way, and, as we have seen in previous guidelines ( Guideline 7 : Server-driven content negotiation and Guideline 8: Provide useful metadata in addition to content negotiation), server-driven negotiation may be used to serve the best available content. It may happen that these mechanisms fail, and in this case, HTTP implementations should try, when possible, to give the requested content to the client. This may be achieved through default and fall-back mechanisms.

9.1: When negotiation fails SI SS

provide multiple or default choice(s) when content/language negotiation fails to give only one result SI SS

Using the verbiage of the HTTP specification, this checkpoint can be paraphrased into "use agent-driven negotiation when the server is unable to provide a varying response using server-driven negotiation".

Section 12 of HTTP [RFC2616] provides mechanisms to leave the final decision to the user-agent (or its user) for cases when the content or language negotiation does not come up with a unique result but with multiple ones.

In such a case, a server can use the 300 (Multiple Choices) status code, or be configured to send, by default, one of the resources among the possible choices.
provide default or fall-back choice(s) when content/language negotiation fails SI SS

Section 12 of HTTP/1.1 [RFC2616] suggests the use of the 406 (Not Acceptable) status code when content or language negotiation fails to find any appropriate negotiated resource.

However the HTTP/1.1 specification [RFC2616] also states that the server should make the best efforts to give the requested content to the client.

One possible interpretation of this is that the server may provide fall-back choice(s): the message body for "HTTP 406 not acceptable" can give a list of available resources and let the user choose, or the server can be configured to serve, arbitrarily, a specific variant of the resource in case negotiation fails.

Note that this is perfectly acceptable with regards to Section 10.4.7 of HTTP/1.1 [RFC2616]: HTTP/1.1 servers are allowed to return responses which are not acceptable according to the accept headers sent in the request. In some cases, this may even be preferable to sending a 406 response. User agents are encouraged to inspect the headers of an incoming response to determine if it is acceptable.
allow the content manager to set up a fall-back behavior content/language for cases when negotiation fails SI SS

This is the practical implementation of the provision above. The server should allow the content manager to decide whether, in case negotiation fails, the server should:
- send a 406 (Not Acceptable) status code with a list of available choices,
- or if it should arbitrarily serve a variant of the resource. (The content manager should, of course, be allowed to choose which variant would be chosen, or how it should be chosen.).
Example:
Through the Accept-Language headers, a client specifies that it prefers Japanese or English versions of the resource, whereas the content is only available in French and Spanish. The content manager may be allowed to choose that the French version will be served as a default version, or let the server send a 406 status code, giving the user-agent a choice between the French and Spanish versions.

9.2: HTTP error messages body SI SS

As a general rule, the content manager should be allowed to change and customize the body of HTTP error messages.

Guideline 10: Serve resources with correct content-type and character encoding information

10.1: Content-type SI SS CM

Send proper Content-type HTTP header SI SS CM

Resources should be served with a proper Content-type Header ([RFC2616] section 14.17). Documents not served with a proper media type may not be interpreted correctly by user agents.

Example of a wrong practice:
CSS style sheets are sometimes served as plain text (text/plain media type), causing the user-agents to ignore the style sheet and rendering the document in an unexpected manner.

Example of a proper practice:
CSS style sheets should be served with the text/css media type.
allow the content manager to override content-type settings SI

In addition to proper default mapping of media types to file extension, since there is no obligation to use "well-known" file extensions in URIs, servers should allow the content manager to set the appropriate media type sent in the Content-type header for resources without such file extension, and to override the default setting at will.

10.2: Character Encoding SI SS CM

Send proper character encoding information SI SS CM

For some document types, the media type sent by the Content-type Header ([RFC2616] section 14.17) may be sent with some information about the character encoding of the document. In some cases, this is mandatory (see the provision below for HTML and XHTML).
Send proper character encoding information for XHTML documents SI SS CM

The HTML 4.01 Recommendation ([HTML 4.01] section 5.2.2) states that the server should provide this information (the character encoding of the HTML document served), e.g:

Content-Type: text/html; charset=EUC-JP

Conforming user agents MUST observe the following priorities when determining an HTML document's character encoding (from highest priority to lowest):
1. An HTTP "charset" parameter in a "Content-Type" field
2. A META declaration with "http-equiv" set to "Content-Type" and a value set for "charset"
3. The charset attribute set on an element that designates an external resource.
Note that The HTTP/1.1 protocol ([RFC2616], section 3.7.1) mentions ISO-8859-1 as a default character encoding when the "charset" parameter is absent from the "Content-Type" header field, but it is now not recommended to follow this practice.

The recommended practice is that the character encoding be both specified be specified in the META declaration, and the "Content-Type" header field.

Example of an HTML 4.01 document written in French with a UTF-8 encoding:
```
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN"
  "http://www.w3.org/TR/html4/strict.dtd">
<html lang="fr">

<head>
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<title>Exemple de document HTML 4.01</title>
</head>

<body>
<h1>Portrait Intérieur</h1>
<h2>Rainer-Maria Rilke</h2>

<p>Ce ne sont pas des souvenirs<br>
qui, en moi, t'entretiennent ;<br>
tu n'es pas non plus mienne<br>
par la force d'un beau désir.</p>
</body>
</html>
```
Send proper character encoding information for XHTML 1.0 documents SI SS CM

The case of XHTML document is similar to the case of HTML, except that, since XHTML is also XML,XHTML document can provide the character encoding via the XML declaration. (but if the XHTML document uses one of the default encodings - UTF-8 or UTF-16 - no declaration is needed).

The recommended practice for XHTML documents is to properly specify the character encoding in both the XML declaration and the the "Content-Type" header field.

Example of an XHTML 1.0 document written in French with an ISO-8859-1 encoding:
```
<?xml version="1.0" encoding="ISO-8859-1"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
  "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="fr" lang="fr">

<head>
<title>Exemple de document XHTML 1.0</title>
</head>

<body>
<h1>Portrait Intérieur</h1>
<h2>Rainer-Maria Rilke</h2>
<p>Ce ne sont pas des souvenirs<br />
qui, en moi, t'entretiennent ;<br />
tu n'es pas non plus mienne<br />
par la force d'un beau désir.</p>
</body>
</html>
```
Allow the content manager to override character encoding settings SI SS

The content manager should be allowed to set the character encoding information.

If the server implementor does not want the content manager, or if the content manager does not want the users to change the charset information sent by the HTTP server, then the server should send none, and the character encoding may be specified at the document level.

Guideline 11: Use flexible technology instead of client sniffing/blocking

11.1: Avoid agent sniffing SS CM

Use content-negotiated resources instead of Agent sniffing SS CM

Server-driven negotiation, based on the agent's capabilities (given though the Accept: header -[RFC2616 section 14.1 -) is a very efficient way of providing agents with content they can display or process, without doubt on their capabilities. It is also a cost-efficient technique, as the negotiation is handled by the server based on what agents declare they can handle, whereas agent sniffing implies knowledge of (potentially all) agents and their capabilities in order to serve (only) content the agents can handle.

Providing (with negotiation) equivalent versions of a resource in flexible technologies should therefore be preferred to agent-sniffing.
Use flexible document technologies instead of Agent sniffing SS CM

Content manager often think they have to serve different content depending on the agent, either by generating different content on the fly using server-side technologies, filtering, negotiating, or Blocking.

However well done (negotiating being the most appropriate way), this practice is very seldom suitable to any possible agent, and implies a lot of extra work.

Content-managers should therefore consider the use of standard (i.e widely implemented), flexible (scalable, multi-platform, device independent, etc.) document technologies whenever possible, either as a primary choice, or, at least, as a negotiated alternative.

Example of an acceptable practice:
The content manager decides to serve a text resource using proprietary, not widely implemented technology, but adds a negotiated plain-text alternative for agents which can not handle the proprietary document format.

11.2: Avoid agent blocking SS CM

Avoid agent blocking SS CM

Even though some agents may be badly broken, refusing to serve content to users of such an agent means lost business (traffic), and flexible technologies, which ensure that the content may be handled by any agent, should be preferred to this practice.

Even worse is to choose which agents are "suitable" and block all the other agents. This is a very bad move, at least because:
- Some of the agents one may block are actually idexing agents for search engines, and may bring back traffic
- Agents are rapidly evolving, and while a specific version of a specific agent may appear better at some point in time, there is no reason to believe another version of another agent may not be more appropriate later, hence making the blocking rules obsolete
- Blocking agents means refusing to serve, and ultimately means lost business (traffic).
Agent blocking should therefore be avoided as much as possible, and instead flexible negotiation and document technologies, as described in Checkpoint 11.1, should be used.

Guideline 12: Enrich and enhance

The previous guidelines showed good practices for the implementation and use of Web server technologies. We will close this document by adding a few leads to practices which, even though they are not crucial, may be followed to enrich or enhance HTTP services...

12.1: Transfer encoding SI

Use transfer encoding for bandwidth-constrained devices SI
Serving content to bandwitdh-constrained devices (this includes among many others, mobile devices), can be improved via on the fly connection, using the Transfer-Encoding HTTP header ([RFC2616] section 14.41).

12.2: From (meta)data to server information SI SS

This checkpoint is on the verge of the server side, and is added here as a proof of concept that the content itself can be used to enhance configuration held by and information sent by the HTTP server.

Convert (meta)data into HTTP information SI SS

Information in or about a resource (data or metadata) may be used by a web server, either as a way to adapt its configuration, as extra information that can be sent in the HTTP headers (standard, or custom), or as an alternate machine-readable (metadata) version of the resouce.

A few examples:
- Extracting
  - meta information (e.g. language, author, the Dublin Core set of information) from HTML documents
  - the content type from the HTML meta tag
  - metadata embedded in images
- and
  - Using it to serve the resource (e.g. the language or the content type can be sent in the standard HTTP headers Content-Type and Content-Language ([RFC2616] sections 14.17 and 14.12)
  - Using it to build a metadata database used by the server (e.g. for the indexing policy, for negotiation)
  - Generating, on the fly, an alternate machine-readable (metadata) version of the resouce (e.g. in RDF)

Tabular checklist of guidelines and checkpoints

You may use this table as a quick and convenient tool to assess your progress in following the guidelines given in this document.

Number	Title	target
Guideline 1: Choose URIs wisely
1.1	Short URIs	SS CM
1.2	URI case policy	SS CM
Guideline 2: Allow URI management
2.1	URI mapping	SI
2.2	Standard redirects	SI CM
Guideline 3: Use independent URIs
3.1	Technology-independent URIs	SS CM
3.2	Identification and Session mechanisms	SS CM
Guideline 4: Use standard redirects for content that changes
4.1	Standard redirects for changing content	SS CM
4.2	HTTP `410 Gone`	CM SI
Guideline 5: Provide indexing agents with useful information
5.1	Indexing policy	CM
5.2	`Content-Location`	SI SS CM
5.3	`Content-Md5`	SI SS
Guideline 6: Provide appropriate caching information
6.1	Cache-related HTTP headers	SI SS
6.2	Cache policy	SI CM
6.3	Caching generated content	SS
6.4	HTTP HEAD and HTTP GET	SI
Guideline 7: Server-driven content negotiation
7.1	Format negotiation	SI CM
7.2	Language negotiation	SI
Guideline 8: Provide useful metadata in addition to content negotiation
8.1	Variants of (X)HTML documents	SS CM
8.2	Navigation among (X)HTML documents	CM
Guideline 9: Provide default and fall-back solutions
9.1	When negotiation fails	SI SS
9.2	HTTP error messages body	SI SS
Guideline 10: Serve resources with correct content-type and character encoding information
10.1	`Content-type`	SI SS CM
10.2	Character Encoding	SI SS CM
Guideline 11: Use flexible technology instead of client sniffing/blocking
11.1	Avoid agent sniffing	SS CM
11.2	Avoid agent blocking	SS CM
Guideline 12: Enrich and Enhance
12.1	Transfer encoding	SI
i12.2	From (meta)data to Server information	SI SS

Acknowledgments

The editor would like to thank the following W3C Team members for the initial input and their collaboration in writing this document.

The editor would also like to thank the following people for their early review of the document: