,QWHUQHW6RFLHW\ ,62&
,QWHUQHW$UFKLWHFWXUH%RDUG
Requests for Comments (RFC)
6HOHFWHG5HTXHVWVIRU&RPPHQWVUHOHYDQWWRLQVWUXFWLRQDOGHVLJQ5HTXHVWV IRU&RPPHQWVRU5)&VLQFOXGH,QWHUQHWVWDQGDUGVGHYHORSHGE\WKH ,QWHUQHW$UFKLWHFWXUH%RDUGIRUWKH,QWHUQHW6RFLHW\0DQ\RIWKH5)&V SURYLGHLPSRUWDQWWHFKQLFDOLQIRUPDWLRQDQGDUHQRWFODVVLILHGDV ³VWDQGDUGV´
5HSURGXFHGLQDFFRUGDQFHZLWKWKHSROLFLHVRIWKH,QWHUQHW6RFLHW\)RUWKH ODWHVWLQIRUPDWLRQDERXWWKH6RFLHW\VHHZZZLVRFRUJ)RUPRUH LQIRUPDWLRQDERXWVWDQGDUGVVHHZZZLDERUJ RFC1314 Web Address: Image Exchange Format http://sunsite.cnlab-switch.ch/ By: Katz & Cohen
Network Working Group A. Katz Request for Comments: 1314 D. Cohen ISI April 1992
A File Format for the Exchange of Images in the Internet
Status of This Memo
This document specifies an IAB standards track protocol for the Internet community, and requests discussion and suggestions for improvements. Please refer to the current edition of the "IAB Official Protocol Standards" for the standardization state and status of this protocol. Distribution of this memo is unlimited.
Abstract
This document defines a standard file format for the exchange of fax-like black and white images within the Internet. It is a product of the Network Fax Working Group of the Internet Engineering Task Force (IETF).
The standard is:
** The file format should be TIFF-B with multi-page files supported. Images should be encoded as one TIFF strip per page.
** Images should be compressed using MMR when possible. Images may also be MH or MR compressed or uncompressed. If MH or MR compression is used, scan lines should be "byte-aligned".
** For maximum interoperability, image resolutions should either be 600, 400, or 300 dpi; or else be one of the standard Group 3 fax resolutions (98 or 196 dpi vertically and 204 dpi horizontally).
Note that this specification is self contained and an implementation should be possible without recourse to the TIFF references, and that only the specific TIFF documents cited are relevant to this specification. Updates to the TIFF documents do not change this specification.
Experimentation with this file format specified here is encouraged.
Formatted and Indexed by: page 1 of 23 instructional media + magic, inc. RFC1314 Web Address: Image Exchange Format http://sunsite.cnlab-switch.ch/ By: Katz & Cohen
1. Introduction
The purpose of this document is to define a standard file format for exchange of black and white images using the Internet. Since many organizations have already started to accumulate and exchange scanned documents it is important to reach agreement about an interchange file format in order to promote and facilitate the exchange and distribution of such documents. These images may originate from scanners, software, or facsimile (fax) machines. They may be manipulated by software, communicated, shared, duplicated, displayed, printed by laser printers, or faxed.
This file format provides for the uniform transfer of high quality images at a reasonable cost and with reasonable speed whether these files are generated by scanners, totally by software (e.g., text-to- fax, bitmap-to-fax, OCR, etc), or by fax. Also the intent of this document is to remain compatible with future moves to multi-level (i.e., gray-scale), higher resolution, or color images. The format proposed here is supported by both commercially available hardware and commercial and public domain software for most popular platforms in current use.
The file format for images is a totally separate issue from how such files are to be communicated. For example, FTP or SMTP could be used to move an image file from one host to another, although there are complications in the use of SMTP as currently implemented due to file size and the need to move binary data. (There is currently a proposal for removing these limitations from SMTP and in particular extending it to allow binary data. See reference [1].)
One major potential application of the communications format defined here is to allow images to be sent to fax machines using the Internet. It is intended that one or more separate companion documents will be formulated to address the issues of standardization in the areas of protocols for transmitting images through the Internet and the issues of addressing fax machines and routing faxes. Just as the exchange format is separate from the transmission mechanism, it is also separate from how hosts store images.
This document specifies a common exchange format; it does not require a host to store images in the format specified here, only to convert between the host's local image storage formats and the exchange format defined here for the purpose of exchanging images with other hosts across the network.
This standard specifies the use of TIFF (Tagged Image File Format, see below) as a format for exchange of image files. This is not a specific image encoding, but a framework for many encoding
Formatted and Indexed by: page 2 of 23 instructional media + magic, inc. RFC1314 Web Address: Image Exchange Format http://sunsite.cnlab-switch.ch/ By: Katz & Cohen
techniques, that can be used within the TIFF framework. For example, within TIFF it is possible to use MMR (the data encoding of CCITT Group 4 fax, see below), MH or MR (the data encodings of CCITT Group 3 fax), or other encoding methods.
Which encoding technique to use is not specified here. Instead, with time the encoding schemes used by most document providers will emerge as the de-facto standard. Therefore, we do not declare any as "the standard data encoding scheme," just as we do not declare that English is the standard publication language. (However, we expect that most document providers will use MMR in the immediate future because it offers much better compression ratios than MH or MR.)
Similarly, TIFF does not require that an image be communicated at a specific resolution. Resolution is a parameter in the TIFF descriptive header. We do suggest that images now be sent using one of a set of common resolutions in the interests of interoperability, but the format accommodates other resolutions that may be required by specialized applications or changing technologies.
Occasionally, image files will have to be converted, such as in the case where a document that was scanned at 400 dpi is to be printed on a 300 dpi printer. This conversion could be performed by the document provider, by the consumer, or by a third party. This document specifies neither who performs the conversion, nor which algorithms should be used to accomplish it.
Note that this standard does not attempt to define an exchange format for all image types that may be transmitted in the Internet. Nothing in this standard precludes it from being used for other image type such as gray-scale (e.g., JPEG) or color images but, for the purposes of standardization, the scope of this document is restricted to monochromatic bitmapped images.
The developers of this standard recognize that it may have a limited lifespan as Office Document Architecture (ODA) matures and comes into use in the Internet; ultimately the class of images covered by this standard will likely be subsumed by the more general class of images supported by the ODA standards. However, at present, there does not appear to be a sufficient installed base of ODA compliant software and the ODA standards are not fully mature. This standard is intended to fill the need for a common image transfer format until ODA is ready. Finally, we believe that it should be possible to automatically map images encoded in the format specified here into a future ODA-based image interchange format, thus providing a reasonable transition path to these future standards.
Formatted and Indexed by: page 3 of 23 instructional media + magic, inc. RFC1314 Web Address: Image Exchange Format http://sunsite.cnlab-switch.ch/ By: Katz & Cohen
2. Relationship to Fax
Transmission of facsimile (fax) images over phone lines is becoming increasingly widespread. The standard of most fax machines in the U.S. is CCITT Group 3 (G3), specified in Recommendations T.4 and T.30 [2] and in EIA Standards EIA-465 and EIA-466. G3 faxes are 204 dots per inch (dpi) horizontally and 98 dpi (196 dpi optionally, in fine-detail mode) vertically. Since G3 neither assumes error free transmission nor retransmits when errors occur, the encoding scheme used is differential only over small segments never exceeding 2 lines at standard resolution or 4 lines for fine-detail. (The incremental G3 encoding scheme is called two-dimensional and the number of lines so encoded is specified by a parameter called k.)
CCITT Group 4 fax (G4) is defined by the T.400 and T.500 series of Recommendations as well as Recommendation T.6 [2]. It provides for 400 dpi (both vertical and horizontal) and is a fully two-dimensional encoding scheme (k is infinite) called MMR (Modified Modified READ, where READ stands for: Relative Element Address Designate). G4 assumes an error free transmission medium (generally an X.25 Public Data Network, or PDN). Because of this, G4 is not in widespread use in the U.S. today.
The traditional fax bundles together four independent issues:
(1) Data presentation and compression; (2) Data transmission; (3) Image input from paper ("scanning"); and (4) Image output to paper ("printing").
This bundling supports, for example, the high quality CCITT Group 4 (G4) images (400x400 dpi) but only over X.25 public data networks with error correction, and similarly it supports the mid-quality CCITT Group 3 (204x98 and 204x196 dpi) but only over phone voice circuits (the Switched Telephone Network, or STN) without error correction. This bundling does not support the use of any other data transmission capabilities (e.g., FTP over LANs and WANs), nor asynchrony between the scanning and the printing, nor image storage, nor the use of the popular laser printers for output (even though they are perfectly capable of doing so).
In conventional fax, images are never stored. In today's computer network environment, a better model is:
(1) Images are scanned into files or created by software; (2) These image files are stored, manipulated, or communicated; (3) Images in a file are printed or displayed.
Formatted and Indexed by: page 4 of 23 instructional media + magic, inc. RFC1314 Web Address: Image Exchange Format http://sunsite.cnlab-switch.ch/ By: Katz & Cohen
The only feature of the CCITT fax that should be used is the encoding technique (preferably MMR, but with MR or MH allowed) which may be implemented with a variety of fax-oriented chips at low cost due to the popularity of fax.
"Sending a fax" means both encoding (and decoding) the fax images as well as transmitting the data. Since the Internet ALREADY provides several mechanisms for data transmission (in particular, FTP for general file transmission), it is unnecessary to use the data transmission methods specified in the CCITT standard. Within the Internet, each fax image should be stored in a file and these files could be transferred (e.g., using FTP, SMTP, RPC-based methods, etc.).
Fax machines should be considered just as scanners and printers are, as I/O devices between paper and files; but not as a transmission means. Higher quality Group 4 images are thus supported at low cost, while enjoying the freedom to use any computerized file transfer and duplication mechanism, standard laser printers, multiple printing (possibly at multiple remote sites) of the same image without having to rescan it physically, and a variety of software for various processing of these images, such as OCR and various drawing programs. We should be able to interoperate with files created by fax machines, scanners, or software and to be able to print all of them on fax machines or on laser printers.
The CCITT Recommendations assume realtime communications between fax machines and do not therefore specify any kind of fax file format. We propose using TIFF [3] which seems to be emerging as a standard, for encapsulation of encoded images. Because they assume realtime communications, the CCITT fax protocols require negotiations to take place between the sender and receiver. For example, they negotiate whether to use two-dimensional coding (and with what k parameter) and what (if any) padding there is between scan lines.
In our approach, the image in the file is already compressed in a particular manner. If it is to be sent to an ordinary fax machine using a fax board/modem, that board will perform the negotiations with the receiving fax machine. In the cases where the receiver cannot handle the type of compression used in the file, it will be necessary to convert the image to another compression scheme before transmission. (Most fax cards seem to either store images using the default values of the parameters which are negotiated or in a format which can quickly be converted to this. With currently available hardware and software, any necessary format conversion should be easy to accomplish.)
In conventional fax, if the compression used for a particular image
Formatted and Indexed by: page 5 of 23 instructional media + magic, inc. RFC1314 Web Address: Image Exchange Format http://sunsite.cnlab-switch.ch/ By: Katz & Cohen
is "negative" (i.e., the compressed form is larger than the uncompressed form, something that happens quite frequently with dithered photographic images), the larger compressed form of the image is still sent. If the images are first scanned into files, this problem could be recognized and the smaller, uncompressed file sent instead. (Also, Recommendations T.4 and T.6 [2] allow for an "uncompressed mode." Thus, lines which have negative compression may each be sent uncompressed. However, very few G3 fax machines support this mode.)
3. Image File Format
Image files should be in the TIFF-B format which is the bi-level subclass of TIFF. TIFF and TIFF-B are described in reference [3], cited at the end of this document. Images should be compressed using MMR (the G4 compression scheme) because it offers superior compression ratios. However, images may also be compressed using MH or MR (the G3 methods). MMR offers much better compression ratios than these (which are used in G3 fax because of the lack of an error-free communications path).
TIFF-F, described in [4], is the proposed subclass of TIFF-B for fax images. However, since TIFF-F was intended for use with G3, it recommends against certain features we recommend. Specifically, it suggests not using MMR or MR compression (we recommend MMR and allow MR) and prohibits uncompressed mode (which we allow and suggest for some photographic images). Apart from these, the TIFF-F restrictions should be followed. (Complete compatibility between the format specified here and TIFF-F can only be guaranteed for MH compressed images.)
[NOTE: Aldus Corp., the TIFF Developer, considers fax applications to be outside the scope of mainstream TIFF since it is not a part of general publishing which is what TIFF was originally designed for. They specify the LZW [5] compression scheme rather than MMR. We, however, are concerned with the transmission and storage of images rather than publishing. Therefore, we are more concerned with compression ratios and compatibility with CCITT fax than Aldus is.]
TIFF itself allows for gray-scale and color images. Image files should be restricted to TIFF-B for now because most of the currently available hardware is bi-level (1 bit per pixel). In the future, when gray-scale or color scanners, printers, and fax becomes available, the file format suggested here can already accommodate it. (For example, though JPEG is not currently a TIFF defined compression type, work is currently underway for including it as such.)
Formatted and Indexed by: page 6 of 23 instructional media + magic, inc. RFC1314 Web Address: Image Exchange Format http://sunsite.cnlab-switch.ch/ By: Katz & Cohen
[NOTE: In this document, we will use the term "reader" or "TIFF reader" to refer to the process or device which reads and parses a TIFF file.]
3.A. TIFF File Format
Figure 1 below (reproduced here from Figure 1 of reference [3]) depicts the structure of a TIFF file.
TIFF files start with a file header which specifies the byte order used in the file (i.e., Big or Little Endian), the TIFF version number, and points to the first "Image File Directory" (IFD). If the first two bytes are hex 4D4D, the byte order is from most to least significant for both 16 and 32 bit integers (Big Endian). If the first two bytes are hex 4949, the byte order is from least to most significant (Little Endian). In both formats, character strings are stored into sequential bytes and are null terminated.
The next two bytes (called the TIFF Version) must be 42 (hex 002A). This does not refer to the current TIFF revision number. The following 4 bytes contain the offset (in bytes from the beginning of the file) to the first IFD.
An IFD contains a 2 byte count of the number of entries in the IFD, a sequence of 12 byte directory entries, and a 4 byte pointer to the next IFD. One of these fields (StripOffsets) points to (parts of) an image in the file. There may be more than one image in the file (e.g., a "multi-page" TIFF file) and therefore more then one IFD. IFD field entries may appear in any order.
Each directory entry is 12 bytes and consists of a tag, its type, a length, and an offset to its value. If the value can fit into 4 bytes (i.e., if the type is BYTE, SHORT, or LONG), the actual value rather than an offset is given. If the value is less than 4 bytes (i.e., if the type is BYTE or SHORT), it is left-justified within the 4 byte value offset. More details about directory entries and the possible tags will be given in Section 3.C.
All pointers (called offsets in the TIFF reference [3]) are the number of bytes from the beginning of the file and are 4 bytes long. The first byte of the file has an offset of 0. In the case of only one image per file, there should therefore be only one IFD. The last IFD's pointer to the next IFD is set to hex 00000000 (32 bits).
The entries in an IFD must be sorted in ascending order by Tag.
Formatted and Indexed by: page 7 of 23 instructional media + magic, inc. RFC1314 Web Address: Image Exchange Format http://sunsite.cnlab-switch.ch/ By: Katz & Cohen
Header +------+------+ Directory Entry 0 | | | Byte Order +------+------+ +------+------| X | | | Tag 2 | | | Version(42) +------+------| +------+------| X+2 | | | Type 4 | | | Offset of +------+------| +- - A - -+ 0th IFD X+4 | | | 6 | | | +- -+ Length +------+------+ | | | | +------+------+ | X+8 | | | Value | +- - Y - -+ or V | | | Value +------+------+ Offset IFD +------+------+ | A | - B - | Entry Count | +------+------| | | | | V A+2 Entry 0 | | | +------+------+ +------+------+ | | | | | | Y Value A+14 Entry 1 | | | | | | +------+------| +------+------+ | | | A+26 Entry 2 | | | +------+------+ | | | . . | | | . +------+------+ | | | Entry B-1 | | | +------+------+ | | | Offset of A+2+B*12 - C - + Next IFD | | | +------+------+ | V (next IFD)
Figure 1: The Structure of a TIFF File
Formatted and Indexed by: page 8 of 23 instructional media + magic, inc. RFC1314 Web Address: Image Exchange Format http://sunsite.cnlab-switch.ch/ By: Katz & Cohen
3.B. Image Format and Encoding Issues
Images in TIFF files are organized as horizontal strips for fast access to individual rows. One can specify how many rows there are in each strip and all of the strips are the same size (except possibly the last one). Each strip must begin on a byte boundary but successive rows are not so required. For two-dimensional G3 compression (MR), each strip must begin with an "absolute" one- dimensional line. For MMR (G4) compression, each strip must be encoded as if it were a separate image.
For a variety of reasons, each page must be a single strip (e.g., not broken up into multiple strips).
One problem with multiple strips per page is that images which come from G4 fax machines as well as most scanned images will be generated as a single strip per page. These would have to be decoded and re- encoded as multiple strips (remember that for MMR compression, each strip must be start with a one-dimensionally encoded line).
Another problem with multiple strips per page arises in MR compression. Here, there MAY be at most k-1 two-dimensionally encoded lines following a one-dimensionally encoded line, but this is not required. It is possible to have one-dimensional lines more frequently than every k lines. However, since each strip (except possibly the last one) is required to be the same size, it may be necessary to re-encode the image to insure that each strip starts with a one-dimensional line. This is not a problem if each page is a single strip.
[NOTE: The TIFF document [3] suggests using strips which are about 8K bytes long. However, TIFF-F [4] recommends that each page be a single strip regardless of its size. The format specified in this document follows the TIFF-F recommendation.]
Also, as TIFF-F recommends, all G3 encoded images (MH and MR) should be "byte-aligned." This means that extra zero bits (fill bits) are added before each EOL (end-of-line) so that every line starts on a byte boundary.
In addition, as in the TIFF-F specification, the RTC (Return to Control signal which consists of 6 continuous EOL's) of G3 shall not be included at the end of G3 encoded documents. RTC is to be considered part of the G3 transmission protocol and not part of the encoding. Most, if not all, G3 fax modems attach RTC to outgoing images and remove it from incoming ones.
Formatted and Indexed by: page 9 of 23 instructional media + magic, inc. RFC1314 Web Address: Image Exchange Format http://sunsite.cnlab-switch.ch/ By: Katz & Cohen
For MMR (G4) encoded files, readers should be able to read images with only one EOFB (End Of Facsimile Block) at the end of the page and should not assume that Facsimile Blocks are of any particular size. (It has been reported that some MMR readers assume that all Facsimile Blocks are the maximum size.)
Systems may optionally choose to store the entire image uncompressed if the compression increases the size of the image file. Also, uncompressed mode (specified in Group3Options or Group4Options, see below) allows portions of the image to be uncompressed.
The multi-page capability of TIFF is supported and should be used for multi-page documents. TIFF files which have multiple pages have an IFD for each page of the document each of which describes and points to a single page image. (Note: though the current TIFF specification does not specifically prohibit having a single IFD point to an image which is actually multiple pages, with one strip for each page, most if not all TIFF readers would probably not be able to read such a file. Therefore, this should not be done.)
[A NOTE ON TIFF AND MULTI-PAGE DOCUMENTS:
Since most publications (e.g., reports, books, and magazine articles) are composed of more than a single page, multi-page TIFF files should be used where appropriate. However, many current TIFF implementations now only handle single-page files.
It is hoped that in the future, more TIFF implementations will handle multi-page files correctly. In the meantime, it would be useful to develop a utility program which could join several single-page TIFF files into a single multi-page file and also separate a multi-page TIFF file into several single page files.
For example, the utility could take a single TIFF file with N pages, called doc.tif, and create the files doc.000, doc.001, doc.002, ..., doc.N. doc.000 would be an ASCII listing of the files created. This naming scheme is compatible with that used by the image systems we have seen which only handle single page files.
In going the other way, the N+1 single page files could be combined into a single multi-page TIFF file. In this case, if the file doc.000 exists but contains information contrary to what is found in looking for the files doc.001, doc.002, ..., the program would notify the user.]
Formatted and Indexed by: page 10 of 23 instructional media + magic, inc. RFC1314 Web Address: Image Exchange Format http://sunsite.cnlab-switch.ch/ By: Katz & Cohen
3.C. TIFF Fields
TIFF is tag or field based. The various fields and their format are listed in [3]. There are Basic Fields (common to all TIFF files), Informational Fields (which provide useful information to a user), Facsimile Fields (used here), and Private Fields.
Each directory entry contains:
The Tag for the field (2 bytes)
The field Type (2 bytes)
The field Length (4 bytes) (This is in terms of the data type, not in bytes. For example, a single 16-bit word or SHORT has a Length of 1, not 2)
The Value Offset (4 bytes) (Pointer to the actual value, which must begin on a word boundary. Therefore, this offset will always be an even number. If the Value fits into 4 bytes, the Value Offset contains the Value instead. If the Value takes less than 4 bytes, it is left justified)
The allowed types and their codes are:
1 = BYTE 8-bit unsigned integer (1 byte)
2 = ASCII 8-bit ASCII terminated with a null (variable length)
3 = SHORT 16-bit unsigned integer (2 bytes)
4 = LONG 32-bit unsigned integer (4 bytes)
5 = RATIONAL Two LONGs (64 bits) representing the numerator and denominator of a fraction. In this document, RATIONAL's will be written as numerator/denominator. (8 bytes)
For ASCII, the Length specifies the number of characters and includes the null. It does not, however, include padding if such is necessary.
(Note that ASCII strings of length 3 or less may be stored in the Value Offset field instead of being pointed to.)
Formatted and Indexed by: page 11 of 23 instructional media + magic, inc. RFC1314 Web Address: Image Exchange Format http://sunsite.cnlab-switch.ch/ By: Katz & Cohen
The following fields should be used in a TIFF image file. Only the Basic Fields are mandatory; the others are optional (except that for MH and MR encoded files, the Group3Options Facsimile Field is mandatory). The optional fields have default values which are given in the TIFF specification. (Note that the TIFF reference [3] recommends not relying on the default values.)
Some fields contain one or more flag bits all stored as one value. In these cases, the bit labeled 0 is the least significant bit (i.e., Little Endian order). Where there is more than one suggested value for a tag, the possible values are separated by |.
Note that some fields (such as ImageLength or ImageWidth) can be of more than one type.
It would be useful to develop a TIFF viewer and editor which would allow one to read, add, and edit the fields in a TIFF file. Such an editor would display fields in sorted order and force the inclusion of all mandatory fields. Also, resolution and position should always be displayed or specified together with their units.
3.C.1. Basic Fields (Mandatory)
Basic Fields are those which are fundamental to the pixel architecture or visual characteristics of an image. The following Basic Fields should be included in a TIFF image file:
FIELD NAME (TAG in hex, TYPE) VALUE DESCRIPTION ------
BitsPerSample 1 Number of bits (0102, SHORT) per pixel (bi-level for now, but may allow more later)
Compression 4 Type of Compression (0103, SHORT) (could also be 1 = Uncompressed 1 or 3) 3 = G3 (MH or MR) 4 = G4 (MMR) Use 4 if possible
ImageLength
ImageWidth
Formatted and Indexed by: page 12 of 23 instructional media + magic, inc. RFC1314 Web Address: Image Exchange Format http://sunsite.cnlab-switch.ch/ By: Katz & Cohen
or LONG)
NewSubFileType 0 usually Flag bits indicating (00FE, LONG) bit 0: 1 if the kind of image. reduced (see the TIFF resolution of reference [3]) another image bit 1: 1 if single page of a multi-page image bit 2: 1 if image defines a transparency mask
Photometric- 0 for positive Interpretation image (0 imaged (0106, SHORT) as white, 1 as black) 1 means reverse black and white
RowsPerStrip
SamplesPerPixel 1 (since are Bi-level (0115, SHORT) images)
StripByteCounts count1, count2... Number of Bytes in (0117, SHORTs each strip of the or LONGs) images. (The Value is an offset which points to a series of counts, each of which is the same Type, LONG or SHORT. The Length is the same as the number of strips.)
StripOffsets off1, off2,... Pointers to the strips (0111, SHORTs of the image (remember, or LONGs) one strip per page). (The Value is an offset which points to a series of offsets,
Formatted and Indexed by: page 13 of 23 instructional media + magic, inc. RFC1314 Web Address: Image Exchange Format http://sunsite.cnlab-switch.ch/ By: Katz & Cohen
each of which points to the actual image data for the strip.)
ResolutionUnit 2 | 3 Units of Resolution (0128, SHORT) See Below, 3.C.6 2: Inches 3: Centimeters
XResolution See Below, 3.C.6 Resolution in the X (011A, RATIONAL) direction in pixels per ResolutionUnit (we suggest 400 dots per inch when possible)
YResolution See Below, 3.C.6 Resolution in the Y (011B, RATIONAL) direction in pixels per ResolutionUnit (we suggest 400 dots per inch when possible)
3.C.2. Informational Fields (Optional)
The following Informational Fields are optional. They provide useful information to a user. All Field values are ASCII strings.
NAME (TAG in hex) DESCRIPTION ------
Artist (013B) Person Who Created the Image
DateTime (0132) Date and Time of Image Creation
HostComputer (013C) Name of Computer Image was Created On
ImageDescription A Short Text Description (010E)
Make (010F) Manufacturer of Hardware (Scanner) Used
Model (0110) Model Number of Hardware (Scanner) Used
Software (0131) Software Package that Created the Image
3.C.3. Facsimile Fields (Optional, Mandatory for G3 Compression)
In addition to the above, the Facsimile Fields below should be used. The TIFF document recommends that they not be used for interchange between applications, but they are now in wide enough
Formatted and Indexed by: page 14 of 23 instructional media + magic, inc. RFC1314 Web Address: Image Exchange Format http://sunsite.cnlab-switch.ch/ By: Katz & Cohen
use for just that. These fields are optional and default to 0 (all bits off).
FIELD NAME (TAG in hex, TYPE) VALUE DESCRIPTION ------
Group3Options bit 0: 1 for Flag bits indicating (0124, LONG) 2-dimensional Options for G3 coding (i.e., MR with k > 1) bit 1: 1 if uncompressed mode MAY be used, 0 if uncompressed mode IS NOT used. bit 2: 1 if fill (As allowed by the G3 bits have been protocol, fill bits added may be added between each line of data and the EOL. Since fill bits are used to "byte-align" G3 image files, bit 2 should be set to 1 for these images.)
Group4Options bit 0: unused Flag bits indicating (0125, LONG) bit 1: 1 if Options for G4 uncompressed mode MAY be used, if this bit is 0 it means that uncompressed mode IS NOT used.
3.C.4. Storage and Retrieval Fields (Optional)
The following fields are optional and may be useful for document storage and retrieval.
Formatted and Indexed by: page 15 of 23 instructional media + magic, inc. RFC1314 Web Address: Image Exchange Format http://sunsite.cnlab-switch.ch/ By: Katz & Cohen
FIELD NAME (TAG in hex, TYPE) DESCRIPTION ------
DocumentName Name of the Document (010D, ASCII)
PageName Name of the Page (011D, ASCII)
PageNumber Page Number in a Multi-Page Document (0129, SHORTs) Two SHORT Values are specified, the first is the page number and the second is the total number of pages in the document. The first page is page 0. (NOTE: This does not necessarily correspond to page numbers which may be printed in the image.)
XPosition X Offset of the Left Side of (011E, RATIONAL) the Image, in ResolutionUnits
YPosition Y Offset of the Top of (011F, RATIONAL) the Image, in ResolutionUnits
3.C.5. TIFF-F Fields (NOT Recommended)
TIFF-F defines the following new fields for G3 (MH) encoded images. Since these fields are not defined in TIFF-B itself, their use is not recommended. However, since TIFF-F files may include these tags for image data which came from a G3 fax machine, readers should be prepared for them.
These three fields deal with corrupted image data which is due to the fact that G3 devices may not perform error correction on bad data.
FIELD NAME (TAG in hex, TYPE) DESCRIPTION ------
BadFaxLines Number of Bad fax scan lines (0146, SHORT or LONG) encountered during fax reception (but not necessarily in the file)
CleanFaxData 0 means no bad lines received (0147, SHORT) 1 means bad lines were regenerated
Formatted and Indexed by: page 16 of 23 instructional media + magic, inc. RFC1314 Web Address: Image Exchange Format http://sunsite.cnlab-switch.ch/ By: Katz & Cohen
by the receiving device 2 means bad lines were detected but not regenerated
ConsecutiveBadFaxLines The maximum number of consecutive (0148, SHORT or LONG) bad fax lines (but not necessarily in the file)
3.C.6. More on Representing Resolutions
The tags XResolution and YResolution are both RATIONALs, i.e., the ratio of two LONGS. G3 fax resolutions are actually specified in dots (or lines) per mm while G4 is in dots per inch (actually, dots per 25.4 mm).
For example, G3 horizontal resolution is defined to be 1728 dots per 215 mm which comes out to 80.4 dots per cm or about 203 dots per inch. It is frequently referred to as just 200 dpi. To avoid any possibility of problems due to round off error, this should be represented by having XResolution = 17280/215 and ResolutionUnit = 3 (cm). However when reading, 204/1 or even 200/1 with ResolutionUnit = 2 (inches) should be recognized as representing the same resolution.
For G4, on the other hand, the resolution 400 dots/inch should be represented by an XResolution of 400/1 and ResolutionUnit = 2.
The following table shows various ways of representing the standard resolutions in order of preference:
ResolutionUnit XResolution YResolution ------
G3 normal 3 17280/215 3850/100 3 80/1 3850/100 3 17280/215 385/10 3 80/1 385/10 2 2042/10 9779/100 2 204/1 98/1 2 200/1 100/1
G3 fine 3 17280/215 77/1 3 80/1 77/1 2 2042/10 19558/100 2 204/1 196/1 2 200/1 200/1
Formatted and Indexed by: page 17 of 23 instructional media + magic, inc. RFC1314 Web Address: Image Exchange Format http://sunsite.cnlab-switch.ch/ By: Katz & Cohen
G4 200 dpi 2 200/1 200/1
G4 300 dpi 2 300/1 300/1
Other 300 dpi 2 300/1 300/1
G4 400 dpi 2 400/1 400/1
600 dpi 2 600/1 600/1
It is suggested that Image readers be able to handle all of the above representations.
4. A Sample TIFF Image File
Below is a sample of what might be in a TIFF file for an MMR (G4) encoded single image which is about 100K bytes compressed at 400 dpi. A generic outline is given first, followed by a more detailed hex listing.
4.A. Sample File
Comments are to the right and are preceded by a semicolon. Note that tags must be sorted in order of the tag codes.
0:, IFDADDR:, and STRIP0: are addresses within the file and denote the number of bytes from the beginning of the file.
Header:
0: Byte Order= hex 4D4D ;first bytes of the file, from ;most significant bit to least ;significant (big endian) Version= 42 (hex 002A) ;Must be 42 First IFD= IFDADDR ;Address of first (and only) IFD
Image File Directory (the only one in this example):
IFDADDR:
IFD Entry Count= 24 ;(NOT A TAG) Count of ; Number of IFD Entries
NewSubFileType= 0 ImageWidth= 3400 ;8.5 inches at 400 dpi ImageLength= 4400 ;11 inches at 400 dpi BitsPerSample= 1 ;Bi-Level
Compression= 4 ;MMR Photometric-
Formatted and Indexed by: page 18 of 23 instructional media + magic, inc. RFC1314 Web Address: Image Exchange Format http://sunsite.cnlab-switch.ch/ By: Katz & Cohen
Interpretation= 0 DocumentName= "LAMap1" ImageDescription= "A map of Los Angeles" Make= "Fujitsu" Model= "M3093E" StripOffsets=
SamplesPerPixel= 1 ;Bi-Level RowsPerStrip= 4400 ;Entire image in 1 strip StripByteCounts=
XResolution= 400/1 YResolution= 400/1 XPosition= 0/1 ;position of left side of image YPosition= 0/1 ;position of top of image Group4Options= hex 00000002 ;bit 1 on means uncompressed ;mode MAY be used
ResolutionUnit= 2 ;Inches Software= "Xionics" DateTime= "1990:10:05 15:00:00" Artist= "Joe Pro" HostComputer= "Tardis.Isi.Edu"
Next IFD Pointer= hex 00000000 ;(NOT A TAG) Indicates no ; more IFDs in this file
Image Data:
[end of TIFF file]
In this example there is only one strip. Note that if there were more than one, the TIFF specification does not require them to be in any particular order. Strips may be given in any order and TIFF readers must use the StripOffsets to locate them.
Also, the TIFF document recommends not relying on the default values of the tags.
4.B. Detailed Hex Listing
Formatted and Indexed by: page 19 of 23 instructional media + magic, inc. RFC1314 Web Address: Image Exchange Format http://sunsite.cnlab-switch.ch/ By: Katz & Cohen
All offsets and values are represented by hex except for ASCII strings which are double quoted. Remember that Value Offsets must always be an even number since the value it points to must always be on a 16-bit word boundary.
Entries in the Name column are for reference and are not actually a part of the TIFF file.
Offset Name Value ------Header (first byte is Offset 0): 0000 Byte Order 4D4D 0002 Version 002A 0004 1st. IFD pointer 00000010
IFD (IFDADDR from above is 0010 here): 0010 Entry Count 0018 0012 NewSubFileType 00FE 0004 00000001 00000000 001E ImageWidth 0100 0004 00000001 00000D48 002A ImageLength 0101 0004 00000001 00001130 0036 BitsPerSample 0102 0003 00000001 00010000 0042 Compression 0103 0003 00000001 00040000 004E Photometric Interp. 0106 0003 00000001 00000000 005A DocumentName 010D 0002 00000007 00000136 0066 ImageDescription 010E 0002 00000015 0000013E 0072 Make 010F 0002 00000008 00000154 007E Model 0110 0002 00000007 0000015C 008A StripOffsets 0111 0004 00000001 000001A8 0096 SamplesPerPixel 0115 0003 00000001 00010000 00A2 RowsPerStrip 0116 0004 00000001 00001130 00AE StripByteCounts 0117 0004 00000001
Fields Offsets Point to: 0136 DocumentName "LAMap1" 013E ImageDescription "A map of Los Angeles"
0154 Make "Fujitsu" 015C Model "M3093E"
Formatted and Indexed by: page 20 of 23 instructional media + magic, inc. RFC1314 Web Address: Image Exchange Format http://sunsite.cnlab-switch.ch/ By: Katz & Cohen
0164 X,Y Resolution 00000190 00000001 016C X,Y Position 00000000 00000001 0174 Software "Xionics" 017C DateTime "1990:10:05 15:00:00" 0190 Artist "Joe Pro" 0198 HostComputer "Tardis.Isi.Edu"
Image Data (
[end of TIFF file]
NOTE: Since in this example there is only a single strip, there is only one count for StripByteCounts and one offset for StripOffsets. Thus, each of these only takes 4 bytes and will fit in the Value Offset instead of being pointed to.
5. Conclusions
Bitmapped images transferred within the Internet should be in the following format:
1. The file format should be TIFF-B with multi-page files supported. Images should be encoded as one TIFF strip per page.
2. Images should be compressed using MMR when possible. Images may also be MH or MR compressed or uncompressed. If MH or MR compression is used, scan lines should be "byte-aligned".
3. For maximum interoperability, image resolutions should either be 600, 400, or 300 dpi; or else be one of the standard Group 3 fax resolutions (98 or 196 dpi vertically and 204 dpi horizontally).
Note that this specification is self contained and an implementation should be possible without recourse to the TIFF references, and that only the specific TIFF documents cited are relevant to this specification. Updates to the TIFF documents do not change this specification.
Existing commercial off-the-shelf products are available which can handle images in the above format. ISI would be delighted to help those interested in assembling a system.
6. Acknowledgments
Formatted and Indexed by: page 21 of 23 instructional media + magic, inc. RFC1314 Web Address: Image Exchange Format http://sunsite.cnlab-switch.ch/ By: Katz & Cohen
Many contributions to this work were made by members of the IETF Network Fax Working Group especially by its chairman, Mark Needleman and by Clifford Lynch of the University of California Office of the President, Library Automation. Also, Kiyo Inaba of Ricoh Co. Ltd. made a number of helpful suggestions.
7. References
[1] Borenstein, N., and N. Freed, "Mechanisms for Specifying and Describing the Format of Internet Message Bodies", RFC in preparation.
[2] International Telegraph and Telephone Consultative Committee (CCITT), Red Book, October, 1984.
[3] Aldus Corp., Microsoft Corp., "Tag Image File Format Specification", Revision 5.0, Final, 1988.
[4] Cygnet Corporation, "The Spirit of TIFF Class F, 1990", available from Cygnet Technologies, 2560 9th., Suite 220, Berkeley, CA 94710, FAX: (415) 540-5835.
[5] Welch, T., "A Technique for High Performance Data Compression", IEEE Computer, Vol. 17, No. 6, Page 8, June 1984.
8. Security Considerations
While security issues are not directly addressed by this document, it is important to note that the file format described in this document is intended for the communications of files between systems and across networks. Thus the same precautions and cares should be applied to these files as would be to any files received from remote and possibly unknown systems.
9. Authors' Addresses
Formatted and Indexed by: page 22 of 23 instructional media + magic, inc. RFC1314 Web Address: Image Exchange Format http://sunsite.cnlab-switch.ch/ By: Katz & Cohen
Alan Katz USC Information Sciences Institute 4676 Admiralty Way #1100 Marina Del Rey, CA 90292-6695
Phone: 310-822-1511 Fax: 310-823-6714 EMail: [email protected]
Danny Cohen USC Information Sciences Institute 4676 Admiralty Way #1100 Marina Del Rey, CA 90292-6695
Phone: 310-822-1511 Fax: 310-823-6714 EMail: [email protected]
_
Formatted and Indexed by: page 23 of 23 instructional media + magic, inc. RFC1459 Web Address: Internet Relay Chat Protocol http://sunsite.cnlab-switch.ch/ By: Oikarinen & Reed
Network Working Group J. Oikarinen Request for Comments: 1459 D. Reed May 1993
Internet Relay Chat Protocol
Status of This Memo
This memo defines an Experimental Protocol for the Internet community. Discussion and suggestions for improvement are requested. Please refer to the current edition of the "IAB Official Protocol Standards" for the standardization state and status of this protocol. Distribution of this memo is unlimited.
Abstract
The IRC protocol was developed over the last 4 years since it was first implemented as a means for users on a BBS to chat amongst themselves. Now it supports a world-wide network of servers and clients, and is stringing to cope with growth. Over the past 2 years, the average number of users connected to the main IRC network has grown by a factor of 10.
The IRC protocol is a text-based protocol, with the simplest client being any socket program capable of connecting to the server.
Table of Contents
1. INTRODUCTION ...... 4 1.1 Servers ...... 4 1.2 Clients ...... 5 1.2.1 Operators ...... 5 1.3 Channels ...... 5 1.3.1 Channel Operators ...... 6 2. THE IRC SPECIFICATION ...... 7 2.1 Overview ...... 7 2.2 Character codes ...... 7 2.3 Messages ...... 7 2.3.1 Message format in 'pseudo' BNF ...... 8 2.4 Numeric replies ...... 10 3. IRC Concepts ...... 10 3.1 One-to-one communication ...... 10 3.2 One-to-many ...... 11 3.2.1 To a list ...... 11 3.2.2 To a group (channel) ...... 11 3.2.3 To a host/server mask ...... 12 3.3 One to all ...... 12
Formatted and Indexed by: page 1 of 65 instructional media + magic, inc. RFC1459 Web Address: Internet Relay Chat Protocol http://sunsite.cnlab-switch.ch/ By: Oikarinen & Reed
3.3.1 Client to Client ...... 12 3.3.2 Clients to Server ...... 12 3.3.3 Server to Server ...... 12 4. MESSAGE DETAILS ...... 13 4.1 Connection Registration ...... 13 4.1.1 Password message ...... 14 4.1.2 Nickname message ...... 14 4.1.3 User message ...... 15 4.1.4 Server message ...... 16 4.1.5 Operator message ...... 17 4.1.6 Quit message ...... 17 4.1.7 Server Quit message ...... 18 4.2 Channel operations ...... 19 4.2.1 Join message ...... 19 4.2.2 Part message ...... 20 4.2.3 Mode message ...... 21 4.2.3.1 Channel modes ...... 21 4.2.3.2 User modes ...... 22 4.2.4 Topic message ...... 23 4.2.5 Names message ...... 24 4.2.6 List message ...... 24 4.2.7 Invite message ...... 25 4.2.8 Kick message ...... 25 4.3 Server queries and commands ...... 26 4.3.1 Version message ...... 26 4.3.2 Stats message ...... 27 4.3.3 Links message ...... 28 4.3.4 Time message ...... 29 4.3.5 Connect message ...... 29 4.3.6 Trace message ...... 30 4.3.7 Admin message ...... 31 4.3.8 Info message ...... 31 4.4 Sending messages ...... 32 4.4.1 Private messages ...... 32 4.4.2 Notice messages ...... 33 4.5 User-based queries ...... 33 4.5.1 Who query ...... 33 4.5.2 Whois query ...... 34 4.5.3 Whowas message ...... 35 4.6 Miscellaneous messages ...... 35 4.6.1 Kill message ...... 36 4.6.2 Ping message ...... 37 4.6.3 Pong message ...... 37 4.6.4 Error message ...... 38 5. OPTIONAL MESSAGES ...... 38 5.1 Away message ...... 38 5.2 Rehash command ...... 39 5.3 Restart command ...... 39
Formatted and Indexed by: page 2 of 65 instructional media + magic, inc. RFC1459 Web Address: Internet Relay Chat Protocol http://sunsite.cnlab-switch.ch/ By: Oikarinen & Reed
5.4 Summon message ...... 40 5.5 Users message ...... 40 5.6 Operwall command ...... 41 5.7 Userhost message ...... 42 5.8 Ison message ...... 42 6. REPLIES ...... 43 6.1 Error Replies ...... 43 6.2 Command responses ...... 48 6.3 Reserved numerics ...... 56 7. Client and server authentication ...... 56 8. Current Implementations Details ...... 56 8.1 Network protocol: TCP ...... 57 8.1.1 Support of Unix sockets ...... 57 8.2 Command Parsing ...... 57 8.3 Message delivery ...... 57 8.4 Connection 'Liveness' ...... 58 8.5 Establishing a server-client connection ...... 58 8.6 Establishing a server-server connection ...... 58 8.6.1 State information exchange when connecting ...... 59 8.7 Terminating server-client connections ...... 59 8.8 Terminating server-server connections ...... 59 8.9 Tracking nickname changes ...... 60 8.10 Flood control of clients ...... 60 8.11 Non-blocking lookups ...... 61 8.11.1 Hostname (DNS) lookups ...... 61 8.11.2 Username (Ident) lookups ...... 61 8.12 Configuration file ...... 61 8.12.1 Allowing clients to connect ...... 62 8.12.2 Operators ...... 62 8.12.3 Allowing servers to connect ...... 62 8.12.4 Administrivia ...... 63 8.13 Channel membership ...... 63 9. Current problems ...... 63 9.1 Scalability ...... 63 9.2 Labels ...... 63 9.2.1 Nicknames ...... 63 9.2.2 Channels ...... 64 9.2.3 Servers ...... 64 9.3 Algorithms ...... 64 10. Support and availability ...... 64 11. Security Considerations ...... 65 12. Authors' Addresses ...... 65
Formatted and Indexed by: page 3 of 65 instructional media + magic, inc. RFC1459 Web Address: Internet Relay Chat Protocol http://sunsite.cnlab-switch.ch/ By: Oikarinen & Reed
1. INTRODUCTION
The IRC (Internet Relay Chat) protocol has been designed over a number of years for use with text based conferencing. This document describes the current IRC protocol.
The IRC protocol has been developed on systems using the TCP/IP network protocol, although there is no requirement that this remain the only sphere in which it operates.
IRC itself is a teleconferencing system, which (through the use of the client-server model) is well-suited to running on many machines in a distributed fashion. A typical setup involves a single process (the server) forming a central point for clients (or other servers) to connect to, performing the required message delivery/multiplexing and other functions.
1.1 Servers
The server forms the backbone of IRC, providing a point to which clients may connect to to talk to each other, and a point for other servers to connect to, forming an IRC network. The only network configuration allowed for IRC servers is that of a spanning tree [see Fig. 1] where each server acts as a central node for the rest of the net it sees.
[ Server 15 ] [ Server 13 ] [ Server 14] / \ / / \ / [ Server 11 ] ------[ Server 1 ] [ Server 12] / \ / / \ / [ Server 2 ] [ Server 3 ] / \ \ / \ \ [ Server 4 ] [ Server 5 ] [ Server 6 ] / | \ / / | \ / / | \____ / / | \ / [ Server 7 ] [ Server 8 ] [ Server 9 ] [ Server 10 ]
: [ etc. ] :
[ Fig. 1. Format of IRC server network ]
Formatted and Indexed by: page 4 of 65 instructional media + magic, inc. RFC1459 Web Address: Internet Relay Chat Protocol http://sunsite.cnlab-switch.ch/ By: Oikarinen & Reed
1.2 Clients
A client is anything connecting to a server that is not another server. Each client is distinguished from other clients by a unique nickname having a maximum length of nine (9) characters. See the protocol grammar rules for what may and may not be used in a nickname. In addition to the nickname, all servers must have the following information about all clients: the real name of the host that the client is running on, the username of the client on that host, and the server to which the client is connected.
1.2.1 Operators
To allow a reasonable amount of order to be kept within the IRC network, a special class of clients (operators) is allowed to perform general maintenance functions on the network. Although the powers granted to an operator can be considered as 'dangerous', they are nonetheless required. Operators should be able to perform basic network tasks such as disconnecting and reconnecting servers as needed to prevent long-term use of bad network routing. In recognition of this need, the protocol discussed herein provides for operators only to be able to perform such functions. See sections 4.1.7 (SQUIT) and 4.3.5 (CONNECT).
A more controversial power of operators is the ability to remove a user from the connected network by 'force', i.e. operators are able to close the connection between any client and server. The justification for this is delicate since its abuse is both destructive and annoying. For further details on this type of action, see section 4.6.1 (KILL).
1.3 Channels
A channel is a named group of one or more clients which will all receive messages addressed to that channel. The channel is created implicitly when the first client joins it, and the channel ceases to exist when the last client leaves it. While channel exists, any client can reference the channel using the name of the channel.
Channels names are strings (beginning with a '&' or '#' character) of length up to 200 characters. Apart from the the requirement that the first character being either '&' or '#'; the only restriction on a channel name is that it may not contain any spaces (' '), a control G (^G or ASCII 7), or a comma (',' which is used as a list item separator by the protocol).
There are two types of channels allowed by this protocol. One is a distributed channel which is known to all the servers that are
Formatted and Indexed by: page 5 of 65 instructional media + magic, inc. RFC1459 Web Address: Internet Relay Chat Protocol http://sunsite.cnlab-switch.ch/ By: Oikarinen & Reed
connected to the network. These channels are marked by the first character being a only clients on the server where it exists may join it. These are distinguished by a leading '&' character. On top of these two types, there are the various channel modes available to alter the characteristics of individual channels. See section 4.2.3 (MODE command) for more details on this.
To create a new channel or become part of an existing channel, a user is required to JOIN the channel. If the channel doesn't exist prior to joining, the channel is created and the creating user becomes a channel operator. If the channel already exists, whether or not your request to JOIN that channel is honoured depends on the current modes of the channel. For example, if the channel is invite-only, (+i), then you may only join if invited. As part of the protocol, a user may be a part of several channels at once, but a limit of ten (10) channels is recommended as being ample for both experienced and novice users. See section 8.13 for more information on this.
If the IRC network becomes disjoint because of a split between two servers, the channel on each side is only composed of those clients which are connected to servers on the respective sides of the split, possibly ceasing to exist on one side of the split. When the split is healed, the connecting servers announce to each other who they think is in each channel and the mode of that channel. If the channel exists on both sides, the JOINs and MODEs are interpreted in an inclusive manner so that both sides of the new connection will agree about which clients are in the channel and what modes the channel has.
1.3.1 Channel Operators
The channel operator (also referred to as a "chop" or "chanop") on a given channel is considered to 'own' that channel. In recognition of this status, channel operators are endowed with certain powers which enable them to keep control and some sort of sanity in their channel. As an owner of a channel, a channel operator is not required to have reasons for their actions, although if their actions are generally antisocial or otherwise abusive, it might be reasonable to ask an IRC operator to intervene, or for the usersjust leave and go elsewhere and form their own channel.
The commands which may only be used by channel operators are:
KICK - Eject a client from the channel MODE - Change the channel's mode INVITE - Invite a client to an invite-only channel (mode +i) TOPIC - Change the channel topic in a mode +t channel
Formatted and Indexed by: page 6 of 65 instructional media + magic, inc. RFC1459 Web Address: Internet Relay Chat Protocol http://sunsite.cnlab-switch.ch/ By: Oikarinen & Reed
A channel operator is identified by the '@' symbol next to their nickname whenever it is associated with a channel (ie replies to the NAMES, WHO and WHOIS commands).
2. The IRC Specification
2.1 Overview
The protocol as described herein is for use both with server to server and client to server connections. There are, however, more restrictions on client connections (which are considered to be untrustworthy) than on server connections.
2.2 Character codes
No specific character set is specified. The protocol is based on a a set of codes which are composed of eight (8) bits, making up an octet. Each message may be composed of any number of these octets; however, some octet values are used for control codes which act as message delimiters.
Regardless of being an 8-bit protocol, the delimiters and keywords are such that protocol is mostly usable from USASCII terminal and a telnet connection.
Because of IRC's scandanavian origin, the characters {}| are considered to be the lower case equivalents of the characters []\, respectively. This is a critical issue when determining the equivalence of two nicknames.
2.3 Messages
Servers and clients send eachother messages which may or may not generate a reply. If the message contains a valid command, as described in later sections, the client should expect a reply as specified but it is not advised to wait forever for the reply; client to server and server to server communication is essentially asynchronous in nature.
Each IRC message may consist of up to three main parts: the prefix (optional), the command, and the command parameters (of which there may be up to 15). The prefix, command, and all parameters are separated by one (or more) ASCII space character(s) (0x20).
The presence of a prefix is indicated with a single leading ASCII colon character (':', 0x3b), which must be the first character of the message itself. There must be no gap (whitespace) between the colon and the prefix. The prefix is used by servers to indicate the true
Formatted and Indexed by: page 7 of 65 instructional media + magic, inc. RFC1459 Web Address: Internet Relay Chat Protocol http://sunsite.cnlab-switch.ch/ By: Oikarinen & Reed
origin of the message. If the prefix is missing from the message, it is assumed to have originated from the connection from which it was received. Clients should not use prefix when sending a message from themselves; if they use a prefix, the only valid prefix is the registered nickname associated with the client. If the source identified by the prefix cannot be found from the server's internal database, or if the source is registered from a different link than from which the message arrived, the server must ignore the message silently.
The command must either be a valid IRC command or a three (3) digit number represented in ASCII text.
IRC messages are always lines of characters terminated with a CR-LF (Carriage Return - Line Feed) pair, and these messages shall not exceed 512 characters in length, counting all characters including the trailing CR-LF. Thus, there are 510 characters maximum allowed for the command and its parameters. There is no provision for continuation message lines. See section 7 for more details about current implementations.
2.3.1 Message format in 'pseudo' BNF
The protocol messages must be extracted from the contiguous stream of octets. The current solution is to designate two characters, CR and LF, as message separators. Empty messages are silently ignored, which permits use of the sequence CR-LF between messages without extra problems.
The extracted message is parsed into the components
The BNF representation for this is:
Formatted and Indexed by: page 8 of 65 instructional media + magic, inc. RFC1459 Web Address: Internet Relay Chat Protocol http://sunsite.cnlab-switch.ch / By: Oikarinen & Reed
NOTES:
1)
2) After extracting the parameter list, all parameters are equal, whether matched by
3) The fact that CR and LF cannot appear in parameter strings is just artifact of the message framing. This might change later.
4) The NUL character is not special in message framing, and basically could end up inside a parameter, but as it would cause extra complexities in normal C string handling. Therefore NUL is not allowed within messages.
5) The last parameter may be an empty string.
6) Use of the extended prefix (['!'
Most protocol messages specify additional semantics and syntax for the extracted parameter strings dictated by their position in the list. For example, many server commands will assume that the first parameter after the command is the list of targets, which can be described with:
Other parameter syntaxes are:
Formatted and Indexed by: page 9 of 65 instructional media + magic, inc. RFC1459 Web Address: Internet Relay Chat Protocol http://sunsite.cnlab-switch.ch/ By: Oikarinen & Reed
2.4 Numeric replies
Most of the messages sent to the server generate a reply of some sort. The most common reply is the numeric reply, used for both errors and normal replies. The numeric reply must be sent as one message consisting of the sender prefix, the three digit numeric, and the target of the reply. A numeric reply is not allowed to originate from a client; any such messages received by a server are silently dropped. In all other respects, a numeric reply is just like a normal message, except that the keyword is made up of 3 numeric digits rather than a string of letters. A list of different replies is supplied in section 6.
3. IRC Concepts.
This section is devoted to describing the actual concepts behind the organization of the IRC protocol and how the current implementations deliver different classes of messages.
1--\ A D---4 2--/ \ / B----C / \ 3 E
Servers: A, B, C, D, E Clients: 1, 2, 3, 4
[ Fig. 2. Sample small IRC network ]
3.1 One-to-one communication
Communication on a one-to-one basis is usually only performed by clients, since most server-server traffic is not a result of servers talking only to each other. To provide a secure means for clients to talk to each other, it is required that all servers be able to send a message in exactly one direction along the spanning tree in order to reach any client. The path of a message being delivered is the shortest path between any two points on the spanning tree.
The following examples all refer to Figure 2 above.
Formatted and Indexed by: page 10 of 65 instructional media + magic, inc. RFC1459 Web Address: Internet Relay Chat Protocol http://sunsite.cnlab-switch.ch/ By: Oikarinen & Reed
Example 1: A message between clients 1 and 2 is only seen by server A, which sends it straight to client 2.
Example 2: A message between clients 1 and 3 is seen by servers A & B, and client 3. No other clients or servers are allowed see the message.
Example 3: A message between clients 2 and 4 is seen by servers A, B, C & D and client 4 only.
3.2 One-to-many
The main goal of IRC is to provide a forum which allows easy and efficient conferencing (one to many conversations). IRC offers several means to achieve this, each serving its own purpose.
3.2.1 To a list
The least efficient style of one-to-many conversation is through clients talking to a 'list' of users. How this is done is almost self explanatory: the client gives a list of destinations to which the message is to be delivered and the server breaks it up and dispatches a separate copy of the message to each given destination. This isn't as efficient as using a group since the destination list is broken up and the dispatch sent without checking to make sure duplicates aren't sent down each path.
3.2.2 To a group (channel)
In IRC the channel has a role equivalent to that of the multicast group; their existence is dynamic (coming and going as people join and leave channels) and the actual conversation carried out on a channel is only sent to servers which are supporting users on a given channel. If there are multiple users on a server in the same channel, the message text is sent only once to that server and then sent to each client on the channel. This action is then repeated for each client-server combination until the original message has fanned out and reached each member of the channel.
The following examples all refer to Figure 2.
Example 4: Any channel with 1 client in it. Messages to the channel go to the server and then nowhere else.
Formatted and Indexed by: page 11 of 65 instructional media + magic, inc. RFC1459 Web Address: Internet Relay Chat Protocol http://sunsite.cnlab-switch.ch/ By: Oikarinen & Reed
Example 5: 2 clients in a channel. All messages traverse a path as if they were private messages between the two clients outside a channel.
Example 6: Clients 1, 2 and 3 in a channel. All messages to the channel are sent to all clients and only those servers which must be traversed by the message if it were a private message to a single client. If client 1 sends a message, it goes back to client 2 and then via server B to client 3.
3.2.3 To a host/server mask
To provide IRC operators with some mechanism to send messages to a large body of related users, host and server mask messages are provided. These messages are sent to users whose host or server information match that of the mask. The messages are only sent to locations where users are, in a fashion similar to that of channels.
3.3 One-to-all
The one-to-all type of message is better described as a broadcast message, sent to all clients or servers or both. On a large network of users and servers, a single message can result in a lot of traffic being sent over the network in an effort to reach all of the desired destinations.
For some messages, there is no option but to broadcast it to all servers so that the state information held by each server is reasonably consistent between servers.
3.3.1 Client-to-Client
There is no class of message which, from a single message, results in a message being sent to every other client.
3.3.2 Client-to-Server
Most of the commands which result in a change of state information (such as channel membership, channel mode, user status, etc) must be sent to all servers by default, and this distribution may not be changed by the client.
3.3.3 Server-to-Server.
While most messages between servers are distributed to all 'other' servers, this is only required for any message that affects either a user, channel or server. Since these are the basic items found in
Formatted and Indexed by: page 12 of 65 instructional media + magic, inc. RFC1459 Web Address: Internet Relay Chat Protocol http://sunsite.cnlab-switch.ch/ By: Oikarinen & Reed
IRC, nearly all messages originating from a server are broadcast to all other connected servers.
4. Message details
On the following pages are descriptions of each message recognized by the IRC server and client. All commands described in this section must be implemented by any server for this protocol.
Where the reply ERR_NOSUCHSERVER is listed, it means that the
The server to which a client is connected is required to parse the complete message, returning any appropriate errors. If the server encounters a fatal error while parsing a message, an error must be sent back to the client and the parsing terminated. A fatal error may be considered to be incorrect command, a destination which is otherwise unknown to the server (server, nick or channel names fit this category), not enough parameters or incorrect privileges.
If a full set of parameters is presented, then each must be checked for validity and appropriate responses sent back to the client. In the case of messages which use parameter lists using the comma as an item separator, a reply must be sent for each item.
In the examples below, some messages appear using the full format:
:Name COMMAND parameter list
Such examples represent a message from "Name" in transit between servers, where it is essential to include the name of the original sender of the message so remote servers may send back a reply along the correct path.
4.1 Connection Registration
The commands described here are used to register a connection with an IRC server as either a user or a server as well as correctly disconnect.
A "PASS" command is not required for either client or server connection to be registered, but it must precede the server message or the latter of the NICK/USER combination. It is strongly recommended that all server connections have a password in order to give some level of security to the actual connections. The recommended order for a client to register is as follows:
Formatted and Indexed by: page 13 of 65 instructional media + magic, inc. RFC1459 Web Address: Internet Relay Chat Protocol http://sunsite.cnlab-switch.ch/ By: Oikarinen & Reed
1. Pass message 2. Nick message 3. User message
4.1.1 Password message
Command: PASS Parameters:
The PASS command is used to set a 'connection password'. The password can and must be set before any attempt to register the connection is made. Currently this requires that clients send a PASS command before sending the NICK/USER combination and servers *must* send a PASS command before any SERVER command. The password supplied must match the one contained in the C/N lines (for servers) or I lines (for clients). It is possible to send multiple PASS commands before registering but only the last one sent is used for verification and it may not be changed once registered. Numeric Replies:
ERR_NEEDMOREPARAMS ERR_ALREADYREGISTRED
Example:
PASS secretpasswordhere
4.1.2 Nick message
Command: NICK Parameters:
NICK message is used to give user a nickname or change the previous one. The
If a NICK message arrives at a server which already knows about an identical nickname for another client, a nickname collision occurs. As a result of a nickname collision, all instances of the nickname are removed from the server's database, and a KILL command is issued to remove the nickname from all other server's database. If the NICK message causing the collision was a nickname change, then the original (old) nick must be removed as well.
If the server recieves an identical NICK from a client which is directly connected, it may issue an ERR_NICKCOLLISION to the local client, drop the NICK command, and not generate any kills.
Formatted and Indexed by: page 14 of 65 instructional media + magic, inc. RFC1459 Web Address: Internet Relay Chat Protocol http://sunsite.cnlab-switch.ch/ By: Oikarinen & Reed
Numeric Replies:
ERR_NONICKNAMEGIVEN ERR_ERRONEUSNICKNAME ERR_NICKNAMEINUSE ERR_NICKCOLLISION
Example:
NICK Wiz ; Introducing new nick "Wiz".
:WiZ NICK Kilroy ; WiZ changed his nickname to Kilroy.
4.1.3 User message
Command: USER Parameters:
The USER message is used at the beginning of connection to specify the username, hostname, servername and realname of s new user. It is also used in communication between servers to indicate new user arriving on IRC, since only after both USER and NICK have been received from a client does a user become registered.
Between servers USER must to be prefixed with client's NICKname. Note that hostname and servername are normally ignored by the IRC server when the USER command comes from a directly connected client (for security reasons), but they are used in server to server communication. This means that a NICK must always be sent to a remote server when a new user is being introduced to the rest of the network before the accompanying USER is sent.
It must be noted that realname parameter must be the last parameter, because it may contain space characters and must be prefixed with a colon (':') to make sure this is recognised as such.
Since it is easy for a client to lie about its username by relying solely on the USER message, the use of an "Identity Server" is recommended. If the host which a user connects from has such a server enabled the username is set to that as in the reply from the "Identity Server".
Numeric Replies:
ERR_NEEDMOREPARAMS ERR_ALREADYREGISTRED
Examples:
USER guest tolmoon tolsun :Ronnie Reagan
Formatted and Indexed by: page 15 of 65 instructional media + magic, inc. RFC1459 Web Address: Internet Relay Chat Protocol http://sunsite.cnlab-switch.ch/ By: Oikarinen & Reed
; User registering themselves with a username of "guest" and real name "Ronnie Reagan".
:testnick USER guest tolmoon tolsun :Ronnie Reagan ; message between servers with the nickname for which the USER command belongs to
4.1.4 Server message
Command: SERVER Parameters:
The server message is used to tell a server that the other end of a new connection is a server. This message is also used to pass server data over whole net. When a new server is connected to net, information about it be broadcast to the whole network.
The SERVER message must only be accepted from either (a) a connection which is yet to be registered and is attempting to register as a server, or (b) an existing connection to another server, in which case the SERVER message is introducing a new server behind that server.
Most errors that occur with the receipt of a SERVER command result in the connection being terminated by the destination host (target SERVER). Error replies are usually sent using the "ERROR" command rather than the numeric since the ERROR command has several useful properties which make it useful here.
If a SERVER message is parsed and attempts to introduce a server which is already known to the receiving server, the connection from which that message must be closed (following the correct procedures), since a duplicate route to a server has formed and the acyclic nature of the IRC tree broken.
Numeric Replies:
ERR_ALREADYREGISTRED
Example:
Formatted and Indexed by: page 16 of 65 instructional media + magic, inc. RFC1459 Web Address: Internet Relay Chat Protocol http://sunsite.cnlab-switch.ch/ By: Oikarinen & Reed
SERVER test.oulu.fi 1 :[tolsun.oulu.fi] Experimental server ; New server test.oulu.fi introducing itself and attempting to register. The name in []'s is the hostname for the host running test.oulu.fi.
:tolsun.oulu.fi SERVER csd.bu.edu 5 :BU Central Server ; Server tolsun.oulu.fi is our uplink for csd.bu.edu which is 5 hops away.
4.1.5 Oper
Command: OPER Parameters:
OPER message is used by a normal user to obtain operator privileges. The combination of
If the client sending the OPER command supplies the correct password for the given user, the server then informs the rest of the network of the new operator by issuing a "MODE +o" for the clients nickname.
The OPER message is client-server only.
Numeric Replies:
ERR_NEEDMOREPARAMS RPL_YOUREOPER ERR_NOOPERHOST ERR_PASSWDMISMATCH
Example:
OPER foo bar ; Attempt to register as an operator using a username of "foo" and "bar" as the password.
4.1.6 Quit
Command: QUIT Parameters: [
A client session is ended with a quit message. The server must close the connection to a client which sends a QUIT message. If a "Quit Message" is given, this will be sent instead of the default message, the nickname.
When netsplits (disconnecting of two servers) occur, the quit message
Formatted and Indexed by: page 17 of 65 instructional media + magic, inc. RFC1459 Web Address: Internet Relay Chat Protocol http://sunsite.cnlab-switch.ch/ By: Oikarinen & Reed
is composed of the names of two servers involved, separated by a space. The first name is that of the server which is still connected and the second name is that of the server that has become disconnected.
If, for some other reason, a client connection is closed without the client issuing a QUIT command (e.g. client dies and EOF occurs on socket), the server is required to fill in the quit message with some sort of message reflecting the nature of the event which caused it to happen.
Numeric Replies:
None.
Examples:
QUIT :Gone to have lunch ; Preferred message format.
4.1.7 Server quit message
Command: SQUIT Parameters:
The SQUIT message is needed to tell about quitting or dead servers. If a server wishes to break the connection to another server it must send a SQUIT message to the other server, using the the name of the other server as the server parameter, which then closes its connection to the quitting server.
This command is also available operators to help keep a network of IRC servers connected in an orderly fashion. Operators may also issue an SQUIT message for a remote server connection. In this case, the SQUIT must be parsed by each server inbetween the operator and the remote server, updating the view of the network held by each server as explained below.
The
Both of the servers which are on either side of the connection being closed are required to to send out a SQUIT message (to all its other server connections) for all other servers which are considered to be behind that link.
Formatted and Indexed by: page 18 of 65 instructional media + magic, inc. RFC1459 Web Address: Internet Relay Chat Protocol http://sunsite.cnlab-switch.ch/ By: Oikarinen & Reed
Similarly, a QUIT message must be sent to the other connected servers rest of the network on behalf of all clients behind that link. In addition to this, all channel members of a channel which lost a member due to the split must be sent a QUIT message.
If a server connection is terminated prematurely (e.g. the server on the other end of the link died), the server which detects this disconnection is required to inform the rest of the network that the connection has closed and fill in the comment field with something appropriate.
Numeric replies:
ERR_NOPRIVILEGES ERR_NOSUCHSERVER
Example:
SQUIT tolsun.oulu.fi :Bad Link ? ; the server link tolson.oulu.fi has been terminated because of "Bad Link".
:Trillian SQUIT cm22.eng.umd.edu :Server out of control ; message from Trillian to disconnect "cm22.eng.umd.edu" from the net because "Server out of control".
4.2 Channel operations
This group of messages is concerned with manipulating channels, their properties (channel modes), and their contents (typically clients). In implementing these, a number of race conditions are inevitable when clients at opposing ends of a network send commands which will ultimately clash. It is also required that servers keep a nickname history to ensure that wherever a
4.2.1 Join message
Command: JOIN Parameters:
The JOIN command is used by client to start listening a specific channel. Whether or not a client is allowed to join a channel is checked only by the server the client is connected to; all other servers automatically add the user to the channel when it is received from other servers. The conditions which affect this are as follows:
1. the user must be invited if the channel is invite-only;
Formatted and Indexed by: page 19 of 65 instructional media + magic, inc. RFC1459 Web Address: Internet Relay Chat Protocol http://sunsite.cnlab-switch.ch/ By: Oikarinen & Reed
2. the user's nick/username/hostname must not match any active bans;
3. the correct key (password) must be given if it is set.
These are discussed in more detail under the MODE command (see section 4.2.3 for more details).
Once a user has joined a channel, they receive notice about all commands their server receives which affect the channel. This includes MODE, KICK, PART, QUIT and of course PRIVMSG/NOTICE. The JOIN command needs to be broadcast to all servers so that each server knows where to find the users who are on the channel. This allows optimal delivery of PRIVMSG/NOTICE messages to the channel.
If a JOIN is successful, the user is then sent the channel's topic (using RPL_TOPIC) and the list of users who are on the channel (using RPL_NAMREPLY), which must include the user joining.
Numeric Replies:
ERR_NEEDMOREPARAMS ERR_BANNEDFROMCHAN ERR_INVITEONLYCHAN ERR_BADCHANNELKEY ERR_CHANNELISFULL ERR_BADCHANMASK ERR_NOSUCHCHANNEL ERR_TOOMANYCHANNELS RPL_TOPIC
Examples:
JOIN #foobar ; join channel #foobar.
JOIN &foo fubar ; join channel &foo using key "fubar".
JOIN #foo,&bar fubar ; join channel #foo using key "fubar" and &bar using no key.
JOIN #foo,#bar fubar,foobar ; join channel #foo using key "fubar". and channel #bar using key "foobar".
JOIN #foo,#bar ; join channels #foo and #bar.
:WiZ JOIN #Twilight_zone ; JOIN message from WiZ
4.2.2 Part message
Command: PART Parameters:
Formatted and Indexed by: page 20 of 65 instructional media + magic, inc. RFC1459 Web Address: Internet Relay Chat Protocol http://sunsite.cnlab-switch.ch/ By: Oikarinen & Reed
The PART message causes the client sending the message to be removed from the list of active users for all given channels listed in the parameter string.
Numeric Replies:
ERR_NEEDMOREPARAMS ERR_NOSUCHCHANNEL ERR_NOTONCHANNEL
Examples:
PART #twilight_zone ; leave channel "#twilight_zone"
PART #oz-ops,&group5 ; leave both channels "&group5" and "#oz-ops".
4.2.3 Mode message
Command: MODE
The MODE command is a dual-purpose command in IRC. It allows both usernames and channels to have their mode changed. The rationale for this choice is that one day nicknames will be obsolete and the equivalent property will be the channel.
When parsing MODE messages, it is recommended that the entire message be parsed first and then the changes which resulted then passed on.
4.2.3.1 Channel modes
Parameters:
The MODE command is provided so that channel operators may change the characteristics of `their' channel. It is also required that servers be able to change channel modes so that channel operators may be created.
The various modes available for channels are as follows:
o - give/take channel operator privileges; p - private channel flag; s - secret channel flag; i - invite-only channel flag; t - topic settable by channel operator only flag; n - no messages to channel from clients on the outside; m - moderated channel; l - set the user limit to channel;
Formatted and Indexed by: page 21 of 65 instructional media + magic, inc. RFC1459 Web Address: Internet Relay Chat Protocol http://sunsite.cnlab-switch.ch/ By: Oikarinen & Reed
b - set a ban mask to keep users out; v - give/take the ability to speak on a moderated channel; k - set a channel key (password).
When using the 'o' and 'b' options, a restriction on a total of three per mode command has been imposed. That is, any combination of 'o' and
4.2.3.2 User modes
Parameters:
The user MODEs are typically changes which affect either how the client is seen by others or what 'extra' messages the client is sent. A user MODE command may only be accepted if both the sender of the message and the nickname given as a parameter are both the same.
The available modes are as follows:
i - marks a users as invisible; s - marks a user for receipt of server notices; w - user receives wallops; o - operator flag.
Additional modes may be available later on.
If a user attempts to make themselves an operator using the "+o" flag, the attempt should be ignored. There is no restriction, however, on anyone `deopping' themselves (using "-o"). Numeric Replies:
ERR_NEEDMOREPARAMS RPL_CHANNELMODEIS ERR_CHANOPRIVSNEEDED ERR_NOSUCHNICK ERR_NOTONCHANNEL ERR_KEYSET RPL_BANLIST RPL_ENDOFBANLIST ERR_UNKNOWNMODE ERR_NOSUCHCHANNEL
ERR_USERSDONTMATCH RPL_UMODEIS ERR_UMODEUNKNOWNFLAG
Examples:
Use of Channel Modes:
MODE #Finnish +im ; Makes #Finnish channel moderated and 'invite-only'.
MODE #Finnish +o Kilroy ; Gives 'chanop' privileges to Kilroy on
Formatted and Indexed by: page 22 of 65 instructional media + magic, inc. RFC1459 Web Address: Internet Relay Chat Protocol http://sunsite.cnlab-switch.ch/ By: Oikarinen & Reed
channel #Finnish.
MODE #Finnish +v Wiz ; Allow WiZ to speak on #Finnish.
MODE #Fins -s ; Removes 'secret' flag from channel #Fins.
MODE #42 +k oulu ; Set the channel key to "oulu".
MODE #eu-opers +l 10 ; Set the limit for the number of users on channel to 10.
MODE &oulu +b ; list ban masks set for channel.
MODE &oulu +b *!*@* ; prevent all users from joining.
MODE &oulu +b *!*@*.edu ; prevent any user from a hostname matching *.edu from joining.
Use of user Modes:
:MODE WiZ -w ; turns reception of WALLOPS messages off for WiZ.
:Angel MODE Angel +i ; Message from Angel to make themselves invisible.
MODE WiZ -o ; WiZ 'deopping' (removing operator status). The plain reverse of this command ("MODE WiZ +o") must not be allowed from users since would bypass the OPER command.
4.2.4 Topic message
Command: TOPIC Parameters:
The TOPIC message is used to change or view the topic of a channel. The topic for channel
Numeric Replies:
ERR_NEEDMOREPARAMS ERR_NOTONCHANNEL RPL_NOTOPIC RPL_TOPIC ERR_CHANOPRIVSNEEDED
Formatted and Indexed by: page 23 of 65 instructional media + magic, inc. RFC1459 Web Address: Internet Relay Chat Protocol http://sunsite.cnlab-switch.ch/ By: Oikarinen & Reed
Examples:
:Wiz TOPIC #test :New topic ;User Wiz setting the topic.
TOPIC #test :another topic ;set the topic on #test to "another topic".
TOPIC #test ; check the topic for #test.
4.2.5 Names message
Command: NAMES Parameters: [
By using the NAMES command, a user can list all nicknames that are visible to them on any channel that they can see. Channel names which they can see are those which aren't private (+p) or secret (+s) or those which they are actually on. The
If no
Numerics:
RPL_NAMREPLY RPL_ENDOFNAMES
Examples:
NAMES #twilight_zone,#42 ; list visible users on #twilight_zone and #42 if the channels are visible to you.
NAMES ; list all visible channels and users
4.2.6 List message
Command: LIST Parameters: [
The list message is used to list channels and their topics. If the
Formatted and Indexed by: page 24 of 65 instructional media + magic, inc. RFC1459 Web Address: Internet Relay Chat Protocol http://sunsite.cnlab-switch.ch/ By: Oikarinen & Reed
at all unless the client is a member of the channel in question.
Numeric Replies:
ERR_NOSUCHSERVER RPL_LISTSTART RPL_LIST RPL_LISTEND
Examples:
LIST ; List all channels.
LIST #twilight_zone,#42 ; List channels #twilight_zone and #42
4.2.7 Invite message
Command: INVITE Parameters:
The INVITE message is used to invite users to a channel. The parameter
Numeric Replies:
ERR_NEEDMOREPARAMS ERR_NOSUCHNICK ERR_NOTONCHANNEL ERR_USERONCHANNEL ERR_CHANOPRIVSNEEDED RPL_INVITING RPL_AWAY
Examples:
:Angel INVITE Wiz #Dust ; User Angel inviting WiZ to channel #Dust
INVITE Wiz #Twilight_Zone ; Command to invite WiZ to #Twilight_zone
4.2.8 Kick command
Command: KICK Parameters:
The KICK command can be used to forcibly remove a user from a channel. It 'kicks them out' of the channel (forced PART).
Formatted and Indexed by: page 25 of 65 instructional media + magic, inc. RFC1459 Web Address: Internet Relay Chat Protocol http://sunsite.cnlab-switch.ch/ By: Oikarinen & Reed
Only a channel operator may kick another user out of a channel. Each server that receives a KICK message checks that it is valid (ie the sender is actually a channel operator) before removing the victim from the channel.
Numeric Replies:
ERR_NEEDMOREPARAMS ERR_NOSUCHCHANNEL ERR_BADCHANMASK ERR_CHANOPRIVSNEEDED ERR_NOTONCHANNEL
Examples:
KICK &Melbourne Matthew ; Kick Matthew from &Melbourne
KICK #Finnish John :Speaking English ; Kick John from #Finnish using "Speaking English" as the reason (comment).
:WiZ KICK #Finnish John ; KICK message from WiZ to remove John from channel #Finnish
NOTE: It is possible to extend the KICK command parameters to the following:
4.3 Server queries and commands
The server query group of commands has been designed to return information about any server which is connected to the network. All servers connected must respond to these queries and respond correctly. Any invalid response (or lack thereof) must be considered a sign of a broken server and it must be disconnected/disabled as soon as possible until the situation is remedied.
In these queries, where a parameter appears as "
4.3.1 Version message
Command: VERSION Parameters: [
Formatted and Indexed by: page 26 of 65 instructional media + magic, inc. RFC1459 Web Address: Internet Relay Chat Protocol http://sunsite.cnlab-switch.ch/ By: Oikarinen & Reed
The VERSION message is used to query the version of the server program. An optional parameter
Numeric Replies:
ERR_NOSUCHSERVER RPL_VERSION
Examples:
:Wiz VERSION *.se ; message from Wiz to check the version of a server matching "*.se"
VERSION tolsun.oulu.fi ; check the version of server "tolsun.oulu.fi".
4.3.2 Stats message
Command: STATS Parameters: [
The stats message is used to query statistics of certain server. If
A query may be given by any single letter which is only checked by the destination server (if given as the
The currently supported queries are:
c - returns a list of servers which the server may connect to or allow connections from; h - returns a list of servers which are either forced to be treated as leaves or allowed to act as hubs; i - returns a list of hosts which the server allows a client to connect from; k - returns a list of banned username/hostname combinations for that server; l - returns a list of the server's connections, showing how
Formatted and Indexed by: page 27 of 65 instructional media + magic, inc. RFC1459 Web Address: Internet Relay Chat Protocol http://sunsite.cnlab-switch.ch/ By: Oikarinen & Reed
long each connection has been established and the traffic over that connection in bytes and messages for each direction; m - returns a list of commands supported by the server and the usage count for each if the usage count is non zero; o - returns a list of hosts from which normal clients may become operators; y - show Y (Class) lines from server's configuration file; u - returns a string showing how long the server has been up.
Numeric Replies:
ERR_NOSUCHSERVER RPL_STATSCLINE RPL_STATSNLINE RPL_STATSILINE RPL_STATSKLINE RPL_STATSQLINE RPL_STATSLLINE RPL_STATSLINKINFO RPL_STATSUPTIME RPL_STATSCOMMANDS RPL_STATSOLINE RPL_STATSHLINE RPL_ENDOFSTATS
Examples:
STATS m ; check the command usage for the server you are connected to
:Wiz STATS c eff.org ; request by WiZ for C/N line information from server eff.org
4.3.3 Links message
Command: LINKS Parameters: [[
With LINKS, a user can list all servers which are known by the server answering the query. The returned list of servers must match the mask, or if no mask is given, the full list is returned.
If
Numeric Replies:
ERR_NOSUCHSERVER RPL_LINKS RPL_ENDOFLINKS
Examples:
Formatted and Indexed by: page 28 of 65 instructional media + magic, inc. RFC1459 Web Address: Internet Relay Chat Protocol http://sunsite.cnlab-switch.ch/ By: Oikarinen & Reed
LINKS *.au ; list all servers which have a name that matches *.au;
:WiZ LINKS *.bu.edu *.edu ; LINKS message from WiZ to the first server matching *.edu for a list of servers matching *.bu.edu.
4.3.4 Time message
Command: TIME Parameters: [
The time message is used to query local time from the specified server. If the server parameter is not given, the server handling the command must reply to the query.
Numeric Replies:
ERR_NOSUCHSERVER RPL_TIME
Examples:
TIME tolsun.oulu.fi ; check the time on the server "tolson.oulu.fi"
Angel TIME *.au ; user angel checking the time on a server matching "*.au"
4.3.5 Connect message
Command: CONNECT Parameters:
The CONNECT command can be used to force a server to try to establish a new connection to another server immediately. CONNECT is a privileged command and is to be available only to IRC Operators. If a remote server is given then the CONNECT attempt is made by that server to
Numeric Replies:
ERR_NOSUCHSERVER ERR_NOPRIVILEGES ERR_NEEDMOREPARAMS
Examples:
CONNECT tolsun.oulu.fi ; Attempt to connect a server to tolsun.oulu.fi
Formatted and Indexed by: page 29 of 65 instructional media + magic, inc. RFC1459 Web Address: Internet Relay Chat Protocol http://sunsite.cnlab-switch.ch/ By: Oikarinen & Reed
:WiZ CONNECT eff.org 6667 csd.bu.edu ; CONNECT attempt by WiZ to get servers eff.org and csd.bu.edu connected on port 6667.
4.3.6 Trace message
Command: TRACE Parameters: [
TRACE command is used to find the route to specific server. Each server that processes this message must tell the sender about it by sending a reply indicating it is a pass-through link, forming a chain of replies similar to that gained from using "traceroute". After sending this reply back, it must then send the TRACE message to the next server until given server is reached. If the
If the destination given by "
Numeric Replies:
ERR_NOSUCHSERVER
If the TRACE message is destined for another server, all intermediate servers must return a RPL_TRACELINK reply to indicate that the TRACE passed through it and where its going next.
RPL_TRACELINK A TRACE reply may be composed of any number of the following numeric replies.
RPL_TRACECONNECTING RPL_TRACEHANDSHAKE RPL_TRACEUNKNOWN RPL_TRACEOPERATOR RPL_TRACEUSER RPL_TRACESERVER RPL_TRACESERVICE RPL_TRACENEWTYPE RPL_TRACECLASS
Examples:
TRACE *.oulu.fi ; TRACE to a server matching *.oulu.fi
Formatted and Indexed by: page 30 of 65 instructional media + magic, inc. RFC1459 Web Address: Internet Relay Chat Protocol http://sunsite.cnlab-switch.ch/ By: Oikarinen & Reed
:WiZ TRACE AngelDust ; TRACE issued by WiZ to nick AngelDust
4.3.7 Admin command
Command: ADMIN Parameters: [
The admin message is used to find the name of the administrator of the given server, or current server if
Numeric Replies:
ERR_NOSUCHSERVER RPL_ADMINME RPL_ADMINLOC1 RPL_ADMINLOC2 RPL_ADMINEMAIL
Examples:
ADMIN tolsun.oulu.fi ; request an ADMIN reply from tolsun.oulu.fi
:WiZ ADMIN *.edu ; ADMIN request from WiZ for first server found to match *.edu.
4.3.8 Info command
Command: INFO Parameters: [
The INFO command is required to return information which describes the server: its version, when it was compiled, the patchlevel, when it was started, and any other miscellaneous information which may be considered to be relevant.
Numeric Replies:
ERR_NOSUCHSERVER RPL_INFO RPL_ENDOFINFO
Examples:
INFO csd.bu.edu ; request an INFO reply from csd.bu.edu
:Avalon INFO *.fi ; INFO request from Avalon for first server found to match *.fi.
Formatted and Indexed by: page 31 of 65 instructional media + magic, inc. RFC1459 Web Address: Internet Relay Chat Protocol http://sunsite.cnlab-switch.ch/ By: Oikarinen & Reed
INFO Angel ; request info from the server that Angel is connected to.
4.4 Sending messages
The main purpose of the IRC protocol is to provide a base for clients to communicate with each other. PRIVMSG and NOTICE are the only messages available which actually perform delivery of a text message from one client to another - the rest just make it possible and try to ensure it happens in a reliable and structured manner.
4.4.1 Private messages
Command: PRIVMSG Parameters:
PRIVMSG is used to send private messages between users.
The
Numeric Replies:
ERR_NORECIPIENT ERR_NOTEXTTOSEND ERR_CANNOTSENDTOCHAN ERR_NOTOPLEVEL ERR_WILDTOPLEVEL ERR_TOOMANYTARGETS ERR_NOSUCHNICK RPL_AWAY
Examples:
:Angel PRIVMSG Wiz :Hello are you receiving this message ? ; Message from Angel to Wiz.
PRIVMSG Angel :yes I'm receiving it !receiving it !'u>(768u+1n) .br ; Message to Angel.
PRIVMSG [email protected] :Hello ! ; Message to a client on server
Formatted and Indexed by: page 32 of 65 instructional media + magic, inc. RFC1459 Web Address: Internet Relay Chat Protocol http://sunsite.cnlab-switch.ch/ By: Oikarinen & Reed
tolsun.oulu.fi with username of "jto".
PRIVMSG $*.fi :Server tolsun.oulu.fi rebooting. ; Message to everyone on a server which has a name matching *.fi.
PRIVMSG #*.edu :NSFNet is undergoing work, expect interruptions ; Message to all users who come from a host which has a name matching *.edu.
4.4.2 Notice
Command: NOTICE Parameters:
The NOTICE message is used similarly to PRIVMSG. The difference between NOTICE and PRIVMSG is that automatic replies must never be sent in response to a NOTICE message. This rule applies to servers too - they must not send any error reply back to the client on receipt of a notice. The object of this rule is to avoid loops between a client automatically sending something in response to something it received. This is typically used by automatons (clients with either an AI or other interactive program controlling their actions) which are always seen to be replying lest they end up in a loop with another automaton.
See PRIVMSG for more details on replies and examples.
4.5 User based queries
User queries are a group of commands which are primarily concerned with finding details on a particular user or group users. When using wildcards with any of these commands, if they match, they will only return information on users who are 'visible' to you. The visibility of a user is determined as a combination of the user's mode and the common set of channels you are both on.
4.5.1 Who query
Command: WHO Parameters: [
The WHO message is used by a client to generate a query which returns a list of information which 'matches' the
Formatted and Indexed by: page 33 of 65 instructional media + magic, inc. RFC1459 Web Address: Internet Relay Chat Protocol http://sunsite.cnlab-switch.ch/ By: Oikarinen & Reed
will end up matching every entry possible.
The
If the "o" parameter is passed only operators are returned according to the name mask supplied.
Numeric Replies:
ERR_NOSUCHSERVER RPL_WHOREPLY RPL_ENDOFWHO
Examples:
WHO *.fi ; List all users who match against "*.fi".
WHO jto* o ; List all users with a match against "jto*" if they are an operator.
4.5.2 Whois query
Command: WHOIS Parameters: [
This message is used to query information about particular user. The server will answer this message with several numeric messages indicating different statuses of each user which matches the nickmask (if you are entitled to see them). If no wildcard is present in the
The latter version sends the query to a specific server. It is useful if you want to know how long the user in question has been idle as only local server (ie. the server the user is directly connected to) knows that information, while everything else is globally known.
Numeric Replies:
ERR_NOSUCHSERVER ERR_NONICKNAMEGIVEN RPL_WHOISUSER RPL_WHOISCHANNELS RPL_WHOISCHANNELS RPL_WHOISSERVER RPL_AWAY RPL_WHOISOPERATOR RPL_WHOISIDLE ERR_NOSUCHNICK RPL_ENDOFWHOIS
Formatted and Indexed by: page 34 of 65 instructional media + magic, inc. RFC1459 Web Address: Internet Relay Chat Protocol http://sunsite.cnlab-switch.ch/ By: Oikarinen & Reed
Examples:
WHOIS wiz ; return available user information about nick WiZ
WHOIS eff.org trillian ; ask server eff.org for user information about trillian
4.5.3 Whowas
Command: WHOWAS Parameters:
Whowas asks for information about a nickname which no longer exists. This may either be due to a nickname change or the user leaving IRC. In response to this query, the server searches through its nickname history, looking for any nicks which are lexically the same (no wild card matching here). The history is searched backward, returning the most recent entry first. If there are multiple entries, up to
Numeric Replies:
ERR_NONICKNAMEGIVEN ERR_WASNOSUCHNICK RPL_WHOWASUSER RPL_WHOISSERVER RPL_ENDOFWHOWAS
Examples:
WHOWAS Wiz ; return all information in the nick history about nick "WiZ";
WHOWAS Mermaid 9 ; return at most, the 9 most recent entries in the nick history for "Mermaid";
WHOWAS Trillian 1 *.edu ; return the most recent history for "Trillian" from the first server found to match "*.edu".
4.6 Miscellaneous messages
Messages in this category do not fit into any of the above categories but are nonetheless still a part of and required by the protocol.
Formatted and Indexed by: page 35 of 65 instructional media + magic, inc. RFC1459 Web Address: Internet Relay Chat Protocol http://sunsite.cnlab-switch.ch/ By: Oikarinen & Reed
4.6.1 Kill message
Command: KILL Parameters:
The KILL message is used to cause a client-server connection to be closed by the server which has the actual connection. KILL is used by servers when they encounter a duplicate entry in the list of valid nicknames and is used to remove both entries. It is also available to operators.
Clients which have automatic reconnect algorithms effectively make this command useless since the disconnection is only brief. It does however break the flow of data and can be used to stop large amounts of being abused, any user may elect to receive KILL messages generated for others to keep an 'eye' on would be trouble spots.
In an arena where nicknames are required to be globally unique at all times, KILL messages are sent whenever 'duplicates' are detected (that is an attempt to register two users with the same nickname) in the hope that both of them will disappear and only 1 reappear.
The comment given must reflect the actual reason for the KILL. For server-generated KILLs it usually is made up of details concerning the origins of the two conflicting nicknames. For users it is left up to them to provide an adequate reason to satisfy others who see it. To prevent/discourage fake KILLs from being generated to hide the identify of the KILLer, the comment also shows a 'kill-path' which is updated by each server it passes through, each prepending its name to the path.
Numeric Replies:
ERR_NOPRIVILEGES ERR_NEEDMOREPARAMS ERR_NOSUCHNICK ERR_CANTKILLSERVER
KILL David (csd.bu.edu <- tolsun.oulu.fi) ; Nickname collision between csd.bu.edu and tolson.oulu.fi
NOTE: It is recommended that only Operators be allowed to kill other users with KILL message. In an ideal world not even operators would need to do this and it would be left to servers to deal with.
Formatted and Indexed by: page 36 of 65 instructional media + magic, inc. RFC1459 Web Address: Internet Relay Chat Protocol http://sunsite.cnlab-switch.ch/ By: Oikarinen & Reed
4.6.2 Ping message
Command: PING Parameters:
The PING message is used to test the presence of an active client at the other end of the connection. A PING message is sent at regular intervals if no other activity detected coming from a connection. If a connection fails to respond to a PING command within a set amount of time, that connection is closed.
Any client which receives a PING message must respond to
Numeric Replies:
ERR_NOORIGIN ERR_NOSUCHSERVER
Examples:
PING tolsun.oulu.fi ; server sending a PING message to another server to indicate it is still alive.
PING WiZ ; PING message being sent to nick WiZ
4.6.3 Pong message
Command: PONG Parameters:
PONG message is a reply to ping message. If parameter
Numeric Replies:
ERR_NOORIGIN ERR_NOSUCHSERVER
Examples:
PONG csd.bu.edu tolsun.oulu.fi ; PONG message from csd.bu.edu to
Formatted and Indexed by: page 37 of 65 instructional media + magic, inc. RFC1459 Web Address: Internet Relay Chat Protocol http://sunsite.cnlab-switch.ch/ By: Oikarinen & Reed
tolsun.oulu.fi
4.6.4 Error
Command: ERROR Parameters:
The ERROR command is for use by servers when reporting a serious or fatal error to its operators. It may also be sent from one server to another but must not be accepted from any normal unknown clients.
An ERROR message is for use for reporting errors which occur with a server-to-server link only. An ERROR message is sent to the server at the other end (which sends it to all of its connected operators) and to all operators currently connected. It is not to be passed onto any other servers by a server if it is received from a server.
When a server sends a received ERROR message to its operators, the message should be encapsulated inside a NOTICE message, indicating that the client was not responsible for the error.
Numerics:
None.
Examples:
ERROR :Server *.fi already exists; ERROR message to the other server which caused this error.
NOTICE WiZ :ERROR from csd.bu.edu -- Server *.fi already exists ; Same ERROR message as above but sent to user WiZ on the other server.
5. OPTIONALS
This section describes OPTIONAL messages. They are not required in a working server implementation of the protocol described herein. In the absence of the option, an error reply message must be generated or an unknown command error. If the message is destined for another server to answer then it must be passed on (elementary parsing required) The allocated numerics for this are listed with the messages below.
5.1 Away
Command: AWAY Parameters: [message]
Formatted and Indexed by: page 38 of 65 instructional media + magic, inc. RFC1459 Web Address: Internet Relay Chat Protocol http://sunsite.cnlab-switch.ch/ By: Oikarinen & Reed
With the AWAY message, clients can set an automatic reply string for any PRIVMSG commands directed at them (not to a channel they are on). The automatic reply is sent by the server to client sending the PRIVMSG command. The only replying server is the one to which the sending client is connected to.
The AWAY message is used either with one parameter (to set an AWAY message) or with no parameters (to remove the AWAY message).
Numeric Replies:
RPL_UNAWAY RPL_NOWAWAY
Examples:
AWAY :Gone to lunch. Back in 5 ; set away message to "Gone to lunch. Back in 5".
:WiZ AWAY ; unmark WiZ as being away.
5.2 Rehash message
Command: REHASH Parameters: None
The rehash message can be used by the operator to force the server to re-read and process its configuration file.
Numeric Replies:
RPL_REHASHING ERR_NOPRIVILEGES
Examples:
REHASH ; message from client with operator status to server asking it to reread its configuration file.
5.3 Restart message
Command: RESTART Parameters: None
The restart message can only be used by an operator to force a server restart itself. This message is optional since it may be viewed as a risk to allow arbitrary people to connect to a server as an operator and execute this command, causing (at least) a disruption to service.
Formatted and Indexed by: page 39 of 65 instructional media + magic, inc. RFC1459 Web Address: Internet Relay Chat Protocol http://sunsite.cnlab-switch.ch/ By: Oikarinen & Reed
The RESTART command must always be fully processed by the server to which the sending client is connected and not be passed onto other connected servers.
Numeric Replies:
ERR_NOPRIVILEGES
Examples:
RESTART ; no parameters required.
5.4 Summon message
Command: SUMMON Parameters:
The SUMMON command can be used to give users who are on a host running an IRC server a message asking them to please join IRC. This message is only sent if the target server (a) has SUMMON enabled, (b) the user is logged in and (c) the server process can write to the user's tty (or similar).
If no
If summon is not enabled in a server, it must return the ERR_SUMMONDISABLED numeric and pass the summon message onwards.
Numeric Replies:
ERR_NORECIPIENT ERR_FILEERROR ERR_NOLOGIN ERR_NOSUCHSERVER RPL_SUMMONING
Examples:
SUMMON jto ; summon user jto on the server's host
SUMMON jto tolsun.oulu.fi ; summon user jto on the host which a server named "tolsun.oulu.fi" is running.
5.5 Users
Command: USERS Parameters: [
Formatted and Indexed by: page 40 of 65 instructional media + magic, inc. RFC1459 Web Address: Internet Relay Chat Protocol http://sunsite.cnlab-switch.ch/ By: Oikarinen & Reed
The USERS command returns a list of users logged into the server in a similar format to who(1), rusers(1) and finger(1). Some people may disable this command on their server for security related reasons. If disabled, the correct numeric must be returned to indicate this.
Numeric Replies:
ERR_NOSUCHSERVER ERR_FILEERROR RPL_USERSSTART RPL_USERS RPL_NOUSERS RPL_ENDOFUSERS ERR_USERSDISABLED
Disabled Reply:
ERR_USERSDISABLED
Examples:
USERS eff.org ; request a list of users logged in on server eff.org
:John USERS tolsun.oulu.fi ; request from John for a list of users logged in on server tolsun.oulu.fi
5.6 Operwall message
Command: WALLOPS Parameters: Text to be sent to all operators currently online
Sends a message to all operators currently online. After implementing WALLOPS as a user command it was found that it was often and commonly abused as a means of sending a message to a lot of people (much similar to WALL). Due to this it is recommended that the current implementation of WALLOPS be used as an example by allowing and recognising only servers as the senders of WALLOPS.
Numeric Replies:
ERR_NEEDMOREPARAMS
Examples:
:csd.bu.edu WALLOPS :Connect '*.uiuc.edu 6667' from Joshua; WALLOPS message from csd.bu.edu announcing a CONNECT message it received and acted upon from Joshua.
Formatted and Indexed by: page 41 of 65 instructional media + magic, inc. RFC1459 Web Address: Internet Relay Chat Protocol http://sunsite.cnlab-switch.ch/ By: Oikarinen & Reed
5.7 Userhost message
Command: USERHOST Parameters:
The USERHOST command takes a list of up to 5 nicknames, each separated by a space character and returns a list of information about each nickname that it found. The returned list has each reply separated by a space.
Numeric Replies:
RPL_USERHOST ERR_NEEDMOREPARAMS
Examples:
USERHOST Wiz Michael Marty p ;USERHOST request for information on nicks "Wiz", "Michael", "Marty" and "p"
5.8 Ison message
Command: ISON Parameters:
The ISON command was implemented to provide a quick and efficient means to get a response about whether a given nickname was currently on IRC. ISON only takes one (1) parameter: a space-separated list of nicks. For each nickname in the list that is present, the server adds that to its reply string. Thus the reply string may return empty (none of the given nicks are present), an exact copy of the parameter string (all of them present) or as any other subset of the set of nicks given in the parameter. The only limit on the number of nicks that may be checked is that the combined length must not be too large as to cause the server to chop it off so it fits in 512 characters.
ISON is only be processed by the server local to the client sending the command and thus not passed onto other servers for further processing.
Numeric Replies:
RPL_ISON ERR_NEEDMOREPARAMS
Examples:
ISON phone trillian WiZ jarlek Avalon Angel Monstah ; Sample ISON request for 7 nicks.
Formatted and Indexed by: page 42 of 65 instructional media + magic, inc. RFC1459 Web Address: Internet Relay Chat Protocol http://sunsite.cnlab-switch.ch/ By: Oikarinen & Reed
6. REPLIES
The following is a list of numeric replies which are generated in response to the commands given above. Each numeric is given with its number, name and reply string.
6.1 Error Replies.
401 ERR_NOSUCHNICK "
- Used to indicate the nickname parameter supplied to a command is currently unused.
402 ERR_NOSUCHSERVER "
- Used to indicate the server name given currently doesn't exist.
403 ERR_NOSUCHCHANNEL "
- Used to indicate the given channel name is invalid.
404 ERR_CANNOTSENDTOCHAN "
- Sent to a user who is either (a) not on a channel which is mode +n or (b) not a chanop (or mode +v) on a channel which has mode +m set and is trying to send a PRIVMSG message to that channel.
405 ERR_TOOMANYCHANNELS "
406 ERR_WASNOSUCHNICK "
- Returned by WHOWAS to indicate there is no history information for that nickname.
407 ERR_TOOMANYTARGETS "
Formatted and Indexed by: page 43 of 65 instructional media + magic, inc. RFC1459 Web Address: Internet Relay Chat Protocol http://sunsite.cnlab-switch.ch/ By: Oikarinen & Reed
delivered"
- Returned to a client which is attempting to send a PRIVMSG/NOTICE using the user@host destination format and for a user@host which has several occurrences.
409 ERR_NOORIGIN ":No origin specified"
- PING or PONG message missing the originator parameter which is required since these commands must work without valid prefixes.
411 ERR_NORECIPIENT ":No recipient given (
- 412 - 414 are returned by PRIVMSG to indicate that the message wasn't delivered for some reason. ERR_NOTOPLEVEL and ERR_WILDTOPLEVEL are errors that are returned when an invalid use of "PRIVMSG $
421 ERR_UNKNOWNCOMMAND "
- Returned to a registered client to indicate that the command sent is unknown by the server.
422 ERR_NOMOTD ":MOTD File is missing"
- Server's MOTD file could not be opened by the server.
423 ERR_NOADMININFO "
- Returned by a server in response to an ADMIN message when there is an error in finding the appropriate information.
424 ERR_FILEERROR ":File error doing
Formatted and Indexed by: page 44 of 65 instructional media + magic, inc. RFC1459 Web Address: Internet Relay Chat Protocol http://sunsite.cnlab-switch.ch/ By: Oikarinen & Reed
- Generic error message used to report a failed file operation during the processing of a message.
431 ERR_NONICKNAMEGIVEN ":No nickname given"
- Returned when a nickname parameter expected for a command and isn't found.
432 ERR_ERRONEUSNICKNAME "
- Returned after receiving a NICK message which contains characters which do not fall in the defined set. See section x.x.x for details on valid nicknames.
433 ERR_NICKNAMEINUSE "
- Returned when a NICK message is processed that results in an attempt to change to a currently existing nickname.
436 ERR_NICKCOLLISION "
- Returned by a server to a client when it detects a nickname collision (registered of a NICK that already exists by another server).
441 ERR_USERNOTINCHANNEL "
- Returned by the server to indicate that the target user of the command is not on the given channel.
442 ERR_NOTONCHANNEL "
- Returned by the server whenever a client tries to perform a channel effecting command for which the client isn't a member.
443 ERR_USERONCHANNEL "
- Returned when a client tries to invite a user to a channel they are already on.
Formatted and Indexed by: page 45 of 65 instructional media + magic, inc. RFC1459 Web Address: Internet Relay Chat Protocol http://sunsite.cnlab-switch.ch/ By: Oikarinen & Reed
444 ERR_NOLOGIN "
- Returned by the summon after a SUMMON command for a user was unable to be performed since they were not logged in.
445 ERR_SUMMONDISABLED ":SUMMON has been disabled"
- Returned as a response to the SUMMON command. Must be returned by any server which does not implement it.
446 ERR_USERSDISABLED ":USERS has been disabled"
- Returned as a response to the USERS command. Must be returned by any server which does not implement it.
451 ERR_NOTREGISTERED ":You have not registered"
- Returned by the server to indicate that the client must be registered before the server will allow it to be parsed in detail.
461 ERR_NEEDMOREPARAMS "
- Returned by the server by numerous commands to indicate to the client that it didn't supply enough parameters.
462 ERR_ALREADYREGISTRED ":You may not reregister"
- Returned by the server to any link which tries to change part of the registered details (such as password or user details from second USER message).
463 ERR_NOPERMFORHOST ":Your host isn't among the privileged"
- Returned to a client which attempts to register with a server which does not been setup to allow connections from the host the attempted connection is tried.
Formatted and Indexed by: page 46 of 65 instructional media + magic, inc. RFC1459 Web Address: Internet Relay Chat Protocol http://sunsite.cnlab-switch.ch/ By: Oikarinen & Reed
464 ERR_PASSWDMISMATCH ":Password incorrect"
- Returned to indicate a failed attempt at registering a connection for which a password was required and was either not given or incorrect.
465 ERR_YOUREBANNEDCREEP ":You are banned from this server"
- Returned after an attempt to connect and register yourself with a server which has been setup to explicitly deny connections to you.
467 ERR_KEYSET "
- Any command requiring operator privileges to operate must return this error to indicate the attempt was unsuccessful.
482 ERR_CHANOPRIVSNEEDED "
- Any command requiring 'chanop' privileges (such as MODE messages) must return this error if the client making the attempt is not a chanop on the specified channel.
483 ERR_CANTKILLSERVER ":You cant kill a server!"
- Any attempts to use the KILL command on a server are to be refused and this error returned directly to the client.
Formatted and Indexed by: page 47 of 65 instructional media + magic, inc. RFC1459 Web Address: Internet Relay Chat Protocol http://sunsite.cnlab-switch.ch/ By: Oikarinen & Reed
491 ERR_NOOPERHOST ":No O-lines for your host"
- If a client sends an OPER message and the server has not been configured to allow connections from the client's host as an operator, this error must be returned.
501 ERR_UMODEUNKNOWNFLAG ":Unknown MODE flag"
- Returned by the server to indicate that a MODE message was sent with a nickname parameter and that the a mode flag sent was not recognized.
502 ERR_USERSDONTMATCH ":Cant change mode for other users"
- Error sent to any user trying to view or change the user mode for a user other than themselves.
6.2 Command responses.
300 RPL_NONE Dummy reply number. Not used.
302 RPL_USERHOST ":[
- Reply format used by USERHOST to list replies to the query list. The reply string is composed as follows:
The '*' indicates whether the client has registered as an Operator. The '-' or '+' characters represent whether the client has set an AWAY message or not respectively.
303 RPL_ISON ":[
- Reply format used by ISON to list replies to the query list.
301 RPL_AWAY "
Formatted and Indexed by: page 48 of 65 instructional media + magic, inc. RFC1459 Web Address: Internet Relay Chat Protocol http://sunsite.cnlab-switch.ch/ By: Oikarinen & Reed
305 RPL_UNAWAY ":You are no longer marked as being away" 306 RPL_NOWAWAY ":You have been marked as being away"
- These replies are used with the AWAY command (if allowed). RPL_AWAY is sent to any client sending a PRIVMSG to a client which is away. RPL_AWAY is only sent by the server to which the client is connected. Replies RPL_UNAWAY and RPL_NOWAWAY are sent when the client removes and sets an AWAY message.
311 RPL_WHOISUSER "
- Replies 311 - 313, 317 - 319 are all replies generated in response to a WHOIS message. Given that there are enough parameters present, the answering server must either formulate a reply out of the above numerics (if the query nick is found) or return an error reply. The '*' in RPL_WHOISUSER is there as the literal character and not as a wild card. For each reply set, only RPL_WHOISCHANNELS may appear more than once (for long lists of channel names). The '@' and '+' characters next to the channel name indicate whether a client is a channel operator or has been granted permission to speak on a moderated channel. The RPL_ENDOFWHOIS reply is used to mark the end of processing a WHOIS message.
314 RPL_WHOWASUSER "
- When replying to a WHOWAS message, a server must use the replies RPL_WHOWASUSER, RPL_WHOISSERVER or ERR_WASNOSUCHNICK for each nickname in the presented
Formatted and Indexed by: page 49 of 65 instructional media + magic, inc. RFC1459 Web Address: Internet Relay Chat Protocol http://sunsite.cnlab-switch.ch/ By: Oikarinen & Reed
list. At the end of all reply batches, there must be RPL_ENDOFWHOWAS (even if there was only one reply and it was an error).
321 RPL_LISTSTART "Channel :Users Name" 322 RPL_LIST "
- Replies RPL_LISTSTART, RPL_LIST, RPL_LISTEND mark the start, actual replies with data and end of the server's response to a LIST command. If there are no channels available to return, only the start and end reply must be sent.
324 RPL_CHANNELMODEIS "
331 RPL_NOTOPIC "
- When sending a TOPIC message to determine the channel topic, one of two replies is sent. If the topic is set, RPL_TOPIC is sent back else RPL_NOTOPIC.
341 RPL_INVITING "
- Returned by the server to indicate that the attempted INVITE message was successful and is being passed onto the end client.
342 RPL_SUMMONING "
- Returned by a server answering a SUMMON message to indicate that it is summoning that user.
351 RPL_VERSION "
- Reply by the server showing its version details. The
Formatted and Indexed by: page 50 of 65 instructional media + magic, inc. RFC1459 Web Address: Internet Relay Chat Protocol http://sunsite.cnlab-switch.ch/ By: Oikarinen & Reed
used (including any patchlevel revisions) and the
The "comments" field may contain any comments about the version or further version details.
352 RPL_WHOREPLY "
- The RPL_WHOREPLY and RPL_ENDOFWHO pair are used to answer a WHO message. The RPL_WHOREPLY is only sent if there is an appropriate match to the WHO query. If there is a list of parameters supplied with a WHO message, a RPL_ENDOFWHO must be sent after processing each list item with
353 RPL_NAMREPLY "
- To reply to a NAMES message, a reply pair consisting of RPL_NAMREPLY and RPL_ENDOFNAMES is sent by the server back to the client. If there is no channel found as in the query, then only RPL_ENDOFNAMES is returned. The exception to this is when a NAMES message is sent with no parameters and all visible channels and contents are sent back in a series of RPL_NAMEREPLY messages with a RPL_ENDOFNAMES to mark the end.
364 RPL_LINKS "
- In replying to the LINKS message, a server must send replies back using the RPL_LINKS numeric and mark the end of the list using an RPL_ENDOFLINKS reply.
367 RPL_BANLIST "
Formatted and Indexed by: page 51 of 65 instructional media + magic, inc. RFC1459 Web Address: Internet Relay Chat Protocol http://sunsite.cnlab-switch.ch/ By: Oikarinen & Reed
"
- When listing the active 'bans' for a given channel, a server is required to send the list back using the RPL_BANLIST and RPL_ENDOFBANLIST messages. A separate RPL_BANLIST is sent for each active banid. After the banids have been listed (or if none present) a RPL_ENDOFBANLIST must be sent.
371 RPL_INFO ":
- A server responding to an INFO message is required to send all its 'info' in a series of RPL_INFO messages with a RPL_ENDOFINFO reply to indicate the end of the replies.
375 RPL_MOTDSTART ":-
- When responding to the MOTD message and the MOTD file is found, the file is displayed line by line, with each line no longer than 80 characters, using RPL_MOTD format replies. These should be surrounded by a RPL_MOTDSTART (before the RPL_MOTDs) and an RPL_ENDOFMOTD (after).
381 RPL_YOUREOPER ":You are now an IRC operator"
- RPL_YOUREOPER is sent back to a client which has just successfully issued an OPER message and gained operator status.
382 RPL_REHASHING "
- If the REHASH option is used and an operator sends a REHASH message, an RPL_REHASHING is sent back to the operator.
391 RPL_TIME
Formatted and Indexed by: page 52 of 65 instructional media + magic, inc. RFC1459 Web Address: Internet Relay Chat Protocol http://sunsite.cnlab-switch.ch/ By: Oikarinen & Reed
"
- When replying to the TIME message, a server must send the reply using the RPL_TIME format above. The string showing the time need only contain the correct day and time there. There is no further requirement for the time string.
392 RPL_USERSSTART ":UserID Terminal Host" 393 RPL_USERS ":%-8s %-9s %-8s" 394 RPL_ENDOFUSERS ":End of users" 395 RPL_NOUSERS ":Nobody logged in"
- If the USERS message is handled by a server, the replies RPL_USERSTART, RPL_USERS, RPL_ENDOFUSERS and RPL_NOUSERS are used. RPL_USERSSTART must be sent first, following by either a sequence of RPL_USERS or a single RPL_NOUSER. Following this is RPL_ENDOFUSERS.
200 RPL_TRACELINK "Link
- The RPL_TRACE* are all returned by the server in response to the TRACE message. How many are returned is dependent on the the TRACE message and
Formatted and Indexed by: page 53 of 65 instructional media + magic, inc. RFC1459 Web Address: Internet Relay Chat Protocol http://sunsite.cnlab-switch.ch/ By: Oikarinen & Reed
whether it was sent by an operator or not. There is no predefined order for which occurs first. Replies RPL_TRACEUNKNOWN, RPL_TRACECONNECTING and RPL_TRACEHANDSHAKE are all used for connections which have not been fully established and are either unknown, still attempting to connect or in the process of completing the 'server handshake'. RPL_TRACELINK is sent by any server which handles a TRACE message and has to pass it on to another server. The list of RPL_TRACELINKs sent in response to a TRACE command traversing the IRC network should reflect the actual connectivity of the servers themselves along that path. RPL_TRACENEWTYPE is to be used for any connection which does not fit in the other categories but is being displayed anyway.
211 RPL_STATSLINKINFO "
221 RPL_UMODEIS "
Formatted and Indexed by: page 54 of 65 instructional media + magic, inc. RFC1459 Web Address: Internet Relay Chat Protocol http://sunsite.cnlab-switch.ch/ By: Oikarinen & Reed
- To answer a query about a client's own mode, RPL_UMODEIS is sent back.
251 RPL_LUSERCLIENT ":There are
- In processing an LUSERS message, the server sends a set of replies from RPL_LUSERCLIENT, RPL_LUSEROP, RPL_USERUNKNOWN, RPL_LUSERCHANNELS and RPL_LUSERME. When replying, a server must send back RPL_LUSERCLIENT and RPL_LUSERME. The other replies are only sent back if a non-zero count is found for them.
256 RPL_ADMINME "
- When replying to an ADMIN message, a server is expected to use replies RLP_ADMINME through to RPL_ADMINEMAIL and provide a text message with each. For RPL_ADMINLOC1 a description of what city, state and country the server is in is expected, followed by details of the university and department (RPL_ADMINLOC2) and finally the administrative contact for the server (an email address here is required) in RPL_ADMINEMAIL.
Formatted and Indexed by: page 55 of 65 instructional media + magic, inc. RFC1459 Web Address: Internet Relay Chat Protocol http://sunsite.cnlab-switch.ch / By: Oikarinen & Reed
6.3 Reserved numerics.
These numerics are not described above since they fall into one of the following categories:
1. no longer in use;
2. reserved for future planned use;
3. in current use but are part of a non-generic 'feature' of the current IRC server.
209 RPL_TRACECLASS 217 RPL_STATSQLINE 231 RPL_SERVICEINFO 232 RPL_ENDOFSERVICES 233 RPL_SERVICE 234 RPL_SERVLIST 235 RPL_SERVLISTEND 316 RPL_WHOISCHANOP 361 RPL_KILLDONE 362 RPL_CLOSING 363 RPL_CLOSEEND 373 RPL_INFOSTART 384 RPL_MYPORTIS 466 ERR_YOUWILLBEBANNED 476 ERR_BADCHANMASK 492 ERR_NOSERVICEHOST
7. Client and server authentication
Clients and servers are both subject to the same level of authentication. For both, an IP number to hostname lookup (and reverse check on this) is performed for all connections made to the server. Both connections are then subject to a password check (if there is a password set for that connection). These checks are possible on all connections although the password check is only commonly used with servers.
An additional check that is becoming of more and more common is that of the username responsible for making the connection. Finding the username of the other end of the connection typically involves connecting to an authentication server such as IDENT as described in RFC 1413.
Given that without passwords it is not easy to reliably determine who is on the other end of a network connection, use of passwords is strongly recommended on inter-server connections in addition to any other measures such as using an ident server.
8. Current implementations
The only current implementation of this protocol is the IRC server, version 2.8. Earlier versions may implement some or all of the commands described by this document with NOTICE messages replacing
Formatted and Indexed by: page 56 of 65 instructional media + magic, inc. RFC1459 Web Address: Internet Relay Chat Protocol http://sunsite.cnlab-switch.ch/ By: Oikarinen & Reed
many of the numeric replies. Unfortunately, due to backward compatibility requirements, the implementation of some parts of this document varies with what is laid out. On notable difference is:
* recognition that any LF or CR anywhere in a message marks the end of that message (instead of requiring CR-LF);
The rest of this section deals with issues that are mostly of importance to those who wish to implement a server but some parts also apply directly to clients as well.
8.1 Network protocol: TCP - why it is best used here.
IRC has been implemented on top of TCP since TCP supplies a reliable network protocol which is well suited to this scale of conferencing. The use of multicast IP is an alternative, but it is not widely available or supported at the present time.
8.1.1 Support of Unix sockets
Given that Unix domain sockets allow listen/connect operations, the current implementation can be configured to listen and accept both client and server connections on a Unix domain socket. These are recognized as sockets where the hostname starts with a '/'.
When providing any information about the connections on a Unix domain socket, the server is required to supplant the actual hostname in place of the pathname unless the actual socket name is being asked for.
8.2 Command Parsing
To provide useful 'non-buffered' network IO for clients and servers, each connection is given its own private 'input buffer' in which the results of the most recent read and parsing are kept. A buffer size of 512 bytes is used so as to hold 1 full message, although, this will usually hold several commands. The private buffer is parsed after every read operation for valid messages. When dealing with multiple messages from one client in the buffer, care should be taken in case one happens to cause the client to be 'removed'.
8.3 Message delivery
It is common to find network links saturated or hosts to which you are sending data unable to send data. Although Unix typically handles this through the TCP window and internal buffers, the server often has large amounts of data to send (especially when a new server-server link forms) and the small buffers provided in the
Formatted and Indexed by: page 57 of 65 instructional media + magic, inc. RFC1459 Web Address: Internet Relay Chat Protocol http://sunsite.cnlab-switch.ch/ By: Oikarinen & Reed
kernel are not enough for the outgoing queue. To alleviate this problem, a "send queue" is used as a FIFO queue for data to be sent. A typical "send queue" may grow to 200 Kbytes on a large IRC network with a slow network connection when a new server connects.
When polling its connections, a server will first read and parse all incoming data, queuing any data to be sent out. When all available input is processed, the queued data is sent. This reduces the number of write() system calls and helps TCP make bigger packets.
8.4 Connection 'Liveness'
To detect when a connection has died or become unresponsive, the server must ping each of its connections that it doesn't get a response from in a given amount of time.
If a connection doesn't respond in time, its connection is closed using the appropriate procedures. A connection is also dropped if its sendq grows beyond the maximum allowed, because it is better to close a slow connection than have a server process block.
8.5 Establishing a server to client connection
Upon connecting to an IRC server, a client is sent the MOTD (if present) as well as the current user/server count (as per the LUSER command). The server is also required to give an unambiguous message to the client which states its name and version as well as any other introductory messages which may be deemed appropriate.
After dealing with this, the server must then send out the new user's nickname and other information as supplied by itself (USER command) and as the server could discover (from DNS/authentication servers). The server must send this information out with NICK first followed by USER.
8.6 Establishing a server-server connection.
The process of establishing of a server-to-server connection is fraught with danger since there are many possible areas where problems can occur - the least of which are race conditions.
After a server has received a connection following by a PASS/SERVER pair which were recognised as being valid, the server should then reply with its own PASS/SERVER information for that connection as well as all of the other state information it knows about as described below.
When the initiating server receives a PASS/SERVER pair, it too then
Formatted and Indexed by: page 58 of 65 instructional media + magic, inc. RFC1459 Web Address: Internet Relay Chat Protocol http://sunsite.cnlab-switch.ch/ By: Oikarinen & Reed
checks that the server responding is authenticated properly before accepting the connection to be that server.
8.6.1 Server exchange of state information when connecting
The order of state information being exchanged between servers is essential. The required order is as follows:
* all known other servers;
* all known user information;
* all known channel information.
Information regarding servers is sent via extra SERVER messages, user information with NICK/USER/MODE/JOIN messages and channels with MODE messages.
NOTE: channel topics are *NOT* exchanged here because the TOPIC command overwrites any old topic information, so at best, the two sides of the connection would exchange topics.
By passing the state information about servers first, any collisions with servers that already exist occur before nickname collisions due to a second server introducing a particular nickname. Due to the IRC network only being able to exist as an acyclic graph, it may be possible that the network has already reconnected in another location, the place where the collision occurs indicating where the net needs to split.
8.7 Terminating server-client connections
When a client connection closes, a QUIT message is generated on behalf of the client by the server to which the client connected. No other message is to be generated or used.
8.8 Terminating server-server connections
If a server-server connection is closed, either via a remotely generated SQUIT or 'natural' causes, the rest of the connected IRC network must have its information updated with by the server which detected the closure. The server then sends a list of SQUITs (one for each server behind that connection) and a list of QUITs (again, one for each client behind that connection).
Formatted and Indexed by: page 59 of 65 instructional media + magic, inc. RFC1459 Web Address: Internet Relay Chat Protocol http://sunsite.cnlab-switch.ch/ By: Oikarinen & Reed
8.9 Tracking nickname changes
All IRC servers are required to keep a history of recent nickname changes. This is required to allow the server to have a chance of keeping in touch of things when nick-change race conditions occur with commands which manipulate them. Commands which must trace nick changes are:
* KILL (the nick being killed)
* MODE (+/- o,v)
* KICK (the nick being kicked)
No other commands are to have nick changes checked for.
In the above cases, the server is required to first check for the existence of the nickname, then check its history to see who that nick currently belongs to (if anyone!). This reduces the chances of race conditions but they can still occur with the server ending up affecting the wrong client. When performing a change trace for an above command it is recommended that a time range be given and entries which are too old ignored.
For a reasonable history, a server should be able to keep previous nickname for every client it knows about if they all decided to change. This size is limited by other factors (such as memory, etc).
8.10 Flood control of clients
With a large network of interconnected IRC servers, it is quite easy for any single client attached to the network to supply a continuous stream of messages that result in not only flooding the network, but also degrading the level of service provided to others. Rather than require every 'victim' to be provide their own protection, flood protection was written into the server and is applied to all clients except services. The current algorithm is as follows:
* check to see if client's `message timer' is less than current time (set to be equal if it is);
* read any data present from the client;
* while the timer is less than ten seconds ahead of the current time, parse any present messages and penalize the client by 2 seconds for each message;
which in essence means that the client may send 1 message every 2
Formatted and Indexed by: page 60 of 65 instructional media + magic, inc. RFC1459 Web Address: Internet Relay Chat Protocol http://sunsite.cnlab-switch.ch/ By: Oikarinen & Reed
seconds without being adversely affected.
8.11 Non-blocking lookups
In a real-time environment, it is essential that a server process do as little waiting as possible so that all the clients are serviced fairly. Obviously this requires non-blocking IO on all network read/write operations. For normal server connections, this was not difficult, but there are other support operations that may cause the server to block (such as disk reads). Where possible, such activity should be performed with a short timeout.
8.11.1 Hostname (DNS) lookups
Using the standard resolver libraries from Berkeley and others has meant large delays in some cases where replies have timed out. To avoid this, a separate set of DNS routines were written which were setup for non-blocking IO operations and then polled from within the main server IO loop.
8.11.2 Username (Ident) lookups
Although there are numerous ident libraries for use and inclusion into other programs, these caused problems since they operated in a synchronous manner and resulted in frequent delays. Again the solution was to write a set of routines which would cooperate with the rest of the server and work using non-blocking IO.
8.12 Configuration File
To provide a flexible way of setting up and running the server, it is recommended that a configuration file be used which contains instructions to the server on the following:
* which hosts to accept client connections from;
* which hosts to allow to connect as servers;
* which hosts to connect to (both actively and passively);
* information about where the server is (university, city/state, company are examples of this);
* who is responsible for the server and an email address at which they can be contacted;
* hostnames and passwords for clients which wish to be given
Formatted and Indexed by: page 61 of 65 instructional media + magic, inc. RFC1459 Web Address: Internet Relay Chat Protocol http://sunsite.cnlab-switch.ch/ By: Oikarinen & Reed
access to restricted operator commands.
In specifying hostnames, both domain names and use of the 'dot' notation (127.0.0.1) should both be accepted. It must be possible to specify the password to be used/accepted for all outgoing and incoming connections (although the only outgoing connections are those to other servers).
The above list is the minimum requirement for any server which wishes to make a connection with another server. Other items which may be of use are:
* specifying which servers other server may introduce;
* how deep a server branch is allowed to become;
* hours during which clients may connect.
8.12.1 Allowing clients to connect
A server should use some sort of 'access control list' (either in the configuration file or elsewhere) that is read at startup and used to decide what hosts clients may use to connect to it.
Both 'deny' and 'allow' should be implemented to provide the required flexibility for host access control.
8.12.2 Operators
The granting of operator privileges to a disruptive person can have dire consequences for the well-being of the IRC net in general due to the powers given to them. Thus, the acquisition of such powers should not be very easy. The current setup requires two 'passwords' to be used although one of them is usually easy guessed. Storage of oper passwords in configuration files is preferable to hard coding them in and should be stored in a crypted format (ie using crypt(3) from Unix) to prevent easy theft.
8.12.3 Allowing servers to connect
The interconnection of server is not a trivial matter: a bad connection can have a large impact on the usefulness of IRC. Thus, each server should have a list of servers to which it may connect and which servers may connect to it. Under no circumstances should a server allow an arbitrary host to connect as a server. In addition to which servers may and may not connect, the configuration file should also store the password and other characteristics of that link.
Formatted and Indexed by: page 62 of 65 instructional media + magic, inc. RFC1459 Web Address: Internet Relay Chat Protocol http://sunsite.cnlab-switch.ch/ By: Oikarinen & Reed
8.12.4 Administrivia
To provide accurate and valid replies to the ADMIN command (see section 4.3.7), the server should find the relevant details in the configuration.
8.13 Channel membership
The current server allows any registered local user to join upto 10 different channels. There is no limit imposed on non-local users so that the server remains (reasonably) consistant with all others on a channel membership basis
9. Current problems
There are a number of recognized problems with this protocol, all of which hope to be solved sometime in the near future during its rewrite. Currently, work is underway to find working solutions to these problems.
9.1 Scalability
It is widely recognized that this protocol does not scale sufficiently well when used in a large arena. The main problem comes from the requirement that all servers know about all other servers and users and that information regarding them be updated as soon as it changes. It is also desirable to keep the number of servers low so that the path length between any two points is kept minimal and the spanning tree as strongly branched as possible.
9.2 Labels
The current IRC protocol has 3 types of labels: the nickname, the channel name and the server name. Each of the three types has its own domain and no duplicates are allowed inside that domain. Currently, it is possible for users to pick the label for any of the three, resulting in collisions. It is widely recognized that this needs reworking, with a plan for unique names for channels and nicks that don't collide being desirable as well as a solution allowing a cyclic tree.
9.2.1 Nicknames
The idea of the nickname on IRC is very convenient for users to use when talking to each other outside of a channel, but there is only a finite nickname space and being what they are, its not uncommon for several people to want to use the same nick. If a nickname is chosen by two people using this protocol, either one will not succeed or
Formatted and Indexed by: page 63 of 65 instructional media + magic, inc. RFC1459 Web Address: Internet Relay Chat Protocol http://sunsite.cnlab-switch.ch/ By: Oikarinen & Reed
both will removed by use of KILL (4.6.1).
9.2.2 Channels
The current channel layout requires that all servers know about all channels, their inhabitants and properties. Besides not scaling well, the issue of privacy is also a concern. A collision of channels is treated as an inclusive event (both people who create the new channel are considered to be members of it) rather than an exclusive one such as used to solve nickname collisions.
9.2.3 Servers
Although the number of servers is usually small relative to the number of users and channels, they two currently required to be known globally, either each one separately or hidden behind a mask.
9.3 Algorithms
In some places within the server code, it has not been possible to avoid N^2 algorithms such as checking the channel list of a set of clients.
In current server versions, there are no database consistency checks, each server assumes that a neighbouring server is correct. This opens the door to large problems if a connecting server is buggy or otherwise tries to introduce contradictions to the existing net.
Currently, because of the lack of unique internal and global labels, there are a multitude of race conditions that exist. These race conditions generally arise from the problem of it taking time for messages to traverse and effect the IRC network. Even by changing to unique labels, there are problems with channel-related commands being disrupted.
10. Current support and availability
Mailing lists for IRC related discussion: Future protocol: ircd[email protected] General discussion: [email protected]
Software implemenations cs.bu.edu:/irc nic.funet.fi:/pub/irc coombs.anu.edu.au:/pub/irc
Newsgroup: alt.irc
Formatted and Indexed by: page 64 of 65 instructional media + magic, inc. RFC1459 Web Address: Internet Relay Chat Protocol http://sunsite.cnlab-switch.ch/ By: Oikarinen & Reed
Security Considerations
Security issues are discussed in sections 4.1, 4.1.1, 4.1.3, 5.5, and 7.
12. Authors' Addresses
Jarkko Oikarinen Tuirantie 17 as 9 90500 OULU FINLAND
Email: [email protected]
Darren Reed 4 Pateman Street Watsonia, Victoria 3087 Australia
Email: [email protected]
Formatted and Indexed by: page 65 of 65 instructional media + magic, inc. RFC1630 Web Address: Universal Resource Identifiers in WWW http://sunsite.cnlab-switch.ch/ By: Berners-Lee
Network Working Group T. Berners-Lee Request for Comments: 1630 CERN Category: Informational June 1994
Universal Resource Identifiers in WWW
A Unifying Syntax for the Expression of Names and Addresses of Objects on the Network as used in the World-Wide Web
Status of this Memo
This memo provides information for the Internet community. This memo does not specify an Internet standard of any kind. Distribution of this memo is unlimited.
IESG Note:
Note that the work contained in this memo does not describe an Internet standard. An Internet standard for general Resource Identifiers is under development within the IETF.
Introduction
This document defines the syntax used by the World-Wide Web initiative to encode the names and addresses of objects on the Internet. The web is considered to include objects accessed using an extendable number of protocols, existing, invented for the web itself, or to be invented in the future. Access instructions for an individual object under a given protocol are encoded into forms of address string. Other protocols allow the use of object names of various forms. In order to abstract the idea of a generic object, the web needs the concepts of the universal set of objects, and of the universal set of names or addresses of objects.
A Universal Resource Identifier (URI) is a member of this universal set of names in registered name spaces and addresses referring to registered protocols or name spaces. A Uniform Resource Locator (URL), defined elsewhere, is a form of URI which expresses an address which maps onto an access algorithm using network protocols. Existing URI schemes which correspond to the (still mutating) concept of IETF URLs are listed here. The Uniform Resource Name (URN) debate attempts to define a name space (and presumably resolution protocols) for persistent object names. This area is not addressed by this document, which is written in order to document existing practice and provide a reference point for URL and URN discussions.
Formatted and Indexed by: page 1 of 29 instructional media + magic, inc. RFC1630 Web Address: Universal Resource Identifiers in WWW http://sunsite.cnlab-switch.ch/ By: Berners-Lee
The world-wide web protocols are discussed on the mailing list www- [email protected] and the newsgroup comp.infosystems.www is preferable for beginner's questions. The mailing list uri- [email protected] has discussion related particularly to the URI issue. The author may be contacted as [email protected].
This document is available in hypertext form at:
http://info.cern.ch/hypertext/WWW/Addressing/URL/URI_Overview.html
The Need For a Universal Syntax
This section describes the concept of the URI and does not form part of the specification.
Many protocols and systems for document search and retrieval are currently in use, and many more protocols or refinements of existing protocols are to be expected in a field whose expansion is explosive.
These systems are aiming to achieve global search and readership of documents across differing computing platforms, and despite a plethora of protocols and data formats. As protocols evolve, gateways can allow global access to remain possible. As data formats evolve, format conversion programs can preserve global access. There is one area, however, in which it is impractical to make conversions, and that is in the names and addresses used to identify objects. This is because names and addresses of objects are passed on in so many ways, from the backs of envelopes to hypertext objects, and may have a long life.
A common feature of almost all the data models of past and proposed systems is something which can be mapped onto a concept of "object" and some kind of name, address, or identifier for that object. One can therefore define a set of name spaces in which these objects can be said to exist.
Practical systems need to access and mix objects which are part of different existing and proposed systems. Therefore, the concept of the universal set of all objects, and hence the universal set of names and addresses, in all name spaces, becomes important. This allows names in different spaces to be treated in a common way, even though names in different spaces have differing characteristics, as do the objects to which they refer.
Formatted and Indexed by: page 2 of 29 instructional media + magic, inc. RFC1630 Web Address: Universal Resource Identifiers in WWW http://sunsite.cnlab-switch.ch/ By: Berners-Lee
URIs
This document defines a way to encapsulate a name in any registered name space, and label it with the the name space, producing a member of the universal set. Such an encoded and labelled member of this set is known as a Universal Resource Identifier, or URI.
The universal syntax allows access of objects available using existing protocols, and may be extended with technology.
The specification of the URI syntax does not imply anything about the properties of names and addresses in the various name spaces which are mapped onto the set of URI strings. The properties follow from the specifications of the protocols and the associated usage conventions for each scheme.
URLs
For existing Internet access protocols, it is necessary in most cases to define the encoding of the access algorithm into something concise enough to be termed address. URIs which refer to objects accessed with existing protocols are known as "Uniform Resource Locators" (URLs) and are listed here as used in WWW, but to be formally defined in a separate document.
URNs
There is currently a drive to define a space of more persistent names than any URLs. These "Uniform Resource Names" are the subject of an IETF working group's discussions. (See Sollins and Masinter, Functional Specifications for URNs, circulated informally.)
The URI syntax and URL forms have been in widespread use by World-Wide Web software since 1990.
Formatted and Indexed by: page 3 of 29 instructional media + magic, inc. RFC1630 Web Address: Universal Resource Identifiers in WWW http://sunsite.cnlab-switch.ch/ By: Berners-Lee
Design Criteria and Choices
This section is not part of the specification: it is simply an explanation of the way in which the specification was derived.
Design criteria
The syntax was designed to be:
Extensible New naming schemes may be added later.
Complete It is possible to encode any naming scheme.
Printable It is possible to express any URI using 7-bit ASCII characters so that URIs may, if necessary, be passed using pen and ink.
Choices for a universal syntax
For the syntax itself there is little choice except for the order and punctuation of the elements, and the acceptable characters and escaping rules.
The extensibility requirement is met by allowing an arbitrary (but registered) string to be used as a prefix. A prefix is chosen as left to right parsing is more common than right to left. The choice of a colon as separator of the prefix from the rest of the URI was arbitrary.
The decoding of the rest of the string is defined as a function of the prefix. New prefixed are introduced for new schemes as necessary, in agreement with the registration authority. The registration of a new scheme clearly requires the definition of the decoding of the URI into a given name space, and a definition of the properties and, where applicable, resolution protocols, for the name space.
The completeness requirement is easily met by allowing particularly strange or plain binary names to be encoded in base 16 or 64 using the acceptable characters.
The printability requirement could have been met by requiring all schemes to encode characters not part of a basic set. This led to many discussions of what the basic set should be. A difficult case, for example, is when an ISO latin 1 string appears in a URL, and within an application with ISO Latin-1 capability, it can be handled intact. However, for transport in general, the non-ASCII
Formatted and Indexed by: page 4 of 29 instructional media + magic, inc. RFC1630 Web Address: Universal Resource Identifiers in WWW http://sunsite.cnlab-switch.ch/ By: Berners-Lee
characters need to be escaped.
The solution to this was to specify a safe set of characters, and a general escaping scheme which may be used for encoding "unsafe" characters. This "safe" set is suitable, for example, for use in electronic mail. This is the canonical form of a URI.
The choice of escape character for introducing representations of non-allowed characters also tends to be a matter of taste. An ANSI standard exists in the C language, using the back-slash character "\". The use of this character on unix command lines, however, can be a problem as it is interpreted by many shell programs, and would have itself to be escaped. It is also a character which is not available on certain keyboards. The equals sign is commonly used in the encoding of names having attribute=value pairs. The percent sign was eventually chosen as a suitable escape character.
There is a conflict between the need to be able to represent many characters including spaces within a URI directly, and the need to be able to use a URI in environments which have limited character sets or in which certain characters are prone to corruption. This conflict has been resolved by use of an hexadecimal escaping method which may be applied to any characters forbidden in a given context. When URLs are moved between contexts, the set of characters escaped may be enlarged or reduced unambiguously.
The use of white space characters is risky in URIs to be printed or sent by electronic mail, and the use of multiple white space characters is very risky. This is because of the frequent introduction of extraneous white space when lines are wrapped by systems such as mail, or sheer necessity of narrow column width, and because of the inter-conversion of various forms of white space which occurs during character code conversion and the transfer of text between applications. This is why the canonical form for URIs has all white spaces encoded.
Reommendations
This section describes the syntax for URIs as used in the WorldWide Web initiative. The generic syntax provides a framework for new schemes for names to be resolved using as yet undefined protocols.
URI syntax
A complete URI consists of a naming scheme specifier followed by a string whose format is a function of the naming scheme. For locators of information on the Internet, a common syntax is used for the IP
Formatted and Indexed by: page 5 of 29 instructional media + magic, inc. RFC1630 Web Address: Universal Resource Identifiers in WWW http://sunsite.cnlab-switch.ch/ By: Berners-Lee
address part. A BNF description of the URL syntax is given in an a later section. The components are as follows. Fragment identifiers and relative URIs are not involved in the basic URL definition.
SCHEME
Within the URI of a object, the first element is the name of the scheme, separated from the rest of the object by a colon.
PATH
The rest of the URI follows the colon in a format depending on the scheme. The path is interpreted in a manner dependent on the protocol being used. However, when it contains slashes, these must imply a hierarchical structure.
Reserved characters
The path in the URI has a significance defined by the particular scheme. Typically, it is used to encode a name in a given name space, or an algorithm for accessing an object. In either case, the encoding may use those characters allowed by the BNF syntax, or hexadecimal encoding of other characters.
Some of the reserved characters have special uses as defined here.
THE PERCENT SIGN
The percent sign ("%", ASCII 25 hex) is used as the escape character in the encoding scheme and is never allowed for anything else.
HIERARCHICAL FORMS
The slash ("/", ASCII 2F hex) character is reserved for the delimiting of substrings whose relationship is hierarchical. This enables partial forms of the URI. Substrings consisting of single or double dots ("." or "..") are similarly reserved.
The significance of the slash between two segments is that the segment of the path to the left is more significant than the segment of the path to the right. ("Significance" in this case refers solely to closeness to the root of the hierarchical structure and makes no value judgement!)
Formatted and Indexed by: page 6 of 29 instructional media + magic, inc. RFC1630 Web Address: Universal Resource Identifiers in WWW http://sunsite.cnlab-switch.ch/ By: Berners-Lee
Note
The similarity to unix and other disk operating system filename conventions should be taken as purely coincidental, and should not be taken to indicate that URIs should be interpreted as file names.
HASH FOR FRAGMENT IDENTIFIERS
The hash ("#", ASCII 23 hex) character is reserved as a delimiter to separate the URI of an object from a fragment identifier .
QUERY STRINGS
The question mark ("?", ASCII 3F hex) is used to delimit the boundary between the URI of a queryable object, and a set of words used to express a query on that object. When this form is used, the combined URI stands for the object which results from the query being applied to the original object.
Within the query string, the plus sign is reserved as shorthand notation for a space. Therefore, real plus signs must be encoded. This method was used to make query URIs easier to pass in systems which did not allow spaces.
The query string represents some operation applied to the object, but this specification gives no common syntax or semantics for it. In practice the syntax and sematics may depend on the scheme and may even on the base URI.
OTHER RESERVED CHARACTERS
The astersik ("*", ASCII 2A hex) and exclamation mark ("!" , ASCII 21 hex) are reserved for use as having special signifiance within specific schemes.
Unsafe characters
In canonical form, certain characters such as spaces, control characters, some characters whose ASCII code is used differently in different national character variant 7 bit sets, and all 8bit characters beyond DEL (7F hex) of the ISO Latin-1 set, shall not be used unencoded. This is a recommendation for trouble-free interchange, and as indicated below, the encoded set may be extended or reduced.
Formatted and Indexed by: page 7 of 29 instructional media + magic, inc. RFC1630 Web Address: Universal Resource Identifiers in WWW http://sunsite.cnlab-switch.ch/ By: Berners-Lee
Encoding reserved characters
When a system uses a local addressing scheme, it is useful to provide a mapping from local addresses into URIs so that references to objects within the addressing scheme may be referred to globally, and possibly accessed through gateway servers.
For a new naming scheme, any mapping scheme may be defined provided it is unambiguous, reversible, and provides valid URIs. It is recommended that where hierarchical aspects to the local naming scheme exist, they be mapped onto the hierarchical URL path syntax in order to allow the partial form to be used.
It is also recommended that the conventional scheme below be used in all cases except for any scheme which encodes binary data as opposed to text, in which case a more compact encoding such as pure hexadecimal or base 64 might be more appropriate. For example, the conventional URI encoding method is used for mapping WAIS, FTP, Prospero and Gopher addresses in the URI specification.
CONVENTIONAL URI ENCODING SCHEME
Where the local naming scheme uses ASCII characters which are not allowed in the URI, these may be represented in the URL by a percent sign "%" immediately followed by two hexadecimal digits (0-9, A-F) giving the ISO Latin 1 code for that character. Character codes other than those allowed by the syntax shall not be used unencoded in a URI.
REDUCED OR INCREASED SAFE CHARACTER SETS
The same encoding method may be used for encoding characters whose use, although technically allowed in a URI, would be unwise due to problems of corruption by imperfect gateways or misrepresentation due to the use of variant character sets, or which would simply be awkward in a given environment. Because a % sign always indicates an encoded character, a URI may be made "safer" simply by encoding any characters considered unsafe, while leaving already encoded characters still encoded. Similarly, in cases where a larger set of characters is acceptable, % signs can be selectively and reversibly expanded.
Before two URIs can be compared, it is therefore necessary to bring them to the same encoding level.
However, the reserved characters mentioned above have a quite different significance when encoded, and so may NEVER be encoded and unencoded in this way.
Formatted and Indexed by: page 8 of 29 instructional media + magic, inc. RFC1630 Web Address: Universal Resource Identifiers in WWW http://sunsite.cnlab-switch.ch/ By: Berners-Lee
The percent sign intended as such must always be encoded, as its presence otherwise always indicates an encoding. Sequences which start with a percent sign but are not followed by two hexadecimal characters are reserved for future extension. (See Example 3.)
Example 1
The URIs
http://info.cern.ch/albert/bertram/marie-claude
and
http://info.cern.ch/albert/bertram/marie%2Dclaude
are identical, as the %2D encodes a hyphen character.
Example 2
The URIs
http://info.cern.ch/albert/bertram/marie-claude
and
http://info.cern.ch/albert/bertram%2Fmarie-claude
are NOT identical, as in the second case the encoded slash does not have hierarchical significance.
Example 3
The URIs
fxqn:/us/va/reston/cnri/ietf/24/asdf%*.fred
and
news:12345667123%[email protected]
are illegal, as all % characters imply encodings, and there is no decoding defined for "%*" or "%as" in this recommendation.
Partial (relative) form
Within a object whose URI is well defined, the URI of another object may be given in abbreviated form, where parts of the two URIs are the same. This allows objects within a group to refer to each other
Formatted and Indexed by: page 9 of 29 instructional media + magic, inc. RFC1630 Web Address: Universal Resource Identifiers in WWW http://sunsite.cnlab-switch.ch/ By: Berners-Lee
without requiring the space for a complete reference, and it incidentally allows the group of objects to be moved without changing any references. It must be emphasized that when a reference is passed in anything other than a well controlled context, the full form must always be used.
In the World-Wide Web applications, the context URI is that of the document or object containing a reference. In this case partial URIs can be generated in virtual objects or stored in real objects, without the need for dramatic change if the higher-order parts of a hierarchical naming system are modified. Apart from terseness, this gives greater robustness to practical systems, by enabling information hiding between system components.
The partial form relies on a property of the URI syntax that certain characters ("/") and certain path elements ("..", ".") have a significance reserved for representing a hierarchical space, and must be recognized as such by both clients and servers.
A partial form can be distinguished from an absolute form in that the latter must have a colon and that colon must occur before any slash characters. Systems not requiring partial forms should not use any unencoded slashes in their naming schemes. If they do, absolute URIs will still work, but confusion may result. (See note on Gopher below.)
The rules for the use of a partial name relative to the URI of the context are:
If the scheme parts are different, the whole absolute URI must be given. Otherwise, the scheme is omitted, and:
If the partial URI starts with a non-zero number of consecutive slashes, then everything from the context URI up to (but not including) the first occurrence of exactly the same number of consecutive slashes which has no greater number of consecutive slashes anywhere to the right of it is taken to be the same and so prepended to the partial URL to form the full URL. Otherwise:
The last part of the path of the context URI (anything following the rightmost slash) is removed, and the given partial URI appended in its place, and then:
Within the result, all occurrences of "xxx/../" or "/." are recursively removed, where xxx, ".." and "." are complete path elements.
Formatted and Indexed by: page 10 of 29 instructional media + magic, inc. RFC1630 Web Address: Universal Resource Identifiers in WWW http://sunsite.cnlab-switch.ch/ By: Berners-Lee
Note: Trailing slashes
If a path of the context locator ends in slash, partial URIs are treated differently to the URI with the same path but without a trailing slash. The trailing slash indicates a void segment of the path.
Note: Gopher
The gopher system does not have the concept of relative URIs, and the gopher community currently allows / as data characters in gopher URIs without escaping them to %2F. Relative forms may not in general be used for documents served by gopher servers. If they are used, then WWW software assumes, normally correctly, that in fact they do have hierarchical significance despite the specifications. The use of HTTP rather than gopher protocol is however recommended.
Examples
In the context of URI
magic://a/b/c//d/e/f
the partial URIs would expand as follows:
g magic://a/b/c//d/e/g
/g magic://a/g
//g magic://g
../g magic://a/b/c//d/g
g:h g:h
and in the context of the URI
magic://a/b/c//d/e/
the results would be exactly the same.
Fragment-id
This represents a part of, fragment of, or a sub-function within, an object. Its syntax and semantics are defined by the application responsible for the object, or the specification of the content type of the object. The only definition here is of the allowed characters by which it may be represented in a URL.
Formatted and Indexed by: page 11 of 29 instructional media + magic, inc. RFC1630 Web Address: Universal Resource Identifiers in WWW http://sunsite.cnlab-switch.ch/ By: Berners-Lee
Specific syntaxes for representing fragments in text documents by line and character range, or in graphics by coordinates, or in structured documents using ladders, are suitable for standardization but not defined here.
The fragment-id follows the URL of the whole object from which it is separated by a hash sign (#). If the fragment-id is void, the hash sign may be omitted: A void fragment-id with or without the hash sign means that the URL refers to the whole object.
While this hook is allowed for identification of fragments, the question of addressing of parts of objects, or of the grouping of objects and relationship between continued and containing objects, is not addressed by this document.
Fragment identifiers do NOT address the question of objects which are different versions of a "living" object, nor of expressing the relationships between different versions and the living object.
There is no implication that a fragment identifier refers to anything which can be extracted as an object in its own right. It may, for example, refer to an indivisible point within an object.
Specific Schemes
The mapping for URIs onto some existing standard and experimental protocols is outlined in the BNF syntax definition. Notes on particular protocols follow. These URIs are frequently referred to as URLs, though the exact definition of the term URL is still under discussion (March 1993). The schemes covered are:
http Hypertext Transfer Protocol (examples)
gopher Gopher protocol
mailto Electronic mail address
news Usenet news
telnet, rlogin and tn3270 Reference to interactive sessions
wais Wide Area Information Servers
file Local file access
Formatted and Indexed by: page 12 of 29 instructional media + magic, inc. RFC1630 Web Address: Universal Resource Identifiers in WWW http://sunsite.cnlab-switch.ch/ By: Berners-Lee
The following schemes are proposed as essential to the unification of the web with electronic mail, but not currently (to the author's knowledge) implemented:
mid Message identifiers for electronic mail
cid Content identifiers for MIME body part
The schemes for X.500, network management database, and Whois++ have not been specified and may be the subject of further study. Schemes for Prospero, and restricted NNTP use are not currently implemented as far as the author is aware.
The "urn" prefix is reserved for use in encoding a Uniform Resource Name when that has been developed by the IETF working group.
New schemes may be registered at a later time.
HTTP
The HTTP protocol specifies that the path is handled transparently by those who handle URLs, except for the servers which de-reference them. The path is passed by the client to the server with any request, but is not otherwise understood by the client.
The host details are not passed on to the client when the URL is an HTTP URL which refers to the server in question. In this case the string sent starts with the slash which follows the host details. However, when an HTTP server is being used as a gateway (or "proxy") then the entire URI, whether HTTP or some other scheme, is passed on the HTTP command line. The search part, if present, is sent as part of the HTTP command, and may in this respect be treated as part of the path. No fragmentid part of a WWW URI (the hash sign and following) is sent with the request. Spaces and control characters in URLs must be escaped for transmission in HTTP, as must other disallowed characters.
EXAMPLES
These examples are not part of the specification: they are provided as illustations only. The URI of the "welcome" page to a server is conventionally
http://www.my.work.com/
As the rest of the URL (after the hostname an port) is opaque to the client, it shows great variety but the following are all fairly typical.
Formatted and Indexed by: page 13 of 29 instructional media + magic, inc. RFC1630 Web Address: Universal Resource Identifiers in WWW http://sunsite.cnlab-switch.ch/ By: Berners-Lee
http://www.my.uni.edu/info/matriculation/enroling.html http://info.my.org/AboutUs/Phonebook http://www.library.my.town.va.us/Catalogue/76523471236%2Fwen44--4.98 http://www.my.org/462F4F2D4241522A314159265358979323846
A URL for a server on a different port to 80 looks like
http://info.cern.ch:8000/imaginary/test
A reference to a particular part of a document may, including the fragment identifier, look like
http://www.myu.edu/org/admin/people#andy
in which case the string "#andy" is not sent to the server, but is retained by the client and used when the whole object had been retrieved.
A search on a text database might look like
http://info.my.org/AboutUs/Index/Phonebook?dobbins
and on another database
http://info.cern.ch/RDB/EMP?*%20where%20name%%3Ddobbins
In all cases the client passes the path string to the server uninterpreted, and for the client to deduce anything from
FTP
The ftp: prefix indicates that the FTP protocol is used, as defined in STD 9, RFC 959 or any successor. The port number, if present, gives the port of the FTP server if not the FTP default.
User name and password
The syntax allows for the inclusion of a user name and even a password for those systems which do not use the anonymous FTP convention. The default, however, if no user or password is supplied, will be to use that convention, viz. that the user name is "anonymous" and the password the user's Internet-style mail address.
Formatted and Indexed by: page 14 of 29 instructional media + magic, inc. RFC1630 Web Address: Universal Resource Identifiers in WWW http://sunsite.cnlab-switch.ch/ By: Berners-Lee
Where possible, this mail address should correspond to a usable mail address for the user, and preferably give a DNS host name which resolves to the IP address of the client. Note that servers currently vary in their treatment of the anonymous password.
Path
The FTP protocol allows for a sequence of CWD commands (change working directory) and a TYPE command prior to service commands such as RETR (retrieve) or NLIST (etc.) which actually access a file.
The arguments of any CWD commands are successive segment parts of the URL delimited by slash, and the final segment is suitable as the filename argument to the RETR command for retrieval or the directory argument to NLIST.
For some file systems (Unix in particular), the "/" used to denote the hierarchical structure of the URL corresponds to the delimiter used to construct a file name hierarchy, and thus, the filename will look the same as the URL path. This does NOT mean that the URL is a Unix filename.
Note: Retrieving subsequent URLs from the same host
There is no common hierarchical model to the FTP protocol, so if a directory change command has been given, it is impossible in general to deduce what sequence should be given to navigate to another directory for a second retrieval, if the paths are different. The only reliable algorithm is to disconnect and reestablish the control connection.
Data type
The data content type of a file can only, in the general FTP case, be deduced from the name, normally the suffix of the name. This is not standardized. An alternative is for it to be transferred in information outside the URL. A suitable FTP transfer type (for example binary "I" or text "A") must in turn be deduced from the data content type. It is recommended that conventions for suffixes of public archives be established, but it is outside the scope of this standard.
An FTP URL may optionally specify the FTP data transfer type by which an object is to be retrieved. Most of the methods correspond to the FTP "Data Types" ASCII and IMAGE for the retrieval of a document, as specified in FTP by the TYPE command. One method indicates directory access.
Formatted and Indexed by: page 15 of 29 instructional media + magic, inc. RFC1630 Web Address: Universal Resource Identifiers in WWW http://sunsite.cnlab-switch.ch/ By: Berners-Lee
The data type is specified by a suffix to the URL. Possible suffixes are:
;type =
/ Use FTP directory list commands to read directory
The type code is in the format defined in RFC 959 except that THE SPACE IS OMITTED FROM THE URL.
Transfer Mode
Stream Mode is always used.
Gopher
The gopher URL specifies the host and optionally the port to which the client should connect. This is followed by a slash and a single gopher type code. This type code is used by the client to determine how to interpret the server's reply and is is not for sending to server. The command string to be sent to the server immediately follows the gopher type character. It consists of the gopher selector string followed by any "Gopher plus" syntax, but always omitting the trainling CR LF pair.
When the gopher command string contains characters (such a embedded CR LF and HT characters) not allowed in a URL, these are encoded using the conventional encoding.
Note that some gopher selector strings begin with a copy of the gopher type character, in which case that character will occur twice consecutively. Also note that the gopher selector string may be an empty string since this is how gopher clients refer to the top-level directory on a gopher server.
If the encoded command string (with trailing CR LF stripped) would be void then the gopher type character may be omiited and "1" (ASCII 31 hex) is assumed.
Note that slash "/" in gopher selector strings may not correspond to a level in a hierarchical structure.
Formatted and Indexed by: page 16 of 29 instructional media + magic, inc. RFC1630 Web Address: Universal Resource Identifiers in WWW http://sunsite.cnlab-switch.ch/ By: Berners-Lee
Mailto
This allows a URL to specify an RFC822 addr-spec mail address. Note that use of % , for example as used in forming a gatewayed mail address, requires conversion to %25 in a URL.
News
The news locators refer to either news group names or article message identifiers which must conform to the rules for a Message-Id of RFC 1036 (Horton 1987). A message identifier may be distinguished from a news group name by the presence of the commercial at "@" character. These rules imply that within an article, a reference to a news group or to another article will be a valid URL (in the partial form).
A news URL may be dereferenced using NNTP (RFC 977, Kantor 1986) (The ARTICLE by message-id command ) or using any other protocol for the conveyance of usenet news articles, or by reference to a body of news articles already received.
Note 1:
Among URLs the "news" URLs are anomalous in that they are location-independent. They are unsuitable as URN candidates because the NNTP architecture relies on the expiry of articles and therefore a small number of articles being available at any time. When a news: URL is quoted, the assumption is that the reader will fetch the article or group from his or her local news host. News host names are NOT part of news URLs.
Note 2:
An outstanding problem is that the message identifier is insufficient to allow the retrieval of an expired article, as no algorithm exists for deriving an archive site and file name. The addition of the date and news group set to the article's URL would allow this if a directory existed of archive sites by news group.
Suggested subject of study in conjunction with NNTP working group. Further extension possible may be to allow the naming of subject threads as addressable objects.
Telnet, rlogin, tn3270
The use of URLs to represent interactive sessions is a convenient extension to their uses for objects. This allows access to information systems which only provide an interactive service, and no information server. As information within the service cannot be
Formatted and Indexed by: page 17 of 29 instructional media + magic, inc. RFC1630 Web Address: Universal Resource Identifiers in WWW http://sunsite.cnlab-switch.ch/ By: Berners-Lee
addressed individually or, in general, automatically retrieved, this is a less desirable, though currently common, solution.
URN
The "Universal Resource Name" is currently (March 1993) under development in the IETF. A requirements specification is in preparation. It currently looks as though it will be a short string suitable for encoding in URI syntax, for which case the "urn:" prefix is reserved. The URN shall be encoded precisely as defined in the (future) URN standard, except in that:
If the official description of the URN syntax includes any constant wrapper characters, then they shall not be omitted from the URI encoding of the URN;
If the URN has a hierarchical nature, then the slash delimiter shall be used in the URI encoding;
If the URN has a hierarchical nature, the most significant part shall be encoded on the left in the URI encoding;
Any characters with reserved meanings in the URI syntax shall be escape encoded
These rules of course apply to any URI scheme. It is of course possible that the URN syntax will be chosen such that the URI encoding will be a 1-1 transcription.
An example might be a name such as
urn:/iana/dns/ch/cern/cn/techdoc/94/1642-3
but the reader should refer to the latest URN drafts or specifications.
WAIS
The current WAIS implementation public domain requires that a client know the "type" of a object prior to retrieval. This value is returned along with the internal object identifier in the search response. It has been encoded into the path part of the URL in order to make the URL sufficient for the retrieval of the object.
Within the WAIS world, names do not of course need to be prefixed by "wais:" (by the partial form rules).
Formatted and Indexed by: page 18 of 29 instructional media + magic, inc. RFC1630 Web Address: Universal Resource Identifiers in WWW http://sunsite.cnlab-switch.ch/ By: Berners-Lee
The wpath of a WAIS URL consists of encoded fields of the WAIS identifier, in the same order as inthe WAIS identifier. For each field, the identifier field number is the digits before the equals sign, and the field contents follow, encoded in the conventional encoding, terminated by ";". file
The other URI schemes (except nntp) share the property that they are equally valid at any geographical place.
There is however a real practical requirement to be able to generate a URL for an object in a machine's local file system.
The syntax is similar to the ftp syntax, but in this case the slash is used to donate boundaries between directory levels of a hierarchical file system is used. The "client" software converts the file URL into a file name in the local file name conventions. This allows local files to be treated just as network objects without any necessity to use a network server for access. This may be used for example for defining a user's "home" document in WWW.
There is clearly a danger of confusion that a link made to a local file should be followed by someone on a different system, with unexpected and possibly harmful results. Therefore, the convention is that even a "file" URL is provided with a host part. This allows a client on another system to know that it cannot access the file system, or perhaps to use some other local mecahnism to access the file.
The special value "localhost" is used in the host field to indicate that the filename should really be used on whatever host one is. This for example allows links to be made to files which are distribted on many machines, or to "your unix local password file" subject of course to consistency across the users of the data.
A void host field is equivalent to "localhost".
Message-Id
For systems which include information transferred using mail protocols, there is a need to be able to make cross-references between different items of information, even though, by the nature of mail, those items are only available to a restricted set of people.
Two schemes are defined. The first, "mid:", refers to the STD 11, RFC 822 Message-Id of a mail message. This Identifier is already used in RFC 822 in for example the References and In-Reply-to field.
Formatted and Indexed by: page 19 of 29 instructional media + magic, inc. RFC1630 Web Address: Universal Resource Identifiers in WWW http://sunsite.cnlab-switch.ch/ By: Berners-Lee
The rest of the URL after the "mid:" is the RFC822 msg-id with the constant <> wrapper removed, leaving an identifier whose format in fact happens to be the same as addr-spec format for mailboxes (though the semantics are different).
The use of a "mid" URL implies access to a body of mail already received. If a message has been distributed using NNTP or other usenet protocols over the news system, then the "news:" form should be used.
Content-Id
The second scheme, "cid:", is similar to "mid:", but makes reference to a body part of a MIME message by the value of its content-id field. This allows, for example, a master document being the first part of a multipart/related MIME message to refer to component parts which are transferred in the same message.
Note
Beware however, that content identifiers are only required to be unique within the context of a given MIME message, and so the cid: URL is only meaningful with the context the same MIME message. For a reference outside the message, it would need to be appended to the message-id of the whole message. A syntax for this has not been defined.
Schemes for Further Study
X500
The mapping of x500 names onto URLs is not defined here. A decision is required as to whether "distinguished names" or "user friendly names" (ufn), or both, should be allowed. If any punctuation conversions are needed from the adopted x500 representation (such as the use of slashes between parts of a ufn) they must be defined. This is a subject for study.
WHOIS
This prefix describes the access using the "whois++" scheme in the process of definition. The host name part is the same as for other IP based schemes. The path part can be either a whois handle for a whois object, or it can be a valid whois query string. This is a subject for further study.
Formatted and Indexed by: page 20 of 29 instructional media + magic, inc. RFC1630 Web Address: Universal Resource Identifiers in WWW http://sunsite.cnlab-switch.ch/ By: Berners-Lee
NETWORK MANAGEMENT DATABASE
This is a subject for study.
NNTP
This is an alternative form of reference for news articles, specifically to be used with NNTP servers, and particularly those incomplete server implementations which do not allow retrieval by message identifier. In all other cases the "news" scheme should be used.
The news server name, newsgroup name, and index number of an article within the newsgroup on that particular server are given. The NNTP protocol must be used.
Note 1.
This form of URL is not of global accessability, as typically NNTP servers only allow access from local clients. Note that the article numbers within groups vary from server to server.
This form or URL should not be quoted outside this local area. It should not be used within news articles for wider circulation than the one server. This is a local identifier for a resource which is often available globally, and so is not recommended except in the case in which incomplete NNTP implementations on the local server force its adoption.
Prospero
The Prospero (Neuman, 1991) directory service is used to resolve the URL yielding an access method for the object (which can then itself be represented as a URL if translated). The host part contains a host name or internet address. The port part is optional.
The path part contains a host specific object name and an optional version number. If present, the version number is separated from the host specific object name by the characters "%00" (percent zero zero), this being an escaped string terminator (null). External Prospero links are represented as URLs of the underlying access method and are not represented as Prospero URLs.
Registration of naming schemes
A new naming scheme may be introduced by defining a mapping onto a conforming URL syntax, using a new prefix. Experimental prefixes may be used by mutual agreement between parties, and must start with the
Formatted and Indexed by: page 21 of 29 instructional media + magic, inc. RFC1630 Web Address: Universal Resource Identifiers in WWW http://sunsite.cnlab-switch.ch/ By: Berners-Lee
characters "x-". The scheme name "urn:" is reserved for the work in progress on a scheme for more persistent names.
It is proposed that the Internet Assigned Numbers Authority (IANA) perform the function of registration of new schemes. Any submission of a new URI scheme must include a definition of an algorithm for the retrieval of any object within that scheme. The algorithm must take the URI and produce either a set of URL(s) which will lead to the desired object, or the object itself, in a well-defined or determinable format.
It is recommended that those proposing a new scheme demonstrate its utility and operability by the provision of a gateway which will provide images of objects in the new scheme for clients using an existing protocol. If the new scheme is not a locator scheme, then the properties of names in the new space should be clearly defined. It is likewise recommended that, where a protocol allows for retrieval by URL, that the client software have provision for being configured to use specific gateway locators for indirect access through new naming schemes.
BNF of Generic URI Syntax
This is a BNF-like description of the URI syntax. at the level at which specific schemes are not considered.
A vertical line "|" indicates alternatives, and [brackets] indicate optional parts. Spaces are represented by the word "space", and the vertical line character by "vline". Single letters stand for single letters. All words of more than one letter below are entities described somewhere in this description.
The "generic" production gives a higher level parsing of the same URIs as the other productions. The "national" and "punctuation" characters do not appear in any productions and therefore may not appear in URIs.
fragmentaddress uri [ # fragmentid ]
uri scheme : path [ ? search ]
scheme ialpha
path void | xpalphas [ / path ]
search xalphas [ + search ]
fragmentid xalphas
Formatted and Indexed by: page 22 of 29 instructional media + magic, inc. RFC1630 Web Address: Universal Resource Identifiers in WWW http://sunsite.cnlab-switch.ch/ By: Berners-Lee
xalpha alpha | digit | safe | extra | escape
xalphas xalpha [ xalphas ]
xpalpha xalpha | +
xpalphas xpalpha [ xpalpha ]
ialpha alpha [ xalphas ]
alpha a | b | c | d | e | f | g | h | i | j | k | l | m | n | o | p | q | r | s | t | u | v | w | x | y | z | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z
digit 0 |1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9
safe $ | - | _ | @ | . | &
extra ! | * | " | ' | ( | ) | ,
reserved = | ; | / | # | ? | : | space
escape % hex hex
hex digit | a | b | c | d | e | f | A | B | C | D | E | F
national { | } | vline | [ | ] | \ | ^ | ~
punctuation < | >
void
(end of URI BNF)
BNF for specific URL schemes
This is a BNF-like description of the Uniform Resource Locator syntax. A vertical line "|" indicates alternatives, and [brackets] indicate optional parts. Spaces are represented by the word "space", and the vertical line character by "vline". Single letters stand for single letters. All words of more than one letter below are entities described somewhere in this description.
Formatted and Indexed by: page 23 of 29 instructional media + magic, inc. RFC1630 Web Address: Universal Resource Identifiers in WWW http://sunsite.cnlab-switch.ch/ By: Berners-Lee
The current IETF URI Working Group preference is for the prefixedurl production. (Nov 1993. July 93: url).
The "national" and "punctuation" characters do not appear in any productions and therefore may not appear in URLs.
The "afsaddress" is left in as historical note, but is not a url production.
prefixedurl u r l : url
url httpaddress | ftpaddress | newsaddress | nntpaddress | prosperoaddress | telnetaddress | gopheraddress | waisaddress | mailtoaddress | midaddress | cidaddress
scheme ialpha
httpaddress h t t p : / / hostport [ / path ] [ ? search ]
ftpaddress f t p : / / login / path [ ftptype ]
afsaddress a f s : / / cellname / path
newsaddress n e w s : groupart
nntpaddress n n t p : group / digits
midaddress m i d : addr-spec
cidaddress c i d : content-identifier
mailtoaddress m a i l t o : xalphas @ hostname
waisaddress waisindex | waisdoc
waisindex w a i s : / / hostport / database [ ? search ]
waisdoc w a i s : / / hostport / database / wtype / wpath
wpath digits = path ; [ wpath ]
groupart * | group | article
group ialpha [ . group ]
Formatted and Indexed by: page 24 of 29 instructional media + magic, inc. RFC1630 Web Address: Universal Resource Identifiers in WWW http://sunsite.cnlab-switch.ch/ By: Berners-Lee
article xalphas @ host
database xalphas
wtype xalphas
prosperoaddress prosperolink
prosperolink p r o s p e r o : / / hostport / hsoname [ % 0 0 version [ attributes ] ]
hsoname path
version digits
attributes attribute [ attributes ]
attribute alphanums
telnetaddress t e l n e t : / / login
gopheraddress g o p h e r : / / hostport [/ gtype [ gcommand ] ]
login [ user [ : password ] @ ] hostport
hostport host [ : port ]
host hostname | hostnumber
ftptype A formcode | E formcode | I | L digits
formcode N | T | C
cellname hostname
hostname ialpha [ . hostname ]
hostnumber digits . digits . digits . digits
port digits
gcommand path
path void | segment [ / path ]
segment xpalphas
Formatted and Indexed by: page 25 of 29 instructional media + magic, inc. RFC1630 Web Address: Universal Resource Identifiers in WWW http://sunsite.cnlab-switch.ch/ By: Berners-Lee
search xalphas [ + search ]
user alphanum2 [ user ]
password alphanum2 [ password ]
fragmentid xalphas
gtype xalpha
alphanum2 alpha | digit | - | _ | . | +
xalpha alpha | digit | safe | extra | escape
xalphas xalpha [ xalphas ]
xpalpha xalpha | +
xpalphas xpalpha [ xpalphas ]
ialpha alpha [ xalphas ]
alpha a | b | c | d | e | f | g | h | i | j | k | l | m | n | o | p | q | r | s | t | u | v | w | x | y | z | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z
digit 0 |1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9
safe $ | - | _ | @ | . | & | + | -
extra ! | * | " | ' | ( | ) | ,
reserved = | ; | / | # | ? | : | space
escape % hex hex
hex digit | a | b | c | d | e | f | A | B | C | D | E | F
national { | } | vline | [ | ] | \ | ^ | ~
punctuation < | >
digits digit [ digits ]
Formatted and Indexed by: page 26 of 29 instructional media + magic, inc. RFC1630 Web Address: Universal Resource Identifiers in WWW http://sunsite.cnlab-switch.ch/ By: Berners-Lee
alphanum alpha | digit
alphanums alphanum [ alphanums ]
void
(end of URL BNF)
References
Alberti, R., et.al., "Notes on the Internet Gopher Protocol", University of Minnesota, December 1991,
Berners-Lee, T., "Hypertext Transfer Protocol (HTTP)", CERN, December 1991, as updated from time to time,
Crocker, D., "Standard for ARPA Internet Text Messages" STD 11, RFC 822, UDel, August 1982.
Davis, F, et al., "WAIS Interface Protocol: Prototype Functional Specification", Thinking Machines Corporation, April 23, 1990.
International Standards Organization, Information and Documentation - Search and Retrieve Application Protocol Specification for open Systems Interconnection, ISO-10163.
Horton, M., and R. Adams, "Standard for Interchange of USENET messages", RFC 1036, AT&T Bell Laboratories, Center for Seismic Studies, December 1987.
Huitema, C., "Naming: strategies and techniques", Computer Networks and ISDN Systems 23 (1991) 107-110.
Kahle, B., "Document Identifiers, or International Standard Book Numbers for the Electronic Age",
Kantor, B., and P. Lapsley, Kantor, B., and P. Lapsley, "Network News Transfer Protocol", RFC 977, UC San Diego & UC Berkeley, February 1986.
Kunze, J., "Requirements for URLs", Work in Progress.
Formatted and Indexed by: page 27 of 29 instructional media + magic, inc. RFC1630 Web Address: Universal Resource Identifiers in WWW http://sunsite.cnlab-switch.ch/ By: Berners-Lee
Lynch, C., Coalition for Networked Information: "Workshop on ID and Reference Structures for Networked Information", November 1991. See
Mockapetris, P., "Domain Names - Concepts and Facilities", STD 13, RFC 1034, USC/Information Sciences Institute, November 1987,
Neuman, B. Clifford, "Prospero: A Tool for Organizing Internet Resources", Electronic Networking: Research, Applications and Policy, Vol 1 No 2, Meckler Westport CT USA, 1992. See also
Postel, J., and J. Reynolds, "File Transfer Protocol (FTP)", STD 9, RFC 959, USC/Information Sciences Institute, October 1985.
Sollins, K., and L. Masinter, "Requiremnets for URNs", Work in Progress.
Yeong, W., "Towards Networked Information Retrieval", Technical report 91-06-25-01, June 1991, Performance Systems International, Inc.
Yeong, W., "Representing Public Archives in the Directory", Work in Progress, November 1991, now expired.
Security Considerations
Security issues are not discussed in this memo.
Author's Address
Tim Berners-Lee World-Wide Web project CERN 1211 Geneva 23, Switzerland
Phone: +41 (22)767 3755 Fax: +41 (22)767 7155 EMail: [email protected]
Formatted and Indexed by: page 28 of 29 instructional media + magic, inc. RFC1630 Web Address: Universal Resource Identifiers in WWW http://sunsite.cnlab-switch.ch/ By: Berners-Lee
_
Formatted and Indexed by: page 29 of 29 instructional media + magic, inc. RFC1729 Web Address: Using the Z39.50 in the Internet Environment http://sunsite.cnlab-switch.ch/ By: C. Lynch
Network Working Group C. Lynch Request for Comments: University of California Category: Informational Office of the President December 1994
Using the Z39.50 Information Retrieval Protocol in the Internet Environment
Status of this Memo
This memo provides information for the Internet community. This memo does not specify an Internet standard of any kind. Distribution of this memo is unlimited.
Summary
This memo describes an approach to the implementation of the ANSI/NISO Z39.50-1992 Standard for Information Retrieval in the TCP/IP environment which is currently in wide use by the Z39.50 implementor community.
Introduction
Z39.50 is a US national standard defining a protocol for computer- to-computer information retrieval that was first adopted in 1988 [1] and extensively revised in 1992 [2]. It was developed by the National Information Standards Organization (NISO), an ANSI-accredited standards development body that serves the publishing, library, and information services communities. The closely related international standard, ISO 10162 (service definition) [3] and 10163 (protocol) [4], colloquially known as Search and Retrieve or SR, reached full International Standard (IS) status in 1991. Work is ongoing within ISO Technical Committee 46 Working Group 4 Subgroup 4 to progress various extensions to SR through the international standards process. The international standard is essentially a compatible subset of the current US Z39.50-1992 standard. Z39.50 is an applications layer protocol within the OSI reference model, which assumes the presence of lower-level OSI services (in particular, the presentation layer [5]) and of the OSI Association Control Service Element (ACSE) [6] within the application layer.
Many institutions implementing this protocol chose, for various reasons, to layer the protocol directly over TCP/IP rather than to implement it in an OSI environment or to use the existing techniques that provide full OSI services at and above the OSI Transport layer on top of TCP connections (as defined in RFC 1006 [7] and implemented, for example, in the ISO Development Environment
Formatted and Indexed by: page 1 of 8 instructional media + magic, inc. RFC1729 Web Address: Using the Z39.50 in the Internet Environment http://sunsite.cnlab-switch.ch/ By: C. Lynch
software). These reasons included concerns about the size and complexity of OSI implementations, the lack of availability of mature OSI software for the full range of computing environments in use at these institutions, and the perception of relative instability of the architectural structures within the OSI applications layer (as opposed to specific application layer protocols such as Z39.50 itself). Most importantly, some of these institutions were concerned that the complexity introduced by the OSI upper layers would outweigh the relatively meager return in functionality that they were likely to gain. Thus, for better or worse, the decision was taken to implement the Z39.50 protocol directly on top of TCP (with the understanding that this decision might be revisited at some point in the future).
During 1991-1993, a group of implementing institutions agreed to participate in the Z39.50 Interoperability Testbed project (sometimes referred to by the acronym "ZIT") under the auspices of the Coalition for Networked Information (CNI). Their primary objective was to encourage the development of many interoperable Z39.50 implementations running over TCP/IP on the Internet. By mid-1993, a number of independent Z39.50 implementations were operational and able to interoperate across the Internet.
The Library of Congress, in its role as the Z39.50 Maintenance Agency for NISO, maintains a registry of the implementors [8], which includes members of the Z39.50 interoperability testbed.
This document describes implementation decisions by current implementors of Z39.50 in the Internet environment. These have been proven within the ZIT project and are being used by most of the members of the Z39.50 Implementors' Group (ZIG), an informal group that meets quarterly to discuss implementation and interoperability issues and to develop extensions to the Z39.50 protocol targeted for inclusion in future versions of the standard. Intended as a guide for other implementors who seek to develop interoperable Z39.50 implementations running over TCP/IP, this document focuses on issues related to TCP/IP, and it does not address other potential interoperability problems or agreements that have been reached among the implementors to address these problems. It does include a few notes about extensions to the existing Version 2 protocol that are being used in the implementor community which have interoperability implications. Potential implementors of Z39.50 should subscribe to the Z3950IW LISTSERV [9] to obtain information specific to the Z39.50 protocol and extensions under development as well as details of current implementations.
Formatted and Indexed by: page 2 of 8 instructional media + magic, inc. RFC1729 Web Address: Using the Z39.50 in the Internet Environment http://sunsite.cnlab-switch.ch/ By: C. Lynch
Except where otherwise noted, the version of Z39.50 discussed here is ANSI/NISO Z39.50-1992, sometimes called Z39.50 Version 2 (the obsolete original version is referred to as Z39.50-1988 or Z39.50 Version 1). The approach defined should also be applicable, perhaps with some minor changes, to future versions of the Z39.50 protocol, and specifically to Version 3 which is currently under development. This document will probably be updated to address new versions of the base Z39.50 protocol as they become stable.
Encoding
The Z39.50 standard specifies its application protocol data units (APDUs) in Abstract Syntax Notation One (ASN.1) [10]. These APDUs include EXTERNAL references to other ASN.1 and non-ASN.1 objects such as those defining record transfer syntaxes to be used in a given application association.
The standard Basic Encoding Rules (BER) [11] are applied to the ASN.1 structures defined by the Z39.50 protocol to produce a byte stream that can be transmitted across a TCP/IP connection. The only restriction on the use of BER to produce this byte stream is that direct, rather than indirect, references must be used for EXTERNAL objects. This is necessary because there is no presentation context in the TCP/IP environment to support indirect reference. A Z39.50 implementation developed according to this specification and running over TCP/IP should produce a valid byte stream according to the Z39.50 standard, in the sense that the same byte stream could be passed to an OSI implementation. However, not all byte streams that can be produced by applying BER to the APDUs specified in the Z39.50 standard in an OSI environment will be legitimate under this specification for the TCP/IP environment; this specification defines a subset of the possible byte streams valid in a pure OSI environment which excludes those using indirect reference for EXTERNAL objects.
All other BER features should be tolerated by Z39.50 implementations running over TCP/IP, including the ability to accept indefinite length encodings, although it is preferable that implementations do not generate such encodings since they have caused problems for some ASN.1/BER parsers. It should also be noted that at least to the best of the author's knowledge, there are no implementations at present that use ASN.1/BER representations of floating point numbers; instead, integers with scaling factors have been used for these purposes. It should also be noted that Z39.50 version 2 does not really address character set encoding issues; these questions, and their interactions with ASN.1/BER support for multiple character sets, are under active discussion as part of the effort to develop Z39.50 version 3.
Formatted and Indexed by: page 3 of 8 instructional media + magic, inc. RFC1729 Web Address: Using the Z39.50 in the Internet Environment http://sunsite.cnlab-switch.ch/ By: C. Lynch
Connection
In the Internet environment, TCP Port 210 has been assigned to Z39.50 by the Internet Assigned Numbers Authority [12]. To initiate a Z39.50 connection to a server in the TCP/IP environment, a client simply opens a TCP connection to port 210 on the server and then, as soon as the TCP connection is established, transmits a Z39.50 INIT APDU using the BER encoding of that INIT APDU as described above.
Implementors should be aware that there is a substantial installed base of implementations of the Wide Area Information Server (WAIS) system. The original versions of this software employed Z39.50 Version 1 with some extensions. Z39.50 Version 1 did not use BER encoding and Z39.50 Version 1 INIT APDUs look very different from the INIT APDUs of Z39.50 Version 2. Implementations of Z39.50 should at least be prepared to reject gracefully WAIS-type INIT APDUs. Some implementations recognize such INIT APDUs and revert to the Z39.50 Version 1 variant used in WAIS upon encountering them, thus providing backwards compatibility with the existing base of WAIS clients and; the usual means of checking for a WAIS, as opposed to Z39.50 Version 2, client is to see if the first byte sent on the connection is an ASCII zero, which indicates a WAIS client. (In version 1 of WAIS, bytes 0-9 of the first PDU contain an ASCII packet length; the lower case ASCII string "wais" appears starting at byte 12.) Work is currently underway to specify a WAIS profile for use with Z39.50 version 2 [13]; it is expected that this will be issued as a Z39.50 Applications Profile through the NIST OIW Library Automation Special Interest Group. This profile is expected to be compatible with the layering defined in this RFC.
Service Mappings
The Z39.50 standard maps Z39.50 services onto a variety of association control and presentation layer services. Connection establishment has already been discussed. The other two association control services that are relevant to Z39.50 are ABORT and RELEASE. The mapping of the RELEASE service to a standard TCP CLOSE is straightforward. The Z39.50 protocol itself does not, in the current version, include a Z39.50 CLOSE APDU. When the client has completed its interaction with the server, it calls the IR-RELEASE service, which is directly mapped to association control's orderly association release. In the TCP/IP environment, the client should simply initiate a TCP CLOSE. The mapping for association abort is more complex, partially because some TCP/IP implementations cannot distinguish a TCP reset from the other side of the connection from other events. To accomplish an abort (that is, a mapping of the IR-ABORT service in the Z39.50 protocol) in the TCP/IP environment, client or server need only terminate the TCP connection either via TCP ABORT or TCP CLOSE.
Formatted and Indexed by: page 4 of 8 instructional media + magic, inc. RFC1729 Web Address: Using the Z39.50 in the Internet Environment http://sunsite.cnlab-switch.ch/ By: C. Lynch
Real-world implementations need to be prepared to deal with both TCP ABORT and CLOSE anyway, so this approach presents no additional problems, other than the somewhat ambiguous nature of the type of association termination.
It is expected that Z39.50 Version 3 will include a termination service which will involve an exchange of Z39.50 CLOSE APDUs, followed by an association RELEASE (which would presumably, in the Internet environment, be mapped to a TCP CLOSE). This new termination service is expected to support both graceful and abrupt termination. Of course, robust implementations will still need to be prepared to encounter TCP CLOSE or ABORT.
Service mappings for the transmission of data by client and server (to the presentation layer P-DATA service) are trivial: They are simply mapped to TCP transmit and receive operations. TCP facilities such as expedited data are not used by Z39.50 in a TCP environment.
Contexts
At the point when the TCP connection is established on TCP port 210, client and server should both assume that the application context given in Appendices A and B of the Z39.50-1992 standard are in place. These are the ASN.1 definitions of the Z39.50 APDUs and the transfer syntax defined by applying the BER to these APDUs.
Implementations can reasonably expect that the diagnostic set BIB-1 is supported, and, if resource control is being used, the resource report format BIB-1 is supported as well.
In the absence of a presentation negotiation mechanism, clients and servers should be cautious about using alternative attribute sets, diagnostic record formats, resource report formats, or other objects defined by optional EXTERNALs within the Z39.50 ASN.1, such as authentication parameters, unless there is known to be prior agreement to support them. Of course, either participant in an association can reference such an object by object ID in an APDU, but there is no guarantee that the other partner in the association will be able to understand it. Robust implementations should be prepared to encounter unknown or unsupported object IDs and generate appropriate diagnostics. Over time, the default, commonly known pool of object IDs may be expanded (for example, to support authentication parameters).
Implementors should refer to the document [14] issued by the Z39.50 maintenance agency in June 1992 for more details on the assumed contexts and object identifiers.
Formatted and Indexed by: page 5 of 8 instructional media + magic, inc. RFC1729 Web Address: Using the Z39.50 in the Internet Environment http://sunsite.cnlab-switch.ch/ By: C. Lynch
Record syntaxes present a serious practical problem. In the OSI environment, the partners in a Z39.50 association are assumed to agree, either through presentation negotiation as part of association establishment, or later, dynamically, as part of the PRESENT process (through the use of the alter presentation context function at the presentation layer), on which record syntaxes the two entities commonly know. There is a preferred record syntax parameter that can be supplied by the client to guide this negotiation. A number of registered record syntaxes exist; some are based on ASN.1 and others use formats such as the MARC standard for the interchange of machine readable cataloging records which predate ASN.1, but are widely implemented. In the TCP/IP environment, if the server cannot supply the record in the preferred syntax, it has no guarantee that the client will understand any other syntax in which it might transmit the record back to the client, and has no means of negotiating such syntaxes.
Several proposals have been suggested to solve this problem. One, which will likely be part of Z39.50 Version 3, is to replace the preferred record syntax parameter with a list of prioritized preferred syntaxes supplied by the client, plus a flag indicating whether the server is allowed to substitute a record syntax not on the list provided by the client. The currently proposed ASN.1 for this extension is upwards compatible with Z39.50 Version 2, although the details are still under discussion within the Z39.50 Implementor's Group. As the Version 3 ASN.1 becomes stable in this area, Z39.50 servers are encouraged to accept the extended ASN.1 for generalized preferred record syntax. The extensibility rules for Z39.50 negotiation let clients and servers negotiate the use of Z39.50 Version 2 plus the generalized preferred syntax feature from Version 3. Thus, a client could support the generalized preferred record syntax, propose its use to any server, and, if the server rejects the proposal, revert to the Version 2 preferred syntax feature.
A second alternative (not incompatible with the Version 3 extension) would be to adopt a convention for TCP/IP implementations that the server not return a record in a syntax not on the preferred record syntax list provided by the client. Instead, it would return a diagnostic record indicating that a suitable record transfer syntax was not available. This strategy could be viewed as simply implementing a subset of the Version 3 solution, and should be considered by implementors of servers as a possible interim measure.
Formatted and Indexed by: page 6 of 8 instructional media + magic, inc. RFC1729 Web Address: Using the Z39.50 in the Internet Environment http://sunsite.cnlab-switch.ch/ By: C. Lynch
Other Interoperability Issues
Version 3 will include an "other" data field in each APDU, which can be used to carry implementation-specific extensions to the protocol. A number of implementations are already employing this field, and interoperable implementations might be wise to include code which at least ignores the presence of such fields rather than considering their presence an error (in contravention of the standard).
References
[1] National Information Standards Organization (NISO). American National Standard Z39.50, Information Retrieval Service Definition and Protocol Specifications for Library Applications (New Brunswick, NJ: Transaction Publishers; 1988).
[2] ANSI/NISO Z39.50-1992 (version 2) Information Retrieval Service and Protocol: American National Standard, Information Retrieval Application Service Definition and Protocol Specification for Open Systems Interconnection, 1992.
[3] ISO 10162 International Organization for Standardization (ISO). Documentation -- Search and Retrieve Service Definition, 1992.
[4] ISO 10163 International Organization for Standardization (ISO). Documentation -- Search and Retrieve Protocol Definition. 1992.
[5] ISO 8822 - Information Processing Systems - Open Systems Interconnection - Connection Oriented Presentation Service Definition, 1988.
[6] ISO 8649 - Information Processing Systems - Open Systems Interconnection - Service Definition for the Association Control Service Element, 1987. See also ISO 8650 - Information Processing Systems - Open Systems Interconnection - Protocol Specification for the Association Control Service Element, 1987.
[7] Rose, M., and D. Cass, "ISO Transport Layer Services on Top of the TCP, Version 3", STD 35, RFC 1006, Northrop Research and Technology Center, May 1987.
[8] Registry of Z39.50 Implementors, available from the Z39.50 Maintenance Agency (Ray Denenberg, [email protected])
Formatted and Indexed by: page 7 of 8 instructional media + magic, inc. RFC1729 Web Address: Using the Z39.50 in the Internet Environment http://sunsite.cnlab-switch.ch/ By: C. Lynch
[9] To subscribe to the Z39.50 Implementor's Workshop list send the message: SUB Z3950IW yourname to: [email protected] (or NERVM.BITNET). Current drafts of the Version 3 Protocol document are available through the Library of Congress GOPHER server, MARVEL.LOC.GOV.
[10] ISO 8824 - Information Processing Systems - Open Systems Interconnection - Specifications for Abstract Syntax Notation One (ASN.1), 1987
[11] ISO 8825 Information Processing Systems - Open Systems Interconnection - Specification of Basic Encoding Rules for Abstract Syntax Notation One (ASN.1) 1987
[12] Reynolds, J., and J. Postel, "Assigned Numbers", STD 2, RFC 1700, USC/Information Sciences Institute, October 1994.
[13] WAIS Profile of Z39.50 Version 2, Revision 1.4, April 26, 1994, available from WAIS Inc.
[14] Registration of Z39.50 OSI Object Identifiers (Z39.50-MA-024), available from the Z39.50 Maintenance Agency (Ray Denenberg, [email protected]).
Security Considerations
This document does not discuss security considerations. However, it should be noted that the Z39.50 protocol includes mechanisms for authentication and security that implementors should review.
Author's Address
Clifford A. Lynch University of California, Office of the President 300 Lakeside Drive, 8th Floor Oakland, CA 94612-3550
Phone: (510) 987-0522 EMail: [email protected]
Formatted and Indexed by: page 8 of 8 instructional media + magic, inc. RFC1737 Web Address: Requirements for Uniform Resource Names http://sunsite.cnlab-switch.ch/ By: Sollins & Masinter
Network Working Group K. Sollins Request for Comments: MIT/LCS Category: Informational L. Masinter Xerox Corporation December 1994
Functional Requirements for Uniform Resource Names
Status of this Memo
This memo provides information for the Internet community. This memo does not specify an Internet standard of any kind. Distribution of this memo is unlimited.
1. Introduction
This document specifies a minimum set of requirements for a kind of Internet resource identifier known as Uniform Resource Names (URNs). URNs fit within a larger Internet information architecture, which in turn is composed of, additionally, Uniform Resource Characteristics (URCs), and Uniform Resource Locators (URLs). URNs are used for identification, URCs for including meta-information, and URLs for locating or finding resources. It is provided as a basis for evaluating standards for URNs. The discussions of this work have occurred on the mailing list [email protected] and at the URI Working Group sessions of the IETF.
The requirements described here are not necessarily exhaustive; for example, there are several issues dealing with support for replication of resources and with security that have been discussed; however, the problems are not well enough understood at this time to include specific requirements in those areas here.
Within the general area of distributed object systems design, there are many concepts and designs that are discussed under the general topic of "naming". The URN requirements here are for a facility that addresses a different (and, in general, more stringent) set of needs than are frequently the domain of general object naming.
The requirements for Uniform Resource Names fit within the overall architecture of Uniform Resource Identification. In order to build applications in the most general case, the user must be able to discover and identify the information, objects, or what we will call in this architecture resources, on which the application is to operate. Beyond this statement, the URI architecture does not define "resource." As the network and interconnectivity grow, the ability to make use of remote, perhaps independently managed, resources will
Formatted and Indexed by: page 1 of 7 instructional media + magic, inc. RFC1737 Web Address: Requirements for Uniform Resource Names http://sunsite.cnlab-switch.ch/ By: Sollins & Masinter
become more and more important. This activity of discovering and utilizing resources can be broken down into those activities where one of the primary constraints is human utility and facility and those in which human involvement is small or nonexistent. Human naming must have such characteristics as being both mnemonic and short. Humans, in contrast with computers, are good at heuristic disambiguation and wide variability in structure. In order for computer and network based systems to support global naming and access to resources that have perhaps an indeterminate lifetime, the flexibility and attendant unreliability of human-friendly names should be translated into a naming infrastructure more appropriate for the underlying support system. It is this underlying support system that the Internet Information Infrastructure Architecture (IIIA) is addressing.
Within the IIIA, several sorts of information about resources are specified and divided among different sorts of structures, along functional lines. In order to access information, one must be able to discover or identify the particular information desired, determined both how and where it might be used or accessed. The partitioning of the functionality in this architecture is into uniform resource names (URN), uniform resource characteristics (URC), and uniform resource locators (URL). A URN identifies a resource or unit of information. It may identify, for example, intellectual content, a particular presentation of intellectual content, or whatever a name assignment authority determines is a distinctly namable entity. A URL identifies the location or a container for an instance of a resource identified by a URN. The resource identified by a URN may reside in one or more locations at any given time, may move, or may not be available at all. Of course, not all resources will move during their lifetimes, and not all resources, although identifiable and identified by a URN will be instantiated at any given time. As such a URL is identifying a place where a resource may reside, or a container, as distinct from the resource itself identified by the URN. A URC is a set of meta-level information about a resource. Some examples of such meta-information are: owner, encoding, access restrictions (perhaps for particular instances), cost.
With this in mind, we can make the following statement:
o The purpose or function of a URN is to provide a globally unique, persistent identifier used for recognition, for access to characteristics of the resource or for access to the resource itself.
Formatted and Indexed by: page 2 of 7 instructional media + magic, inc. RFC1737 Web Address: Requirements for Uniform Resource Names http://sunsite.cnlab-switch.ch/ By: Sollins & Masinter
More specifically, there are two kinds of requirements on URNs: requirements on the functional capabilities of URNs, and requirements on the way URNs are encoded in data streams and written communications.
2. Requirements for functional capabilities
These are the requirements for URNs' functional capabilities:
o Global scope: A URN is a name with global scope which does not imply a location. It has the same meaning everywhere.
o Global uniqueness: The same URN will never be assigned to two different resources.
o Persistence: It is intended that the lifetime of a URN be permanent. That is, the URN will be globally unique forever, and may well be used as a reference to a resource well beyond the lifetime of the resource it identifies or of any naming authority involved in the assignment of its name.
o Scalability: URNs can be assigned to any resource that might conceivably be available on the network, for hundreds of years.
o Legacy support: The scheme must permit the support of existing legacy naming systems, insofar as they satisfy the other requirements described here. For example, ISBN numbers, ISO public identifiers, and UPC product codes seem to satisfy the functional requirements, and allow an embedding that satisfies the syntactic requirements described here.
o Extensibility: Any scheme for URNs must permit future extensions to the scheme.
o Independence: It is solely the responsibility of a name issuing authority to determine the conditions under which it will issue a name.
o Resolution: A URN will not impede resolution (translation into a URL, q.v.). To be more specific, for URNs that have corresponding URLs, there must be some feasible mechanism to translate a URN to a URL.
3. Requirements for URN encoding
In addition to requirements on the functional elements of the URNs, there are requirements for how they are encoded in a string:
Formatted and Indexed by: page 3 of 7 instructional media + magic, inc. RFC1737 Web Address: Requirements for Uniform Resource Names http://sunsite.cnlab-switch.ch/ By: Sollins & Masinter
o Single encoding: The encoding for presentation for people in clear text, electronic mail and the like is the same as the encoding in other transmissions.
o Simple comparison: A comparison algorithm for URNs is simple, local, and deterministic. That is, there is a single algorithm for comparing two URNs that does not require contacting any external server, is well specified and simple.
o Human transcribability: For URNs to be easily transcribable by humans without error, they should be short, use a minimum of special characters, and be case insensitive. (There is no strong requirement that it be easy for a human to generate or interpret a URN; explicit human-accessible semantics of the names is not a requirement.) For this reason, URN comparison is insensitive to case, and probably white space and some punctuation marks.
o Transport friendliness: A URN can be transported unmodified in the common Internet protocols, such as TCP, SMTP, FTP, Telnet, etc., as well as printed paper.
o Machine consumption: A URN can be parsed by a computer.
o Text recognition: The encoding of a URN should enhance the ability to find and parse URNs in free text.
4. Implications
For a URN specification to be acceptible, it must meet the previous requirements. We draw a set of conclusions, listed below, from those requirements; a specification that satisfies the requirments without meetings these conclusions is deemed acceptable, although unlikely to occur.
o To satisfy the requirements of uniqueness and scalability, name assignment is delegated to naming authorities, who may then assign names directly or delegate that authority to sub-authorities. Uniqueness is guaranteed by requiring each naming authority to guarantee uniqueness. The names of the naming authorities themselves are persistent and globally unique and top level authorities will be centrally registered.
o Naming authorities that support scalable naming are encouraged, but not required. Scalability implies that a scheme for devising names may be scalable both at its terminators as well as within the structure; e.g., in a hierarchical naming scheme, a naming authority might have an extensible mechanism for adding new sub-registries.
Formatted and Indexed by: page 4 of 7 instructional media + magic, inc. RFC1737 Web Address: Requirements for Uniform Resource Names http://sunsite.cnlab-switch.ch/ By: Sollins & Masinter
o It is strongly recommended that there be a mapping between the names generated by each naming authority and URLs. At any specific time there will be zero or more URLs into which a particular URN can be mapped. The naming authority itself need not provide the mapping from URN to URL.
o For URNs to be transcribable and transported in mail, it is necessary to limit the character set usable in URNs, although there is not yet consensus on what the limit might be.
In assigning names, a name assignment authority must abide by the preceding constraints, as well as defining its own criteria for determining the necessity or indication of a new name assignment.
5. Other considerations
There are three issues about which this document has intentionally not taken a position, because it is believed that these are issues to be decided by local determination or other services within an information infrastructure. These issues are equality of resources, reflection of visible semantics in a URN, and name resolution.
One of the ways in which naming authorities, the assigners of names, may choose to make themselves distinctive is by the algorithms by which they distinguish or do not distinguish resources from each other. For example, a publisher may choose to distinguish among multiple printings of a book, in which minor spelling and typographical mistakes have been made, but a library may prefer not to make that distinction. Furthermore, no one algorithm for testing for equality is likely to applicable to all sorts of information. For example, an algorithm based on testing the equality of two books is unlikely to be useful when testing the equality of two spreadsheets. Thus, although this document requires that any particular naming authority use one algorithm for determining whether two resources it is comparing are the same or different, each naming authority can use a different such algorithm and a naming authority may restrict the set of resources it chooses to identify in any way at all.
A naming authority will also have some algorithm for actually choosing a name within its namespace. It may have an algorithm that actually embeds in some way some knowledge about the resource. In turn, that embedding may or may not be made public, and may or may not be visible to potential clients. For example, an unreflective URN, simply provides monotonically increasing serial numbers for resources. This conveys nothing other than the identity determined by the equality testing algorithm and an ordering of name assignment by this server. It carries no information about the resource itself.
Formatted and Indexed by: page 5 of 7 instructional media + magic, inc. RFC1737 Web Address: Requirements for Uniform Resource Names http://sunsite.cnlab-switch.ch/ By: Sollins & Masinter
An MD5 of the resource at some point, in and of itself may be reflective of its contents, and, in fact, the naming authority may be perfectly willing to publish the fact that it is using MD5, but if the resource is mutable, it still will be the case that any potential client cannot do much with the URN other than check for equality. If, in contrast, a URN scheme has much in common with the assignment ISBN numbers, the algorithm for assigning them is public and by knowing it, given a particular ISBN number, one can learn something more about the resource in question. This full range of possibilities is allowed according to this requirements document, although it is intended that naming authorities be discouraged from making accessible to clients semantic information about the resource, on the assumption that that may change with time and therefore it is unwise to encourage people in any way to depend on that semantics being valid.
Last, this document intentionally does not address the problem of name resolution, other than to recommend that for each naming authority a name translation mechanism exist. Naming authorities assign names, while resolvers or location services of some sort assist or provide URN to URL mapping. There may be one or many such services for the resources named by a particular naming authority. It may also be the case that there are generic ones providing service for many resources of differing naming authorities. Some may be authoritative and others not. Some may be highly reliable or highly available or highly responsive to updates or highly focussed by other criteria such as subject matter. Of course, it is also possible that some naming authorities will also act as resolvers for the resources they have named. This document supports and encourages third party and distributed services in this area, and therefore intentionally makes no statements about requirements of URNs or naming authorities on resolvers.
Security Considerations
Applications that require translation from names to locations, and the resources themselves may require the resources to be authenticated. It seems generally that the information about the authentication of either the name or the resource to which it refers should be carried by separate information passed along with the URN rather than in the URN itself.
Formatted and Indexed by: page 6 of 7 instructional media + magic, inc. RFC1737 Web Address: Requirements for Uniform Resource Names http://sunsite.cnlab-switch.ch/ By: Sollins & Masinter
Authors' Addresses
Larry Masinter Xerox Palo Alto Research Center 3333 Coyote Hill Road Palo Alto, CA 94304
Phone: (415) 812-4365 Fax: (415) 812-4333 EMail: [email protected]
Karen Sollins MIT Laboratory for Computer Science 545 Technology Square Cambridge, MA 02139
Voice: (617) 253-6006 Phone: (617) 253-2673 EMail: [email protected]
_
Formatted and Indexed by: page 7 of 7 instructional media + magic, inc. RFC1738 Web Address: Uniform Resource Locators (URL) http://sunsite.cnlab-switch.ch/ By: Berners-Lee, Masinter & McCahill
Network Working Group T. Berners-Lee Request for Comments: CERN Category: Standards Track L. Masinter Xerox Corporation M. McCahill University of Minnesota Editors December 1994
Uniform Resource Locators (URL)
Status of this Memo
This document specifies an Internet standards track protocol for the Internet community, and requests discussion and suggestions for improvements. Please refer to the current edition of the "Internet Official Protocol Standards" (STD 1) for the standardization state and status of this protocol. Distribution of this memo is unlimited.
Abstract
This document specifies a Uniform Resource Locator (URL), the syntax and semantics of formalized information for location and access of resources via the Internet.
1. Introduction
This document describes the syntax and semantics for a compact string representation for a resource available via the Internet. These strings are called "Uniform Resource Locators" (URLs).
The specification is derived from concepts introduced by the World- Wide Web global information initiative, whose use of such objects dates from 1990 and is described in "Universal Resource Identifiers in WWW", RFC 1630. The specification of URLs is designed to meet the requirements laid out in "Functional Requirements for Internet Resource Locators" [12].
This document was written by the URI working group of the Internet Engineering Task Force. Comments may be addressed to the editors, or to the URI-WG
Formatted and Indexed by: page 1 of 25 instructional media + magic, inc. RFC1738 Web Address: Uniform Resource Locators (URL) http://sunsite.cnlab-switch.ch/ By: Berners-Lee, Masinter & McCahill
2. General URL Syntax
Just as there are many different methods of access to resources, there are several schemes for describing the location of such resources.
The generic syntax for URLs provides a framework for new schemes to be established using protocols other than those defined in this document.
URLs are used to `locate' resources, by providing an abstract identification of the resource location. Having located a resource, a system may perform a variety of operations on the resource, as might be characterized by such words as `access', `update', `replace', `find attributes'. In general, only the `access' method needs to be specified for any URL scheme.
2.1. The main parts of URLs
A full BNF description of the URL syntax is given in Section 5.
In general, URLs are written as follows:
A URL contains the name of the scheme being used (
Scheme names consist of a sequence of characters. The lower case letters "a"--"z", digits, and the characters plus ("+"), period ("."), and hyphen ("-") are allowed. For resiliency, programs interpreting URLs should treat upper case letters as equivalent to lower case in scheme names (e.g., allow "HTTP" as well as "http").
2.2. URL Character Encoding Issues
URLs are sequences of characters, i.e., letters, digits, and special characters. A URLs may be represented in a variety of ways: e.g., ink on paper, or a sequence of octets in a coded character set. The interpretation of a URL depends only on the identity of the characters used.
In most URL schemes, the sequences of characters in different parts of a URL are used to represent sequences of octets used in Internet protocols. For example, in the ftp scheme, the host name, directory name and file names are such sequences of octets, represented by parts of the URL. Within those parts, an octet may be represented by
Formatted and Indexed by: page 2 of 25 instructional media + magic, inc. RFC1738 Web Address: Uniform Resource Locators (URL) http://sunsite.cnlab-switch.ch/ By: Berners-Lee, Masinter & McCahill
the chararacter which has that octet as its code within the US-ASCII [20] coded character set.
In addition, octets may be encoded by a character triplet consisting of the character "%" followed by the two hexadecimal digits (from "0123456789ABCDEF") which forming the hexadecimal value of the octet. (The characters "abcdef" may also be used in hexadecimal encodings.)
Octets must be encoded if they have no corresponding graphic character within the US-ASCII coded character set, if the use of the corresponding character is unsafe, or if the corresponding character is reserved for some other interpretation within the particular URL scheme.
No corresponding graphic US-ASCII:
URLs are written only with the graphic printable characters of the US-ASCII coded character set. The octets 80-FF hexadecimal are not used in US-ASCII, and the octets 00-1F and 7F hexadecimal represent control characters; these must be encoded.
Unsafe:
Characters can be unsafe for a number of reasons. The space character is unsafe because significant spaces may disappear and insignificant spaces may be introduced when URLs are transcribed or typeset or subjected to the treatment of word-processing programs. The characters "<" and ">" are unsafe because they are used as the delimiters around URLs in free text; the quote mark (""") is used to delimit URLs in some systems. The character "#" is unsafe and should always be encoded because it is used in World Wide Web and in other systems to delimit a URL from a fragment/anchor identifier that might follow it. The character "%" is unsafe because it is used for encodings of other characters. Other characters are unsafe because gateways and other transport agents are known to sometimes modify such characters. These characters are "{", "}", "|", "\", "^", "~", "[", "]", and "`".
All unsafe characters must always be encoded within a URL. For example, the character "#" must be encoded within URLs even in systems that do not normally deal with fragment or anchor identifiers, so that if the URL is copied into another system that does use them, it will not be necessary to change the URL encoding.
Formatted and Indexed by: page 3 of 25 instructional media + magic, inc. RFC1738 Web Address: Uniform Resource Locators (URL) http://sunsite.cnlab-switch.ch/ By: Berners-Lee, Masinter & McCahill
Reserved:
Many URL schemes reserve certain characters for a special meaning: their appearance in the scheme-specific part of the URL has a designated semantics. If the character corresponding to an octet is reserved in a scheme, the octet must be encoded. The characters ";", "/", "?", ":", "@", "=" and "&" are the characters which may be reserved for special meaning within a scheme. No other characters may be reserved within a scheme.
Usually a URL has the same interpretation when an octet is represented by a character and when it encoded. However, this is not true for reserved characters: encoding a character reserved for a particular scheme may change the semantics of a URL.
Thus, only alphanumerics, the special characters "$-_.+!*'(),", and reserved characters used for their reserved purposes may be used unencoded within a URL.
On the other hand, characters that are not required to be encoded (including alphanumerics) may be encoded within the scheme-specific part of a URL, as long as they are not being used for a reserved purpose.
2.3 Hierarchical schemes and relative links
In some cases, URLs are used to locate resources that contain pointers to other resources. In some cases, those pointers are represented as relative links where the expression of the location of the second resource is in terms of "in the same place as this one except with the following relative path". Relative links are not described in this document. However, the use of relative links depends on the original URL containing a hierarchical structure against which the relative link is based.
Some URL schemes (such as the ftp, http, and file schemes) contain names that can be considered hierarchical; the components of the hierarchy are separated by "/".
Formatted and Indexed by: page 4 of 25 instructional media + magic, inc. RFC1738 Web Address: Uniform Resource Locators (URL) http://sunsite.cnlab-switch.ch/ By: Berners-Lee, Masinter & McCahill
3. Specific Schemes
The mapping for some existing standard and experimental protocols is outlined in the BNF syntax definition. Notes on particular protocols follow. The schemes covered are:
ftp File Transfer protocol http Hypertext Transfer Protocol gopher The Gopher protocol mailto Electronic mail address news USENET news nntp USENET news using NNTP access telnet Reference to interactive sessions wais Wide Area Information Servers file Host-specific file names prospero Prospero Directory Service
Other schemes may be specified by future specifications. Section 4 of this document describes how new schemes may be registered, and lists some scheme names that are under development.
3.1. Common Internet Scheme Syntax
While the syntax for the rest of the URL may vary depending on the particular scheme selected, URL schemes that involve the direct use of an IP-based protocol to a specified host on the Internet use a common syntax for the scheme-specific data:
//
Some or all of the parts "
user An optional user name. Some schemes (e.g., ftp) allow the specification of a user name.
password An optional password. If present, it follows the user name separated from it by a colon.
The user name (and password), if present, are followed by a commercial at-sign "@". Within the user and password field, any ":", "@", or "/" must be encoded.
Formatted and Indexed by: page 5 of 25 instructional media + magic, inc. RFC1738 Web Address: Uniform Resource Locators (URL) http://sunsite.cnlab-switch.ch/ By: Berners-Lee, Masinter & McCahill
Note that an empty user name or password is different than no user name or password; there is no way to specify a password without specifying a user name. E.g.,
host The fully qualified domain name of a network host, or its IP address as a set of four decimal digit groups separated by ".". Fully qualified domain names take the form as described in Section 3.5 of RFC 1034 [13] and Section 2.1 of RFC 1123 [5]: a sequence of domain labels separated by ".", each domain label starting and ending with an alphanumerical character and possibly also containing "-" characters. The rightmost domain label will never start with a digit, though, which syntactically distinguishes all domain names from the IP addresses.
port The port number to connect to. Most schemes designate protocols that have a default port number. Another port number may optionally be supplied, in decimal, separated from the host by a colon. If the port is omitted, the colon is as well.
url-path The rest of the locator consists of data specific to the scheme, and is known as the "url-path". It supplies the details of how the specified resource can be accessed. Note that the "/" between the host (or port) and the url-path is NOT part of the url-path.
The url-path syntax depends on the scheme being used, as does the manner in which it is interpreted.
3.2. FTP
The FTP URL scheme is used to designate files and directories on Internet hosts accessible using the FTP protocol (RFC959).
A FTP URL follow the syntax described in Section 3.1. If :
Formatted and Indexed by: page 6 of 25 instructional media + magic, inc. RFC1738 Web Address: Uniform Resource Locators (URL) http://sunsite.cnlab-switch.ch/ By: Berners-Lee, Masinter & McCahill
3.2.1. FTP Name and Password
A user name and password may be supplied; they are used in the ftp "USER" and "PASS" commands after first making the connection to the FTP server. If no user name or password is supplied and one is requested by the FTP server, the conventions for "anonymous" FTP are to be used, as follows:
The user name "anonymous" is supplied.
The password is supplied as the Internet e-mail address of the end user accessing the resource.
If the URL supplies a user name but no password, and the remote server requests a password, the program interpreting the FTP URL should request one from the user.
3.2.2. FTP url-path
The url-path of a FTP URL has the following syntax:
Where
The url-path is interpreted as a series of FTP commands as follows:
Each of the
If the typecode is "d", perform a NLST (name list) command with
Otherwise, perform a TYPE command with
Within a name or CWD component, the characters "/" and ";" are reserved and must be encoded. The components are decoded prior to their use in the FTP protocol. In particular, if the appropriate FTP sequence to access a particular file requires supplying a string containing a "/" as an argument to a CWD or RETR command, it is
Formatted and Indexed by: page 7 of 25 instructional media + magic, inc. RFC1738 Web Address: Uniform Resource Locators (URL) http://sunsite.cnlab-switch.ch/ By: Berners-Lee, Masinter & McCahill
necessary to encode each "/".
For example, the URL
FTP URLs may also be used for other operations; for example, it is possible to update a file on a remote file server, or infer information about it from the directory listings. The mechanism for doing so is not spelled out here.
3.2.3. FTP Typecode is Optional
The entire ;type=
3.2.4 Hierarchy
For some file systems, the "/" used to denote the hierarchical structure of the URL corresponds to the delimiter used to construct a file name hierarchy, and thus, the filename will look similar to the URL path. This does NOT mean that the URL is a Unix filename.
3.2.5. Optimization
Clients accessing resources via FTP may employ additional heuristics to optimize the interaction. For some FTP servers, for example, it may be reasonable to keep the control connection open while accessing multiple URLs from the same server. However, there is no common hierarchical model to the FTP protocol, so if a directory change command has been given, it is impossible in general to deduce what sequence should be given to navigate to another directory for a second retrieval, if the paths are different. The only reliable algorithm is to disconnect and reestablish the control connection.
Formatted and Indexed by: page 8 of 25 instructional media + magic, inc. RFC1738 Web Address: Uniform Resource Locators (URL) http://sunsite.cnlab-switch.ch/ By: Berners-Lee, Masinter & McCahill
3.3. HTTP
The HTTP URL scheme is used to designate Internet resources accessible using HTTP (HyperText Transfer Protocol).
The HTTP protocol is specified elsewhere. This specification only describes the syntax of HTTP URLs.
An HTTP URL takes the form:
http://
where
Within the
3.4. GOPHER
The Gopher URL scheme is used to designate Internet resources accessible using the Gopher protocol.
The base Gopher protocol is described in RFC 1436 and supports items and collections of items (directories). The Gopher+ protocol is a set of upward compatible extensions to the base Gopher protocol and is described in [2]. Gopher+ supports associating arbitrary sets of attributes and alternate data representations with Gopher items. Gopher URLs accommodate both Gopher and Gopher+ items and item attributes.
3.4.1. Gopher URL syntax
A Gopher URL takes the form:
gopher://
where
Formatted and Indexed by: page 9 of 25 instructional media + magic, inc. RFC1738 Web Address: Uniform Resource Locators (URL) http://sunsite.cnlab-switch.ch/ By: Berners-Lee, Masinter & McCahill
If :
Gopher clients specify which item to retrieve by sending the Gopher selector string to a Gopher server.
Within the
Note that some Gopher
3.4.2 Specifying URLs for Gopher Search Engines
If the URL refers to a search to be submitted to a Gopher search engine, the selector is followed by an encoded tab (%09) and the search string. To submit a search to a Gopher search engine, the Gopher client sends the
3.4.3 URL syntax for Gopher+ items
URLs for Gopher+ items have a second encoded tab (%09) and a Gopher+ string. Note that in this case, the %09
The
To retrieve the data associated with a Gopher+ URL, a client will connect to the server and send the Gopher selector, followed by a tab and the search string (which may be empty), followed by a tab and the Gopher+ commands.
Formatted and Indexed by: page 10 of 25 instructional media + magic, inc. RFC1738 Web Address: Uniform Resource Locators (URL) http://sunsite.cnlab-switch.ch/ By: Berners-Lee, Masinter & McCahill
3.4.4 Default Gopher+ data representation
When a Gopher server returns a directory listing to a client, the Gopher+ items are tagged with either a "+" (denoting Gopher+ items) or a "?" (denoting Gopher+ items which have a +ASK form associated with them). A Gopher URL with a Gopher+ string consisting of only a "+" refers to the default view (data representation) of the item while a Gopher+ string containing only a "?" refer to an item with a Gopher electronic form associated with it.
3.4.5 Gopher+ items with electronic forms
Gopher+ items which have a +ASK associated with them (i.e. Gopher+ items tagged with a "?") require the client to fetch the item's +ASK attribute to get the form definition, and then ask the user to fill out the form and return the user's responses along with the selector string to retrieve the item. Gopher+ clients know how to do this but depend on the "?" tag in the Gopher+ item description to know when to handle this case. The "?" is used in the Gopher+ string to be consistent with Gopher+ protocol's use of this symbol.
3.4.6 Gopher+ item attribute collections
To refer to the Gopher+ attributes of an item, the Gopher URL's Gopher+ string consists of "!" or "$". "!" refers to the all of a Gopher+ item's attributes. "$" refers to all the item attributes for all items in a Gopher directory.
3.4.7 Referring to specific Gopher+ attributes
To refer to specific attributes, the URL's gopher+_string is "!
To refer to several attributes, the gopher+_string consists of the attribute names separated by coded spaces. For example, "!+ABSTRACT%20+SMELL" refers to the +ABSTRACT and +SMELL attributes of an item.
3.4.8 URL syntax for Gopher+ alternate views
Gopher+ allows for optional alternate data representations (alternate views) of items. To retrieve a Gopher+ alternate view, a Gopher+ client sends the appropriate view and language identifier (found in the item's +VIEW attribute). To refer to a specific Gopher+ alternate view, the URL's Gopher+ string would be in the form:
Formatted and Indexed by: page 11 of 25 instructional media + magic, inc. RFC1738 Web Address: Uniform Resource Locators (URL) http://sunsite.cnlab-switch.ch/ By: Berners-Lee, Masinter & McCahill
+
For example, a Gopher+ string of "+application/postscript%20Es_ES" refers to the Spanish language postscript alternate view of a Gopher+ item.
3.4.9 URL syntax for Gopher+ electronic forms
The gopher+_string for a URL that refers to an item referenced by a Gopher+ electronic form (an ASK block) filled out with specific values is a coded version of what the client sends to the server. The gopher+_string is of the form:
+%091%0D%0A+-1%0D%0A
To retrieve this item, the Gopher client sends:
to the Gopher server.
3.5. MAILTO
The mailto URL scheme is used to designate the Internet mailing address of an individual or service. No additional information other than an Internet mailing address is present or implied.
A mailto URL takes the form:
mailto:
where
Note that the percent sign ("%") is commonly used within RFC 822 addresses and must be encoded.
Unlike many URLs, the mailto scheme does not represent a data object to be accessed directly; there is no sense in which it designates an object. It has a different use than the message/external-body type in MIME.
Formatted and Indexed by: page 12 of 25 instructional media + magic, inc. RFC1738 Web Address: Uniform Resource Locators (URL) http://sunsite.cnlab-switch.ch/ By: Berners-Lee, Masinter & McCahill
3.6. NEWS
The news URL scheme is used to refer to either news groups or individual articles of USENET news, as specified in RFC 1036.
A news URL takes one of two forms:
news:
A
If
The news URLs are unusual in that by themselves, they do not contain sufficient information to locate a single resource, but, rather, are location-independent.
3.7. NNTP
The nntp URL scheme is an alternative method of referencing news articles, useful for specifying news articles from NNTP servers (RFC 977).
A nntp URL take the form:
nntp://
where
The
Note that while nntp: URLs specify a unique location for the article resource, most NNTP servers currently on the Internet today are configured only to allow access from local clients, and thus nntp URLs do not designate globally accessible resources. Thus, the news: form of URL is preferred as a way of identifying news articles.
Formatted and Indexed by: page 13 of 25 instructional media + magic, inc. RFC1738 Web Address: Uniform Resource Locators (URL) http://sunsite.cnlab-switch.ch/ By: Berners-Lee, Masinter & McCahill
3.8. TELNET
The Telnet URL scheme is used to designate interactive services that may be accessed by the Telnet protocol.
A telnet URL takes the form:
telnet://
as specified in Section 3.1. The final "/" character may be omitted. If :
This URL does not designate a data object, but rather an interactive service. Remote interactive services vary widely in the means by which they allow remote logins; in practice, the
3.9. WAIS
The WAIS URL scheme is used to designate WAIS databases, searches, or individual documents available from a WAIS database. WAIS is described in [7]. The WAIS protocol is described in RFC 1625 [17]; Although the WAIS protocol is based on Z39.50-1988, the WAIS URL scheme is not intended for use with arbitrary Z39.50 services.
A WAIS URL takes one of the following forms:
wais://
where
The third form designates a particular document within a WAIS database to be retrieved. In this form
Formatted and Indexed by: page 14 of 25 instructional media + magic, inc. RFC1738 Web Address: Uniform Resource Locators (URL) http://sunsite.cnlab-switch.ch/ By: Berners-Lee, Masinter & McCahill
The
3.10 FILES
The file URL scheme is used to designate files accessible on a particular host computer. This scheme, unlike most other URL schemes, does not designate a resource that is universally accessible over the Internet.
A file URL takes the form:
file://
where
For example, a VMS file
DISK$USER:[MY.NOTES]NOTE123456.TXT
might become
As a special case,
The file URL scheme is unusual in that it does not specify an Internet protocol or access method for such files; as such, its utility in network protocols between hosts is limited.
3.11 PROSPERO
The Prospero URL scheme is used to designate resources that are accessed via the Prospero Directory Service. The Prospero protocol is described elsewhere [14].
A prospero URLs takes the form:
prospero://
where
Formatted and Indexed by: page 15 of 25 instructional media + magic, inc. RFC1738 Web Address: Uniform Resource Locators (URL) http://sunsite.cnlab-switch.ch/ By: Berners-Lee, Masinter & McCahill
allowed.
The
Prospero URLs are interpreted by contacting a Prospero directory server on the specified host and port to determine appropriate access methods for a resource, which might themselves be represented as different URLs. External Prospero links are represented as URLs of the underlying access method and are not represented as Prospero URLs.
Note that a slash "/" may appear in the
In addition, after the
4. REGISTRATION OF NEW SCHEMES
A new scheme may be introduced by defining a mapping onto a conforming URL syntax, using a new prefix. URLs for experimental schemes may be used by mutual agreement between parties. Scheme names starting with the characters "x-" are reserved for experimental purposes.
The Internet Assigned Numbers Authority (IANA) will maintain a registry of URL schemes. Any submission of a new URL scheme must include a definition of an algorithm for accessing of resources within that scheme and the syntax for representing such a scheme.
URL schemes must have demonstrable utility and operability. One way to provide such a demonstration is via a gateway which provides objects in the new scheme for clients using an existing protocol. If
Formatted and Indexed by: page 16 of 25 instructional media + magic, inc. RFC1738 Web Address: Uniform Resource Locators (URL) http://sunsite.cnlab-switch.ch/ By: Berners-Lee, Masinter & McCahill
the new scheme does not locate resources that are data objects, the properties of names in the new space must be clearly defined.
New schemes should try to follow the same syntactic conventions of existing schemes, where appropriate. It is likewise recommended that, where a protocol allows for retrieval by URL, that the client software have provision for being configured to use specific gateway locators for indirect access through new naming schemes.
The following scheme have been proposed at various times, but this document does not define their syntax or use at this time. It is suggested that IANA reserve their scheme names for future definition:
afs Andrew File System global file names. mid Message identifiers for electronic mail. cid Content identifiers for MIME body parts. nfs Network File System (NFS) file names. tn3270 Interactive 3270 emulation sessions. mailserver Access to data available from mail servers. z39.50 Access to ANSI Z39.50 services.
5. BNF for specific URL schemes
This is a BNF-like description of the Uniform Resource Locator syntax, using the conventions of RFC822, except that "|" is used to designate alternatives, and brackets [] are used around optional or repeated elements. Briefly, literals are quoted with "", optional elements are enclosed in [brackets], and elements may be preceded with
; The generic form of a URL is: genericurl = scheme ":" schemepart
; Specific predefined schemes are defined here; new schemes ; may be registered with IANA url = httpurl | ftpurl | newsurl | nntpurl | telneturl | gopherurl | waisurl | mailtourl | fileurl | prosperourl | otherurl
; new schemes follow the general syntax otherurl = genericurl
; the scheme is in lower case; interpreters should use case-ignore scheme = 1*[ lowalpha | digit | "+" | "-" | "." ]
Formatted and Indexed by: page 17 of 25 instructional media + magic, inc. RFC1738 Web Address: Uniform Resource Locators (URL) http://sunsite.cnlab-switch.ch/ By: Berners-Lee, Masinter & McCahill schemepart = *xchar | ip-schemepart
; URL schemeparts for ip based protocols: ip-schemepart = "//" login [ "/" urlpath ] login = [ user [ ":" password ] "@" ] hostport hostport = host [ ":" port ] host = hostname | hostnumber hostname = *[ domainlabel "." ] toplabel domainlabel = alphadigit | alphadigit *[ alphadigit | "-" ] alphadigit toplabel = alpha | alpha *[ alphadigit | "-" ] alphadigit alphadigit = alpha | digit hostnumber = digits "." digits "." digits "." digits port = digits user = *[ uchar | ";" | "?" | "&" | "=" ] password = *[ uchar | ";" | "?" | "&" | "=" ] urlpath = *xchar ; depends on protocol see section 3.1
; The predefined schemes:
; FTP (see also RFC959) ftpurl = "ftp://" login [ "/" fpath [ ";type=" ftptype ]] fpath = fsegment *[ "/" fsegment ] fsegment = *[ uchar | "?" | ":" | "@" | "&" | "=" ] ftptype = "A" | "I" | "D" | "a" | "i" | "d"
; FILE fileurl = "file://" [ host | "localhost" ] "/" fpath
; HTTP httpurl = "http://" hostport [ "/" hpath [ "?" search ]] hpath = hsegment *[ "/" hsegment ] hsegment = *[ uchar | ";" | ":" | "@" | "&" | "=" ] search = *[ uchar | ";" | ":" | "@" | "&" | "=" ]
; GOPHER (see also RFC1436) gopherurl = "gopher://" hostport [ / [ gtype [ selector [ "%09" search [ "%09" gopher+_string ] ] ] ] ] gtype = xchar selector = *xchar gopher+_string = *xchar
Formatted and Indexed by: page 18 of 25 instructional media + magic, inc. RFC1738 Web Address: Uniform Resource Locators (URL) http://sunsite.cnlab-switch.ch/ By: Berners-Lee, Masinter & McCahill
; MAILTO (see also RFC822) mailtourl = "mailto:" encoded822addr encoded822addr = 1*xchar ; further defined in RFC822
; NEWS (see also RFC1036) newsurl = "news:" grouppart grouppart = "*" | group | article group = alpha *[ alpha | digit | "-" | "." | "+" | "_" ] article = 1*[ uchar | ";" | "/" | "?" | ":" | "&" | "=" ] "@" host
; NNTP (see also RFC977) nntpurl = "nntp://" hostport "/" group [ "/" digits ]
; TELNET telneturl = "telnet://" login [ "/" ]
; WAIS (see also RFC1625) waisurl = waisdatabase | waisindex | waisdoc waisdatabase = "wais://" hostport "/" database waisindex = "wais://" hostport "/" database "?" search waisdoc = "wais://" hostport "/" database "/" wtype "/" wpath database = *uchar wtype = *uchar wpath = *uchar
; PROSPERO prosperourl = "prospero://" hostport "/" ppath *[ fieldspec ] ppath = psegment *[ "/" psegment ] psegment = *[ uchar | "?" | ":" | "@" | "&" | "=" ] fieldspec = ";" fieldname "=" fieldvalue fieldname = *[ uchar | "?" | ":" | "@" | "&" ] fieldvalue = *[ uchar | "?" | ":" | "@" | "&" ]
; Miscellaneous definitions lowalpha = "a" | "b" | "c" | "d" | "e" | "f" | "g" | "h" | "i" | "j" | "k" | "l" | "m" | "n" | "o" | "p" | "q" | "r" | "s" | "t" | "u" | "v" | "w" | "x" | "y" | "z" hialpha = "A" | "B" | "C" | "D" | "E" | "F" | "G" | "H" | "I" | "J" | "K" | "L" | "M" | "N" | "O" | "P" | "Q" | "R" | "S" | "T" | "U" | "V" | "W" | "X" | "Y" | "Z"
Formatted and Indexed by: page 19 of 25 instructional media + magic, inc. RFC1738 Web Address: Uniform Resource Locators (URL) http://sunsite.cnlab-switch.ch/ By: Berners-Lee, Masinter & McCahill alpha = lowalpha | hialpha digit = "0" | "1" | "2" | "3" | "4" | "5" | "6" | "7" | "8" | "9" safe = "$" | "-" | "_" | "." | "+" extra = "!" | "*" | "'" | "(" | ")" | "," national = "{" | "}" | "|" | "\" | "^" | "~" | "[" | "]" | "`" punctuation = "<" | ">" | "#" | "%" | <"> reserved = ";" | "/" | "?" | ":" | "@" | "&" | "=" hex = digit | "A" | "B" | "C" | "D" | "E" | "F" | "a" | "b" | "c" | "d" | "e" | "f" escape = "%" hex hex unreserved = alpha | digit | safe | extra uchar = unreserved | escape xchar = unreserved | reserved | escape digits = 1*digit
6. Security Considerations
The URL scheme does not in itself pose a security threat. Users should beware that there is no general guarantee that a URL which at one time points to a given object continues to do so, and does not even at some later time point to a different object due to the movement of objects on servers.
A URL-related security threat is that it is sometimes possible to construct a URL such that an attempt to perform a harmless idempotent operation such as the retrieval of the object will in fact cause a possibly damaging remote operation to occur. The unsafe URL is typically constructed by specifying a port number other than that reserved for the network protocol in question. The client unwittingly contacts a server which is in fact running a different protocol. The content of the URL contains instructions which when interpreted according to this other protocol cause an unexpected operation. An example has been the use of gopher URLs to cause a rude message to be sent via a SMTP server. Caution should be used when using any URL which specifies a port number other than the default for the protocol, especially when it is a number within the reserved space.
Care should be taken when URLs contain embedded encoded delimiters for a given protocol (for example, CR and LF characters for telnet protocols) that these are not unencoded before transmission. This would violate the protocol but could be used to simulate an extra operation or parameter, again causing an unexpected and possible harmful remote operation to be performed.
Formatted and Indexed by: page 20 of 25 instructional media + magic, inc. RFC1738 Web Address: Uniform Resource Locators (URL) http://sunsite.cnlab-switch.ch/ By: Berners-Lee, Masinter & McCahill
The use of URLs containing passwords that should be secret is clearly unwise.
7. Acknowledgements
This paper builds on the basic WWW design (RFC 1630) and much discussion of these issues by many people on the network. The discussion was particularly stimulated by articles by Clifford Lynch, Brewster Kahle [10] and Wengyik Yeong [18]. Contributions from John Curran, Clifford Neuman, Ed Vielmetti and later the IETF URL BOF and URI working group were incorporated.
Most recently, careful readings and comments by Dan Connolly, Ned Freed, Roy Fielding, Guido van Rossum, Michael Dolan, Bert Bos, John Kunze, Olle Jarnefors, Peter Svanberg and many others have helped refine this RFC.
Formatted and Indexed by: page 21 of 25 instructional media + magic, inc. RFC1738 Web Address: Uniform Resource Locators (URL) http://sunsite.cnlab-switch.ch/ By: Berners-Lee, Masinter & McCahill
APPENDIX: Recommendations for URLs in Context
URIs, including URLs, are intended to be transmitted through protocols which provide a context for their interpretation.
In some cases, it will be necessary to distinguish URLs from other possible data structures in a syntactic structure. In this case, is recommended that URLs be preceeded with a prefix consisting of the characters "URL:". For example, this prefix may be used to distinguish URLs from other kinds of URIs.
In addition, there are many occasions when URLs are included in other kinds of text; examples include electronic mail, USENET news messages, or printed on paper. In such cases, it is convenient to have a separate syntactic wrapper that delimits the URL and separates it from the rest of the text, and in particular from punctuation marks that might be mistaken for part of the URL. For this purpose, is recommended that angle brackets ("<" and ">"), along with the prefix "URL:", be used to delimit the boundaries of the URL. This wrapper does not form part of the URL and should not be used in contexts in which delimiters are already specified.
In the case where a fragment/anchor identifier is associated with a URL (following a "#"), the identifier would be placed within the brackets as well.
In some cases, extra whitespace (spaces, linebreaks, tabs, etc.) may need to be added to break long URLs across lines. The whitespace should be ignored when extracting the URL.
No whitespace should be introduced after a hyphen ("-") character. Because some typesetters and printers may (erroneously) introduce a hyphen at the end of line when breaking a line, the interpreter of a URL containing a line break immediately after a hyphen should ignore all unencoded whitespace around the line break, and should be aware that the hyphen may or may not actually be part of the URL.
Examples:
Yes, Jim, I found it under
Formatted and Indexed by: page 22 of 25 instructional media + magic, inc. RFC1738 Web Address: Uniform Resource Locators (URL) http://sunsite.cnlab-switch.ch/ By: Berners-Lee, Masinter & McCahill
References
[1] Anklesaria, F., McCahill, M., Lindner, P., Johnson, D., Torrey, D., and B. Alberti, "The Internet Gopher Protocol (a distributed document search and retrieval protocol)", RFC 1436, University of Minnesota, March 1993.
[2] Anklesaria, F., Lindner, P., McCahill, M., Torrey, D., Johnson, D., and B. Alberti, "Gopher+: Upward compatible enhancements to the Internet Gopher protocol", University of Minnesota, July 1993.
[3] Berners-Lee, T., "Universal Resource Identifiers in WWW: A Unifying Syntax for the Expression of Names and Addresses of Objects on the Network as used in the World-Wide Web", RFC 1630, CERN, June 1994.
[4] Berners-Lee, T., "Hypertext Transfer Protocol (HTTP)", CERN, November 1993.
[5] Braden, R., Editor, "Requirements for Internet Hosts -- Application and Support", STD 3, RFC 1123, IETF, October 1989.
[6] Crocker, D. "Standard for the Format of ARPA Internet Text Messages", STD 11, RFC 822, UDEL, April 1982.
[7] Davis, F., Kahle, B., Morris, H., Salem, J., Shen, T., Wang, R., Sui, J., and M. Grinbaum, "WAIS Interface Protocol Prototype Functional Specification", (v1.5), Thinking Machines Corporation, April 1990.
[8] Horton, M. and R. Adams, "Standard For Interchange of USENET Messages", RFC 1036, AT&T Bell Laboratories, Center for Seismic Studies, December 1987.
[9] Huitema, C., "Naming: Strategies and Techniques", Computer Networks and ISDN Systems 23 (1991) 107-110.
Formatted and Indexed by: page 23 of 25 instructional media + magic, inc. RFC1738 Web Address: Uniform Resource Locators (URL) http://sunsite.cnlab-switch.ch/ By: Berners-Lee, Masinter & McCahill
[10] Kahle, B., "Document Identifiers, or International Standard Book Numbers for the Electronic Age", 1991.
[11] Kantor, B. and P. Lapsley, "Network News Transfer Protocol: A Proposed Standard for the Stream-Based Transmission of News", RFC 977, UC San Diego & UC Berkeley, February 1986.
[12] Kunze, J., "Functional Requirements for Internet Resource Locators", Work in Progress, December 1994.
[13] Mockapetris, P., "Domain Names - Concepts and Facilities", STD 13, RFC 1034, USC/Information Sciences Institute, November 1987.
[14] Neuman, B., and S. Augart, "The Prospero Protocol", USC/Information Sciences Institute, June 1993.
[15] Postel, J. and J. Reynolds, "File Transfer Protocol (FTP)", STD 9, RFC 959, USC/Information Sciences Institute, October 1985.
[16] Sollins, K. and L. Masinter, "Functional Requirements for Uniform Resource Names", RFC 1737, MIT/LCS, Xerox Corporation, December 1994.
[17] St. Pierre, M, Fullton, J., Gamiel, K., Goldman, J., Kahle, B., Kunze, J., Morris, H., and F. Schiettecatte, "WAIS over Z39.50-1988", RFC 1625, WAIS, Inc., CNIDR, Thinking Machines Corp., UC Berkeley, FS Consulting, June 1994.
[18] Yeong, W. "Towards Networked Information Retrieval", Technical report 91-06-25-01, Performance Systems International, Inc.
[19] Yeong, W., "Representing Public Archives in the Directory", Work in Progress, November 1991.
Formatted and Indexed by: page 24 of 25 instructional media + magic, inc. RFC1738 Web Address: Uniform Resource Locators (URL) http://sunsite.cnlab-switch.ch/ By: Berners-Lee, Masinter & McCahill
[20] "Coded Character Set -- 7-bit American Standard Code for Information Interchange", ANSI X3.4-1986.
Editors' Addresses
Tim Berners-Lee World-Wide Web project CERN, 1211 Geneva 23, Switzerland
Phone: +41 (22)767 3755 Fax: +41 (22)767 7155 EMail: [email protected]
Larry Masinter Xerox PARC 3333 Coyote Hill Road Palo Alto, CA 94034
Phone: (415) 812-4365 Fax: (415) 812-4333 EMail: [email protected]
Mark McCahill Computer and Information Services, University of Minnesota Room 152 Shepherd Labs 100 Union Street SE Minneapolis, MN 55455
Phone: (612) 625 1300 EMail: [email protected]
Formatted and Indexed by: page 25 of 25 instructional media + magic, inc. RFC1766 Web Address: Language Tag http://sunsite.cnlab-switch.ch/ By: Alvestrand
Network Working Group H. Alvestrand Request for Comments: 1766 UNINETT Category: Standards Track March 1995
Tags for the Identification of Languages
Status of this Memo
This document specifies an Internet standards track protocol for the Internet community, and requests discussion and suggestions for improvements. Please refer to the current edition of the "Internet Official Protocol Standards" (STD 1) for the standardization state and status of this protocol. Distribution of this memo is unlimited.
Abstract
This document describes a language tag for use in cases where it is desired to indicate the language used in an information object.
It also defines a Content-language: header, for use in the case where one desires to indicate the language of something that has RFC-822- like headers, like MIME body parts or Web documents, and a new parameter to the Multipart/Alternative type, to aid in the usage of the Content-Language: header.
1. Introduction
There are a number of languages spoken by human beings in this world.
A great number of these people would prefer to have information presented in a language that they understand.
In some contexts, it is possible to have information in more than one language, or it might be possible to provide tools for assisting in the understanding of a language (like dictionaries).
A prerequisite for any such function is a means of labelling the information content with an identifier for the language in which is is written.
In the tradition of solving only problems that we think we understand, this document specifies an identifier mechanism, and one possible use for it.
Formatted and Indexed by: page 1 of 7 instructional media + magic, inc. RFC1766 Web Address: Language Tag http://sunsite.cnlab-switch.ch/ By: Alvestrand
2. The Language tag
The language tag is composed of 1 or more parts: A primary language tag and a (possibly empty) series of subtags.
The syntax of this tag in RFC-822 EBNF is:
Language-Tag = Primary-tag *( "-" Subtag ) Primary-tag = 1*8ALPHA Subtag = 1*8ALPHA
Whitespace is not allowed within the tag.
All tags are to be treated as case insensitive; there exist conventions for capitalization of some of them, but these should not be taken to carry meaning.
The namespace of language tags is administered by the IANA according to the rules in section 5 of this document.
The following registrations are predefined:
In the primary language tag:
- All 2-letter tags are interpreted according to ISO standard 639, "Code for the representation of names of languages" [ISO 639].
- The value "i" is reserved for IANA-defined registrations
- The value "x" is reserved for private use. Subtags of "x" will not be registered by the IANA.
- Other values cannot be assigned except by updating this standard.
The reason for reserving all other tags is to be open towards new revisions of ISO 639; the use of "i" and "x" is the minimum we can do here to be able to extend the mechanism to meet our requirements.
In the first subtag:
- All 2-letter codes are interpreted as ISO 3166 alpha-2 country codes denoting the area in which the language is used.
- Codes of 3 to 8 letters may be registered with the IANA by anyone who feels a need for it, according to the rules in
Formatted and Indexed by: page 2 of 7 instructional media + magic, inc. RFC1766 Web Address: Language Tag http://sunsite.cnlab-switch.ch/ By: Alvestrand
chapter 5 of this document.
The information in the subtag may for instance be:
- Country identification, such as en-US (this usage is described in ISO 639)
- Dialect or variant information, such as no-nynorsk or en- cockney
- Languages not listed in ISO 639 that are not variants of any listed language, which can be registered with the i- prefix, such as i-cherokee
- Script variations, such as az-arabic and az-cyrillic
In the second and subsequent subtag, any value can be registered.
NOTE: The ISO 639/ISO 3166 convention is that language names are written in lower case, while country codes are written in upper case. This convention is recommended, but not enforced; the tags are case insensitive.
NOTE: ISO 639 defines a registration authority for additions to and changes in the list of languages in ISO 639. This authority is:
International Information Centre for Terminology (Infoterm) P.O. Box 130 A-1021 Wien Austria Phone: +43 1 26 75 35 Ext. 312 Fax: +43 1 216 32 72
The following codes have been added in 1989 (nothing later): ug (Uigur), iu (Inuktitut, also called Eskimo), za (Zhuang), he (Hebrew, replacing iw), yi (Yiddish, replacing ji), and id (Indonesian, replacing in).
NOTE: The registration agency for ISO 3166 (country codes) is:
ISO 3166 Maintenance Agency Secretariat c/o DIN Deutches Institut fuer Normung Burggrafenstrasse 6 Postfach 1107 D-10787 Berlin Germany Phone: +49 30 26 01 320 Fax: +49 30 26 01 231
Formatted and Indexed by: page 3 of 7 instructional media + magic, inc. RFC1766 Web Address: Language Tag http://sunsite.cnlab-switch.ch/ By: Alvestrand
The country codes AA, QM-QZ, XA-XZ and ZZ are reserved by ISO 3166 as user-assigned codes.
2.1. Meaning of the language tag
The language tag always defines a language as spoken (or written) by human beings for communication of information to other human beings. Computer languages are explicitly excluded.
There is no guaranteed relationship between languages whose tags start out with the same series of subtags; especially, they are NOT guraranteed to be mutually comprehensible, although this will sometimes be the case.
Applications should always treat language tags as a single token; the division into main tag and subtags is an administrative mechanism, not a navigation aid.
The relationship between the tag and the information it relates to is defined by the standard describing the context in which it appears. So, this section can only give possible examples of its usage.
- For a single information object, it should be taken as the set of languages that is required for a complete comprehension of the complete object. Example: Simple text.
- For an aggregation of information objects, it should be taken as the set of languages used inside components of that aggregation. Examples: Document stores and libraries.
- For information objects whose purpose in life is providing alternatives, it should be regarded as a hint that the material inside is provided in several languages, and that one has to inspect each of the alternatives in order to find its language or languages. In this case, multiple languages need not mean that one needs to be multilingual to get complete understanding of the document. Example: MIME multipart/alternative.
- It would be possible to define (for instance) an SGML DTD that defines a tag for indicating that following or contained text is written in this language, such that one could write "C'est la vie"; the Norwegian- speaking user could then access a French-Norwegian dictionary to find out what the quote meant.
Formatted and Indexed by: page 4 of 7 instructional media + magic, inc. RFC1766 Web Address: Language Tag http://sunsite.cnlab-switch.ch/ By: Alvestrand
3. The Content-language header
The Language header is intended for use in the case where one desires to indicate the language(s) of something that has RFC-822-like headers, like MIME body parts or Web documents.
The RFC-822 EBNF of the Language header is:
Language-Header = "Content-Language" ":" 1#Language-tag
Note that the Language-Header is allowed to list several languages in a comma-separated list.
Whitespace is allowed, which means also that one can place parenthesized comments anywhere in the language sequence.
3.1. Examples of Content-language values
NOTE: NONE of the subtags shown in this document have actually been assigned; they are used for illustration purposes only.
Norwegian official document, with parallel text in both official versions of Norwegian. (Both versions are readable by all Norwegians).
Content-Type: multipart/alternative; differences=content-language Content-Language: no-nynorsk, no-bokmaal
Voice recording from the London docks
Content-type: audio/basic Content-Language: en-cockney
Document in Sami, which does not have an ISO 639 code, and is spoken in several countries, but with about half the speakers in Norway, with six different, mutually incomprehensible dialects:
Content-type: text/plain; charset=iso-8859-10 Content-Language: i-sami-no (North Sami)
An English-French dictionary
Content-type: application/dictionary Content-Language: en, fr (This is a dictionary)
An official EC document (in a few of its official languages)
_
Formatted and Indexed by: page 5 of 7 instructional media + magic, inc. RFC1766 Web Address: Language Tag http://sunsite.cnlab-switch.ch/ By: Alvestrand
Content-type: multipart/alternative Content-Language: en, fr, de, da, el, it
An excerpt from Star Trek
Content-type: video/mpeg Content-Language: x-klingon
4. Use of Content-Language with Multipart/Alternative
When using the Multipart/Alternative body part of MIME, it is possible to have the body parts giving the same information content in different languages. In this case, one should put a Content- Language header on each of the body parts, and a summary Content- Language header onto the Multipart/Alternative itself.
4.1. The differences parameter to multipart/alternative
As defined in RFC 1541, Multipart/Alternative only has one parameter: boundary.
The common usage of Multipart/Alternative is to have more than one format of the same message (f.ex. PostScript and ASCII).
The use of language tags to differentiate between different alternatives will certainly not lead all MIME UAs to present the most sensible body part as default.
Therefore, a new parameter is defined, to allow the configuration of MIME readers to handle language differences in a sensible manner.
Name: Differences Value: One or more of Content-Type Content-Language
Further values can be registered with IANA; it must be the name of a header for which a definition exists in a published RFC. If not present, Differences=Content-Type is assumed.
The intent is that the MIME reader can look at these headers of the message component to do an intelligent choice of what to present to the user, based on knowledge about the user preferences and capabilities.
(The intent of having registration with IANA of the fields used in this context is to maintain a list of usages that a mail UA may expect to see, not to reject usages.)
Formatted and Indexed by: page 6 of 7 instructional media + magic, inc. RFC1766 Web Address: Language Tag http://sunsite.cnlab-switch.ch/ By: Alvestrand
(NOTE: The MIME specification [RFC 1521], section 7.2, states that headers not beginning with "Content-" are generally to be ignored in body parts. People defining a header for use with "differences=" should take note of this.)
The mechanism for deciding which body part to present is outside the scope of this document.
MIME EXAMPLE:
Content-Type: multipart/alternative; differences=Content-Language; boundary="limit" Content-Language: en, fr, de
--limit Content-Language: fr
Le renard brun et agile saute par dessus le chien paresseux --limit Content-Language: de Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-encoding: quoted-printable
Der schnelle braune Fuchs h=FCpft =FCber den faulen Hund --limit Content-Language: en
The quick brown fox jumps over the lazy dog --limit--
When composing a message, the choice of sequence may be somewhat arbitrary. However, non-MIME mail readers will show the first body part first, meaning that this should most likely be the language understood by most of the recipients.
5. IANA registration procedure for language tags
Any language tag must start with an existing tag, and extend it.
This registration form should be used by anyone who wants to use a language tag not defined by ISO or IANA.
Formatted and Indexed by: page 7 of 7 instructional media + magic, inc. RFC1807 Web Address: A Format for Bibliographic Records http://sunsite.cnlab-switch.ch/ By: Lasher & Cohen
Network Working Group R. Lasher Request For Comments: 1807 Stanford Obsoletes: 1357 D. Cohen Category: Informational Myricom June 1995
A Format for Bibliographic Records
Status of this Memo
This memo provides information for the Internet community. This memo does not specify an Internet standard of any kind. Distribution of this memo is unlimited.
Abstract
This RFC defines a format for bibliographic records describing technical reports. This format is used by the Cornell University Dienst protocol and the Stanford University SIFT system. The original RFC (RFC 1357) was written by D. Cohen, ISI, July 1992. This is a revision of RFC 1357. New fields include handle, other_access, keyword, and withdraw.
Introduction
Many universities and other R&D organizations routinely announce new technical reports by mailing (via the postal services) the bibliographic records of these reports.
These mailings have non-trivial cost and delay. In addition, their recipients cannot conveniently file them, electronically, for later retrieval and searches.
Publishing organizations that wish to use e-mail or file transfer to obtain these announcements can do so by using the following format.
Organizations may automate to any degree (or not at all) both the creation of these records (about their own publications) and the handling of the records received from other organizations.
This format is designed to be simple, for people and for machines, to be easy to read ("human readable") and create without any special programs.
This RFC defines the format of bibliographic records, not how to process them.
Formatted and Indexed by: page 1 of 15 instructional media + magic, inc. RFC1807 Web Address: A Format for Bibliographic Records http://sunsite.cnlab-switch.ch/ By: Lasher & Cohen
This format is a "tagged" format with self-explaining alphabetic tags. It should be possible to prepare and to read bibliographic records using any text editor, without any special programs.
This RFC includes the CR-CATEGORY, a field useful for Computer Science publications. It is expected that similar fields will be added for other domains.
This format, as described in RFC 1357, was implemented as part of the Dienst system and has been in use by the five ARPA-funded computer science institutions to exchange bibliographic records (Cornell, SU, UC, MIT, and CMU). Programs have been written to map between this RFC and structured USMARC (format developed at the Library of Congress) cataloging records, also from USMARC to the RFC.
The focus of this ARPA-funded research has been into many aspects of digital libraries including searching and accessing techniques that do not necessarily use bibliographic records (for example, natural language processing, automatic and full-text indexing). However, the continued use of bibliographic records is expected to remain an important part of the library system environment of the future and its use is an important link between the physical world of scientific works and the on-line world of digital objects. The format described in this paper allows a link between these two worlds to be created.
This format was developed with considerable help and involvement of Computer Science and Library personnel from several organizations, including Carnegie Mellon University, Corporation for National Research Initiatives (CNRI), Cornell University, University of Southern California/Information Sciences Institute (ISI), Meridian (now called DynCorp), Massachusetts Institute of Technology, Stanford University, and the University of California. Key contributions were provided by Jerry Saltzer of MIT, and Larry Lannom of DynCorp. The initial draft was prepared by Danny Cohen and Larry Miller of ISI. The revision was done by Rebecca Lasher from Stanford with assistance from the CS-TR participants.
This RFC does not place any limitations on the dissemination of the bibliographic records. If there are limitations on the dissemination of the publication, it should be protected by some means such as passwords. This RFC does not address this protection.
The use of this format is encouraged. There are no limitations on its use.
Formatted and Indexed by: page 2 of 15 instructional media + magic, inc. RFC1807 Web Address: A Format for Bibliographic Records http://sunsite.cnlab-switch.ch/ By: Lasher & Cohen
The Information Fields
The various fields should follow the format described below.
The tags (aka Field-IDs) are shown in upper case.
Formatted and Indexed by: page 3 of 15 instructional media + magic, inc. RFC1807 Web Address: A Format for Bibliographic Records http://sunsite.cnlab-switch.ch/ By: Lasher & Cohen
Meta Format
* Keep It Simple.
* One bibliographic record for each publication, where a "publication" is whatever the publishing institution defines as such.
* A record contains several fields.
* Each field starts with its tag (aka the field-ID) which is a reserved identifier (containing no separators) at the beginning of a new line with or without spaces before it), followed by two colons ("::"), followed by the field data.
* Continuation lines: Lines are limited to 79 characters. When needed, fields may continue over several lines, with an implied space in between. In order to simplify the use no special marking is used to indicate continuation line. Hence, fields are terminated by a line that starts (apart from white space) with a word followed by two colons. Except for the "END::" that is terminated by the end of line.) For improved human readability it is suggested to start continuation lines with some spaces.
* Several fields are mandatory and must appear in the record. All fields (unless specifically not permitted to) may be in any order and may be repeated as needed (e.g., the AUTHOR field). The order of the repeated fields is always preserved.
* Only printable ASCII characters are to be used. The permissible characters are ASCII codes 040 (Space) through 176(~) and line breaks which are \012 (LF) or \012\015 (CRLF). Empty lines indicate paragraph break. \009 (tab) must be replaced by spaces. This specifically forbids tabs, null characters, DEL, backspaces, etc. (i.e., if used, the record is invalid.)
However full 8 bit ASCII may be used. WARNING: some electronic mailers cannot handle 8 bit ASCII and these records may need to be transported via other mechanisms.
Throughout this document the word "publisher" means the publishing organization of a report (e.g., a university or a department thereof), not necessarily an organization authorized to issue ISBN numbers.
Formatted and Indexed by: page 4 of 15 instructional media + magic, inc. RFC1807 Web Address: A Format for Bibliographic Records http://sunsite.cnlab-switch.ch/ By: Lasher & Cohen
EXAMPLE ------BIB-VERSION:: CS-TR-v2.1 ID:: OUKS//CS-TR-91-123 ENTRY:: January 15, 1992 ORGANIZATION:: Oceanview University, Kansas, Computer Science TYPE:: Technical Report REVISION:: January 5, 1995; FTP access information added TITLE:: Scientific Communication must be timely AUTHOR:: Finnegan, James A. CONTACT:: Prof. J. A. Finnegan, CS Dept, Oceanview Univ, Oceanview, KS 54321 Tel: 913-456-7890
Many alchemists in the country work on important fusion problems. All of them cooperate and interact with each other through the scientific literature. This scientific communication methodology has many advantages. Timeliness is not one of them.
END:: OUKS//CS-TR-91-123 ------End of Example ------
For reference, the above example has about 1,689 characters (184 words) including about 249 characters (36 words) in the abstract.
Formatted and Indexed by: page 5 of 15 instructional media + magic, inc. RFC1807 Web Address: A Format for Bibliographic Records http://sunsite.cnlab-switch.ch/ By: Lasher & Cohen
The Actual Format
The term "Open Ended Format" in the following means arbitrary text.
In the following double-quotes indicate complete strings. They are included only for grouping and are not expected to be used in the actual records.
The BIB-VERSION, ID, ENTRY, and END field must appear as the first, second, third, and last fields, and may not be repeated in the record. All other fields may be repeated as needed.
BIB-VERSION (M) -- This is the first field of any record. It is a mandatory field. It identifies the version of the format used to create this bibliographic record. This RFC defines BIB-Version TR-v2.1
BIB-VERSIONs that start with the letter X (case independent) are considered experimental. Bib-records sent with such a BIB-VERSION should NOT be incorporated in the permanent database of the recipient.
Using this version of this format, this field is always:
Format: BIB-VERSION:: CS-TR-v2.1
ID (M) -- This is the second field of any record. It is also a mandatory field. The ID field identifies the bibliographic record and is used in management of these records. Its format is "ID:: XXX//YYY", where XXX is the publisher-ID (the controlled symbol of the publisher) and YYY is the ID (e.g., report number) of the publication as assigned by the publisher. This ID is typically printed on the cover, and may contain slashes.
The organization symbols "DUMMY" and "TEST" (case independent) are reserved for test records that should NOT be incorporated in the permanent database of the recipients.
Format: ID::
Example: ID:: OUKS//CS-TR-91-123
**** See the note at the end regarding the **** **** controlled symbols of the publishers *****
Formatted and Indexed by: page 6 of 15 instructional media + magic, inc. RFC1807 Web Address: A Format for Bibliographic Records http://sunsite.cnlab-switch.ch/ By: Lasher & Cohen
ENTRY (M) -- This is a mandatory field. It is the date of creating this bibliographic record.
The format for ENTRY date is "Month Day, Year". The month must be alphabetic (spelled out). The "Day" is a 1- or 2-digit number. The "Year" is a 4-digit number.
Format: ENTRY::
Example: ENTRY:: January 15, 1992
ORGANIZATION (O) -- It is the full name spelled out (no acronyms, please) of the publishing organization. The use of this name is controlled together with the controlled symbol of the publisher (as discussed above for the ID field).
Avoid acronyms because there are many common acronyms, such as ISI and USC. Please provide it in ascending order, such as "X University, Y Department" (not "Y Department, X University").
Format: ORGANIZATION::
TITLE (O) -- This is the title of the work as assigned by the author. This field should include the complete title with all the subtitles, if any.
Format: TITLE::
Example: TITLE:: The Computerization of Oceanview with High Speed Fiber Optics Communication
TYPE (O) -- Indicates the type of publication (summary, final project report, etc.) as assigned by the issuing organization.
Format: TYPE::
Example: TYPE:: Technical Report
REVISION (O) -- Indicates that the current bibliographic record is a revision of a previously issued record and is intended
Formatted and Indexed by: page 7 of 15 instructional media + magic, inc. RFC1807 Web Address: A Format for Bibliographic Records http://sunsite.cnlab-switch.ch/ By: Lasher & Cohen
to replace it. Revision information consists of a date and/or followed by a semicolon and by text in an open ended format. The revised bibliographic record should contain a complete record for the publication, not just a list of changes to the old record. If revision is omitted, the record is assumed to be a new record and not a revision. If the revision date is specified as 0, this is assumed to be January 1, 1900 (the previous RFC, used revision data of 0, 1, 2, 3, etc. this specification is for programs that might process records from RFC1357).
The text before the semicolon in this field is a date of the form month day, year. Any record with a more recent revision date replaces completely any record with an earlier revision date (supplied either explicitly or by default). Use the text to describe the revision. Reasons to send out a revised record include an error in the original, or change in the access information.
Format: REVISION:: January 1, 1995;
Example: REVISION:: January 1, 1995; FTP information added
WITHDRAW (O) Withdraw means the document is no longer available. Some Institutions choose to delete the record others remove some of the fields. It is up to each institution to decide how to process withdraw records.
A withdraw record has all of the mandatory fields plus the withdraw field and a mandatory revision field. The Withdraw field should indicate the reason for the withdraw in free text. Example for withdrawing a bibliographic record::
BIB-VERSION:: CS-TR-v2.1 ID:: OUKS//CS-TR-91-123 ENTRY:: January 21, 1995 ORGANIZATION:: Oceanview University, Kansas, Computer Science TITLE:: The Computerization of Oceanview with High Speed Fiber Optics Communication REVISION:: January 21, 1995 WITHDRAW:: Withdrawn, found to be irrelevant END:: OUKS//CS-TR-91-123
Formatted and Indexed by: page 8 of 15 instructional media + magic, inc. RFC1807 Web Address: A Format for Bibliographic Records http://sunsite.cnlab-switch.ch/ By: Lasher & Cohen
AUTHOR (O) -- Personal names only. Normal last name first inversion. Editors should be listed here as well, identified with the usual "(ed.)" as shown below in the last example.
If the report was not authored by a person (e.g., it was authored by a committee or a panel) use CORP-AUTHOR (see below) instead of AUTHOR.
Multiple authors are entered by using multiple lines, each in the form of "AUTHOR::
The system preserves the order of the authors.
Format: AUTHOR::
Example: AUTHOR:: Finnegan, James A. AUTHOR:: Pooh, Winnie The AUTHOR:: Lastname, Firstname (ed.)
CORP-AUTHOR (O) -- The corporate author (e.g., a committee or a panel) that authored the report, which may be different from the ORGANIZATION issuing the report.
In entering the corporate name please omit initial "the" or "a". If it is really part of the name, please invert it.
Format: CORP-AUTHOR::
Example: CORP-AUTHOR:: Committee on long-range computing
CONTACT (O) -- The contact for the author(s). Open-ended, most likely E-mail and postal addresses.
A CONTACT field for each author should be provided, separately, or for all the AUTHOR fields. E-mail addresses should always be in "pointy brackets" (as in the example below).
Format: CONTACT::
Example: CONTACT:: Prof. J. A. Finnegan, CS Dept, Oceanview Univ., Oceanview, Kansas, 54321 Tel: 913-456-7890
Formatted and Indexed by: page 9 of 15 instructional media + magic, inc. RFC1807 Web Address: A Format for Bibliographic Records http://sunsite.cnlab-switch.ch/ By: Lasher & Cohen
DATE (O) -- The publication date. The formats are "Month Year" and "Month Day, Year". The month must be alphabetic (spelled out). The "Day" is a 1- or 2-digit number. The "Year" is a 4- digit number.
Format: DATE::
Example: DATE:: January 1992 Example: DATE:: January 15, 1992
PAGES (O) -- Total number of pages, without being too picky about it. Final numbered page is actually preferred, if it is a reasonable approximation to the total number of pages.
Format: PAGES::
Example: PAGES:: 48
COPYRIGHT (O) -- Copyright information. Open ended format. The COPYRIGHT field applies to the cited report, rather than to the current bibliographic record.
Format: COPYRIGHT::
Example: COPYRIGHT:: Copyright for the report (c) 1991, by J. A. Finnegan. All rights reserved. Permission is granted for any academic use of the report.
HANDLE (O) -- Handles are unique permanent identifiers that are used in the Handle Management System to retrieve location data. A handle is a printable string which when given to a handle server returns the location of the data.
Handles are used to identify digital objects stored within a digital library. If the technical report is available in electronic form, the Handle MUST be supplied in the bibliographic record.
Format is "HANDLE:: hdl:
Formatted and Indexed by: page 10 of 15 instructional media + magic, inc. RFC1807 Web Address: A Format for Bibliographic Records http://sunsite.cnlab-switch.ch/ By: Lasher & Cohen
http://www.cnri.reston.va.us.
**** NOTE: White space in HANDLE due to line wrap is ignored.
Format: HANDLE:: hdl:
Example: HANDLE:: hdl:oceanview.electr/CS-TR-91-123
OTHER_ACCESS (O) -- For URLs, URNs, and other yet to be invented formatted retrieval systems.
Only one URL or URN per occurrence of the field.
URL and URN information is available in the internet drafts from the IETF (Internet Engineering Task Force). The most recent drafts can be found on the CNRI WEB page at http://www.cnri.reston.va.us.
**** NOTE: White space in a URL or URN due to line wrap is ignored.
Format: OTHER_ACCESS:: URL:
Example: OTHER_ACCESS:: URL:http://elib.stanford.edu/Docume nt/STANFORD.CS:CS-TN-94-1
Example: OTHER_ACCESS:: URL:ftp://JUPITER.CS.OUKS.EDU/PUBS/ computerization.txt.
When the URN standard is finalized naming authorities will be registered and URNs will be viable unique identifiers. Until then this is a place holder. For the latest URN drafts see CNRI WEB page at http://www.cnri.reston.va.us.
RETRIEVAL (O) -- Open-ended format describing how to get a copy of the full text. This is an optional, repeatable field.
No limitations are placed on the dissemination of the bibliographic records. If there are limitations on the dissemination of the publication, it should be protected by some means such as passwords. This format does not address this protection.
Format: RETRIEVAL::
Formatted and Indexed by: page 11 of 15 instructional media + magic, inc. RFC1807 Web Address: A Format for Bibliographic Records http://sunsite.cnlab-switch.ch/ By: Lasher & Cohen
RETRIEVAL:: for full text with color pictures send a self-addressed stamped envelope to Prof. J.A. Finnegan, CS Dept, Oceanview University, Oceanview, KS 54321
KEYWORD (O) -- Specify any keywords, controlled or uncontrolled. This is an optional, repeatable field. Multiple keywords are entered using multiple lines in the form of "KEYWORD::
Format: KEYWORD::
Example: KEYWORD:: Scientific Communication KEYWORD:: Communication Theory
CR-CATEGORY (O) -- Specify the CR-category. The CR-category (the Computer Reviews Category) index (e.g., "B.3") should always be included, optionally followed by the name of that category. If the name is specified it should be fully specified with parent levels as needed to clarify it, as in the second example below. Use multiple lines for multiple categories.
Every year, the January issue of CR has the full list of these categories, with a detailed discussion of the CR Classification System, and a full index. Typically the full index appears in every January issue, and the top two levels in every issue.
Format: CR-CATEGORY::
Example: CR-CATEGORY:: D.1
Example: CR-CATEGORY:: B.3 Hardware, Memory Structures
PERIOD (O) -- Time period covered (date range). Applicable primarily to progress reports, etc. Any format is acceptable, as long as the two dates are separated with " to " (the word "to" surrounded by spaces) and each date is in the format allowed for dates, as described above for the date field.
Format: PERIOD::
Example: PERIOD:: January 1990 to March 1990
Formatted and Indexed by: page 12 of 15 instructional media + magic, inc. RFC1807 Web Address: A Format for Bibliographic Records http://sunsite.cnlab-switch.ch/ By: Lasher & Cohen
SERIES (O) -- Series title, including volume number within series. Open-ended format, with producing institution strongly encouraged to be internally consistent.
Format: SERIES::
Example: SERIES:: Communication
FUNDING (O) -- The name(s) of the funding organization(s).
Format: FUNDING::
Example: FUNDING:: ARPA
MONITORING (O) -- The name(s) of the monitoring organization(s).
Format: MONITORING::
CONTRACT (O) -- The contract number(s).
Format: CONTRACT::
Example: CONTRACT:: MMA-90-23-456
GRANT (O) -- The grant number(s).
Format: GRANT::
Example: GRANT:: NASA-91-2345
LANGUAGE (O) -- The language in which the report is written. Please use the full English name of that language.
Please include the Abstract in English, if possible.
If the language is not specified, English is assumed.
Format: LANGUAGE::
Example: LANGUAGE:: English Example: LANGUAGE:: French
Formatted and Indexed by: page 13 of 15 instructional media + magic, inc. RFC1807 Web Address: A Format for Bibliographic Records http://sunsite.cnlab-switch.ch/ By: Lasher & Cohen
NOTES (O) -- Miscellaneous free text.
Format: NOTES::
Example: NOTES:: This report is the full version of the paper with the same title in IEEE Trans ASSP Dec 1976
ABSTRACT (O) -- Highly recommended, but not mandatory. Even though no limit is defined for its length, it is suggested not to expect applications to be able to handle more than 10,000 characters.
The ABSTRACT is expected to be used for subject searching since titles are not enough. Even if the report is not in English, an English ABSTRACT is preferable. If no formal abstract appears on document, the producers of the bibliographic records are encouraged to use pieces of the introduction, first paragraph, etc.
Format: ABSTRACT:: xxxx ...... xxxxxxxx xxxx ...... xxxxxxxx
xxxx ...... xxxxxxxx xxxx ...... xxxxxxxx
END (M) -- This is a mandatory field. It must be the last entry of a record, identifying the record that it ends, by stating the same ID that was used at the beginning of the records, in its "ID::".
Format: END:: XXX//YYY
Example: END:: OUKS//CS-TR-91-123
>>>>>>> [END OF FORMAT DEFINITION] <<<<<<<
A Note Regarding the Controlled Symbols of the Publishers
In order to avoid conflicts among the symbols of the publishing organizations (the XXX part of the "ID:: XXX//YYY") it is suggested that the various organizations that publish reports (such as universities, departments, and laboratories) register their
Formatted and Indexed by: page 14 of 15 instructional media + magic, inc. RFC1807 Web Address: A Format for Bibliographic Records http://sunsite.cnlab-switch.ch/ By: Lasher & Cohen
Rebecca Lasher ([email protected]), of Stanford working with CNRI has agreed to coordinate this registration with the IANA for the publishers of Computer Science technical reports. It is suggested that before using this format the publishing organizations would coordinate with her (by e-mail) their symbols and the names of their organizations.
In order to help automated handling of the received bibliographic records, it is expected that the producers of bibliographic records will always use the same name, exactly, in the ORGANIZATION field.
Security Considerations
Security issues are not discussed in this memo.
Acknowledgements
This work was supported by the Advanced Research Projects Agency under Grant No. MDA-972-92-J-1029 with the Corporation for National Research Initiatives (CNRI). Its content does not necessarily reflect the position or the policy of the Government or CNRI, and no official endorsement should be inferred.
Authors' Addresses
Rebecca Lasher Mathematical and Computer Sciences Library M.S. 2125 Stanford University Stanford, CA, USA 94305
Phone: +1 415 723 0864 EMail: [email protected]
Danny Cohen Myricom 325 N. Santa Anita Ave. Arcadia, CA 91006 USA
Phone: +1 818 821 5555 EMail: [email protected]
_
Formatted and Indexed by: page 15 of 15 instructional media + magic, inc. RFC1808 Web Address: Relative Uniform Resource Locators http://sunsite.cnlab-switch.ch/ By: Fielding
Network Working Group R. Fielding Request for Comments: 1808 UC Irvine Category: Standards Track June 1995
Relative Uniform Resource Locators
Status of this Memo
This document specifies an Internet standards track protocol for the Internet community, and requests discussion and suggestions for improvements. Please refer to the current edition of the "Internet Official Protocol Standards" (STD 1) for the standardization state and status of this protocol. Distribution of this memo is unlimited.
Abstract
A Uniform Resource Locator (URL) is a compact representation of the location and access method for a resource available via the Internet. When embedded within a base document, a URL in its absolute form may contain a great deal of information which is already known from the context of that base document's retrieval, including the scheme, network location, and parts of the url-path. In situations where the base URL is well-defined and known to the parser (human or machine), it is useful to be able to embed URL references which inherit that context rather than re-specifying it in every instance. This document defines the syntax and semantics for such Relative Uniform Resource Locators.
1. Introduction
This document describes the syntax and semantics for "relative" Uniform Resource Locators (relative URLs): a compact representation of the location of a resource relative to an absolute base URL. It is a companion to RFC 1738, "Uniform Resource Locators (URL)" [2], which specifies the syntax and semantics of absolute URLs.
A common use for Uniform Resource Locators is to embed them within a document (referred to as the "base" document) for the purpose of identifying other Internet-accessible resources. For example, in hypertext documents, URLs can be used as the identifiers for hypertext link destinations.
Absolute URLs contain a great deal of information which may already be known from the context of the base document's retrieval, including the scheme, network location, and parts of the URL path. In situations where the base URL is well-defined and known, it is useful to be able to embed a URL reference which inherits that context
Formatted and Indexed by: page 1 of 16 instructional media + magic, inc. RFC1808 Web Address: Relative Uniform Resource Locators http://sunsite.cnlab-switch.ch/ By: Fielding
rather than re-specifying it within each instance. Relative URLs can also be used within data-entry dialogs to decrease the number of characters necessary to describe a location.
In addition, it is often the case that a group or "tree" of documents has been constructed to serve a common purpose; the vast majority of URLs in these documents point to locations within the tree rather than outside of it. Similarly, documents located at a particular Internet site are much more likely to refer to other resources at that site than to resources at remote sites.
Relative addressing of URLs allows document trees to be partially independent of their location and access scheme. For instance, it is possible for a single set of hypertext documents to be simultaneously accessible and traversable via each of the "file", "http", and "ftp" schemes if the documents refer to each other using relative URLs. Furthermore, document trees can be moved, as a whole, without changing any of the embedded URLs. Experience within the World-Wide Web has demonstrated that the ability to perform relative referencing is necessary for the long-term usability of embedded URLs.
2. Relative URL Syntax
The syntax for relative URLs is a shortened form of that for absolute URLs [2], where some prefix of the URL is missing and certain path components ("." and "..") have a special meaning when interpreting a relative path. Because a relative URL may appear in any context that could hold an absolute URL, systems that support relative URLs must be able to recognize them as part of the URL parsing process.
Although this document does not seek to define the overall URL syntax, some discussion of it is necessary in order to describe the parsing of relative URLs. In particular, base documents can only make use of relative URLs when their base URL fits within the generic-RL syntax described below. Although some URL schemes do not require this generic-RL syntax, it is assumed that any document which contains a relative reference does have a base URL that obeys the syntax. In other words, relative URLs cannot be used within documents that have unsuitable base URLs.
2.1. URL Syntactic Components
The URL syntax is dependent upon the scheme. Some schemes use reserved characters like "?" and ";" to indicate special components, while others just consider them to be part of the path. However, there is enough uniformity in the use of URLs to allow a parser to resolve relative URLs based upon a single, generic-RL syntax. This generic-RL syntax consists of six components:
Formatted and Indexed by: page 2 of 16 instructional media + magic, inc. RFC1808 Web Address: Relative Uniform Resource Locators http://sunsite.cnlab-switch.ch/ By: Fielding
each of which, except
scheme ":" ::= scheme name, as per Section 2.1 of RFC 1738 [2].
"//" net_loc ::= network location and login information, as per Section 3.1 of RFC 1738 [2].
"/" path ::= URL path, as per Section 3.1 of RFC 1738 [2].
";" params ::= object parameters (e.g., ";type=a" as in Section 3.2.2 of RFC 1738 [2]).
"?" query ::= query information, as per Section 3.3 of RFC 1738 [2].
"#" fragment ::= fragment identifier.
Note that the fragment identifier (and the "#" that precedes it) is not considered part of the URL. However, since it is commonly used within the same string context as a URL, a parser must be able to recognize the fragment when it is present and set it aside as part of the parsing process.
The order of the components is important. If both
2.2. BNF for Relative URLs
This is a BNF-like description of the Relative Uniform Resource Locator syntax, using the conventions of RFC 822 [5], except that "|" is used to designate alternatives. Briefly, literals are quoted with "", parentheses "(" and ")" are used to group elements, optional elements are enclosed in [brackets], and elements may be preceded with
This BNF also describes the generic-RL syntax for valid base URLs. Note that this differs from the URL syntax defined in RFC 1738 [2] in that all schemes are required to use a single set of reserved characters and use them consistently within the major URL components.
Formatted and Indexed by: page 3 of 16 instructional media + magic, inc. RFC1808 Web Address: Relative Uniform Resource Locators http://sunsite.cnlab-switch.ch/ By: Fielding
URL = ( absoluteURL | relativeURL ) [ "#" fragment ]
absoluteURL = generic-RL | ( scheme ":" *( uchar | reserved ) )
generic-RL = scheme ":" relativeURL
relativeURL = net_path | abs_path | rel_path
net_path = "//" net_loc [ abs_path ] abs_path = "/" rel_path rel_path = [ path ] [ ";" params ] [ "?" query ]
path = fsegment *( "/" segment ) fsegment = 1*pchar segment = *pchar
params = param *( ";" param ) param = *( pchar | "/" )
scheme = 1*( alpha | digit | "+" | "-" | "." ) net_loc = *( pchar | ";" | "?" ) query = *( uchar | reserved ) fragment = *( uchar | reserved )
pchar = uchar | ":" | "@" | "&" | "=" uchar = unreserved | escape unreserved = alpha | digit | safe | extra
escape = "%" hex hex hex = digit | "A" | "B" | "C" | "D" | "E" | "F" | "a" | "b" | "c" | "d" | "e" | "f"
alpha = lowalpha | hialpha lowalpha = "a" | "b" | "c" | "d" | "e" | "f" | "g" | "h" | "i" | "j" | "k" | "l" | "m" | "n" | "o" | "p" | "q" | "r" | "s" | "t" | "u" | "v" | "w" | "x" | "y" | "z" hialpha = "A" | "B" | "C" | "D" | "E" | "F" | "G" | "H" | "I" | "J" | "K" | "L" | "M" | "N" | "O" | "P" | "Q" | "R" | "S" | "T" | "U" | "V" | "W" | "X" | "Y" | "Z"
digit = "0" | "1" | "2" | "3" | "4" | "5" | "6" | "7" | "8" | "9"
safe = "$" | "-" | "_" | "." | "+" extra = "!" | "*" | "'" | "(" | ")" | "," national = "{" | "}" | "|" | "\" | "^" | "~" | "[" | "]" | "`" reserved = ";" | "/" | "?" | ":" | "@" | "&" | "=" punctuation = "<" | ">" | "#" | "%" | <">
Formatted and Indexed by: page 4 of 16 instructional media + magic, inc. RFC1808 Web Address: Relative Uniform Resource Locators http://sunsite.cnlab-switch.ch/ By: Fielding
2.3. Specific Schemes and their Syntactic Categories
Each URL scheme has its own rules regarding the presence or absence of the syntactic components described in Sections 2.1 and 2.2. In addition, some schemes are never appropriate for use with relative URLs. However, since relative URLs will only be used within contexts in which they are useful, these scheme-specific differences can be ignored by the resolution process.
Within this section, we include as examples only those schemes that have a defined URL syntax in RFC 1738 [2]. The following schemes are never used with relative URLs:
mailto Electronic Mail news USENET news telnet TELNET Protocol for Interactive Sessions
Some URL schemes allow the use of reserved characters for purposes outside the generic-RL syntax given above. However, such use is rare. Relative URLs can be used with these schemes whenever the applicable base URL follows the generic-RL syntax.
gopher Gopher and Gopher+ Protocols prospero Prospero Directory Service wais Wide Area Information Servers Protocol
Users of gopher URLs should note that gopher-type information is almost always included at the beginning of what would be the generic-RL path. If present, this type information prevents relative-path references to documents with differing gopher-types.
Finally, the following schemes can always be parsed using the generic-RL syntax. This does not necessarily imply that relative URLs will be useful with these schemes -- that decision is left to the system implementation and the author of the base document.
file Host-specific Files ftp File Transfer Protocol http Hypertext Transfer Protocol nntp USENET news using NNTP access
NOTE: Section 5 of RFC 1738 specifies that the question-mark character ("?") is allowed in an ftp or file path segment. However, this is not true in practice and is believed to be an error in the RFC. Similarly, RFC 1738 allows the reserved character semicolon (";") within an http path segment, but does not define its semantics; the correct semantics are as defined by this document for
Formatted and Indexed by: page 5 of 16 instructional media + magic, inc. RFC1808 Web Address: Relative Uniform Resource Locators http://sunsite.cnlab-switch.ch/ By: Fielding
We recommend that new schemes be designed to be parsable via the generic-RL syntax if they are intended to be used with relative URLs. A description of the allowed relative forms should be included when a new scheme is registered, as per Section 4 of RFC 1738 [2].
2.4. Parsing a URL
An accepted method for parsing URLs is useful to clarify the generic-RL syntax of Section 2.2 and to describe the algorithm for resolving relative URLs presented in Section 4. This section describes the parsing rules for breaking down a URL (relative or absolute) into the component parts described in Section 2.1. The rules assume that the URL has already been separated from any surrounding text and copied to a "parse string". The rules are listed in the order in which they would be applied by the parser.
2.4.1. Parsing the Fragment Identifier
If the parse string contains a crosshatch "#" character, then the substring after the first (left-most) crosshatch "#" and up to the end of the parse string is the
Note that the fragment identifier is not considered part of the URL. However, since it is often attached to the URL, parsers must be able to recognize and set aside fragment identifiers as part of the process.
2.4.2. Parsing the Scheme
If the parse string contains a colon ":" after the first character and before any characters not allowed as part of a scheme name (i.e., any not an alphanumeric, plus "+", period ".", or hyphen "-"), the
2.4.3. Parsing the Network Location/Login
If the parse string begins with a double-slash "//", then the substring of characters after the double-slash and up to, but not including, the next slash "/" character is the network location/login (
Formatted and Indexed by: page 6 of 16 instructional media + magic, inc. RFC1808 Web Address: Relative Uniform Resource Locators http://sunsite.cnlab-switch.ch/ By: Fielding
continuing.
2.4.4. Parsing the Query Information
If the parse string contains a question mark "?" character, then the substring after the first (left-most) question mark "?" and up to the end of the parse string is the
2.4.5. Parsing the Parameters
If the parse string contains a semicolon ";" character, then the substring after the first (left-most) semicolon ";" and up to the end of the parse string is the parameters (
2.4.6. Parsing the Path
After the above steps, all that is left of the parse string is the URL
3. Establishing a Base URL
The term "relative URL" implies that there exists some absolute "base URL" against which the relative reference is applied. Indeed, the base URL is necessary to define the semantics of any embedded relative URLs; without it, a relative reference is meaningless. In order for relative URLs to be usable within a document, the base URL of that document must be known to the parser.
Formatted and Indexed by: page 7 of 16 instructional media + magic, inc. RFC1808 Web Address: Relative Uniform Resource Locators http://sunsite.cnlab-switch.ch/ By: Fielding
The base URL of a document can be established in one of four ways, listed below in order of precedence. The order of precedence can be thought of in terms of layers, where the innermost defined base URL has the highest precedence. This can be visualized graphically as:
.------. | .------. | | | .------. | | | | | .------. | | | | | | | (3.1) Base URL embedded in the | | | | | | | | document's content | | | | | | | `------' | | | | | | (3.2) Base URL of the encapsulating entity | | | | | | (message, document, or none). | | | | | `------' | | | | (3.3) URL used to retrieve the entity | | | `------' | | (3.4) Base URL = "" (undefined) | `------'
3.1. Base URL within Document Content
Within certain document media types, the base URL of the document can be embedded within the content itself such that it can be readily obtained by a parser. This can be useful for descriptive documents, such as tables of content, which may be transmitted to others through protocols other than their usual retrieval context (e.g., E-Mail or USENET news).
It is beyond the scope of this document to specify how, for each media type, the base URL can be embedded. It is assumed that user agents manipulating such media types will be able to obtain the appropriate syntax from that media type's specification. An example of how the base URL can be embedded in the Hypertext Markup Language (HTML) [3] is provided in an Appendix (Section 10).
Messages are considered to be composite documents. The base URL of a message can be specified within the message headers (or equivalent tagged metainformation) of the message. For protocols that make use of message headers like those described in RFC 822 [5], we recommend that the format of this header be:
base-header = "Base" ":" "
where "Base" is case-insensitive and any whitespace (including that used for line folding) inside the angle brackets is ignored. For example, the header field
Formatted and Indexed by: page 8 of 16 instructional media + magic, inc. RFC1808 Web Address: Relative Uniform Resource Locators http://sunsite.cnlab-switch.ch/ By: Fielding
Base:
would indicate that the base URL for that message is the string "http://www.ics.uci.edu/Test/a/b/c". The base URL for a message serves as both the base for any relative URLs within the message headers and the default base URL for documents enclosed within the message, as described in the next section.
Protocols which do not use the RFC 822 message header syntax, but which do allow some form of tagged metainformation to be included within messages, may define their own syntax for defining the base URL as part of a message.
3.2. Base URL from the Encapsulating Entity
If no base URL is embedded, the base URL of a document is defined by the document's retrieval context. For a document that is enclosed within another entity (such as a message or another document), the retrieval context is that entity; thus, the default base URL of the document is the base URL of the entity in which the document is encapsulated.
Composite media types, such as the "multipart/*" and "message/*" media types defined by MIME (RFC 1521, [4]), define a hierarchy of retrieval context for their enclosed documents. In other words, the retrieval context of a component part is the base URL of the composite entity of which it is a part. Thus, a composite entity can redefine the retrieval context of its component parts via the inclusion of a base-header, and this redefinition applies recursively for a hierarchy of composite parts. Note that this might not change the base URL of the components, since each component may include an embedded base URL or base-header that takes precedence over the retrieval context.
3.3. Base URL from the Retrieval URL
If no base URL is embedded and the document is not encapsulated within some other entity (e.g., the top level of a composite entity), then, if a URL was used to retrieve the base document, that URL shall be considered the base URL. Note that if the retrieval was the result of a redirected request, the last URL used (i.e., that which resulted in the actual retrieval of the document) is the base URL.
3.4. Default Base URL
If none of the conditions described in Sections 3.1 -- 3.3 apply, then the base URL is considered to be the empty string and all embedded URLs within that document are assumed to be absolute URLs.
Formatted and Indexed by: page 9 of 16 instructional media + magic, inc. RFC1808 Web Address: Relative Uniform Resource Locators http://sunsite.cnlab-switch.ch/ By: Fielding
It is the responsibility of the distributor(s) of a document containing relative URLs to ensure that the base URL for that document can be established. It must be emphasized that relative URLs cannot be used reliably in situations where the document's base URL is not well-defined.
4. Resolving Relative URLs
This section describes an example algorithm for resolving URLs within a context in which the URLs may be relative, such that the result is always a URL in absolute form. Although this algorithm cannot guarantee that the resulting URL will equal that intended by the original author, it does guarantee that any valid URL (relative or absolute) can be consistently transformed to an absolute form given a valid base URL.
The following steps are performed in order:
Step 1: The base URL is established according to the rules of Section 3. If the base URL is the empty string (unknown), the embedded URL is interpreted as an absolute URL and we are done.
Step 2: Both the base and embedded URLs are parsed into their component parts as described in Section 2.4.
a) If the embedded URL is entirely empty, it inherits the entire base URL (i.e., is set equal to the base URL) and we are done.
b) If the embedded URL starts with a scheme name, it is interpreted as an absolute URL and we are done.
c) Otherwise, the embedded URL inherits the scheme of the base URL.
Step 3: If the embedded URL's
Step 4: If the embedded URL path is preceded by a slash "/", the path is not relative and we skip to Step 7.
Formatted and Indexed by: page 10 of 16 instructional media + magic, inc. RFC1808 Web Address: Relative Uniform Resource Locators http://sunsite.cnlab-switch.ch/ By: Fielding
Step 5: If the embedded URL path is empty (and not preceded by a slash), then the embedded URL inherits the base URL path, and
a) if the embedded URL's
b) if the embedded URL's
Step 6: The last segment of the base URL's path (anything following the rightmost slash "/", or the entire path if no slash is present) is removed and the embedded URL's path is appended in its place. The following operations are then applied, in order, to the new path:
a) All occurrences of "./", where "." is a complete path segment, are removed.
b) If the path ends with "." as a complete path segment, that "." is removed.
c) All occurrences of "
d) If the path ends with "
Step 7: The resulting URL components, including any inherited from the base URL, are recombined to give the absolute form of the embedded URL.
Parameters, regardless of their purpose, do not form a part of the URL path and thus do not affect the resolving of relative paths. In particular, the presence or absence of the ";type=d" parameter on an ftp URL does not affect the interpretation of paths relative to that URL. Fragment identifiers are only inherited from the base URL when the entire embedded URL is empty.
Formatted and Indexed by: page 11 of 16 instructional media + magic, inc. RFC1808 Web Address: Relative Uniform Resource Locators http://sunsite.cnlab-switch.ch/ By: Fielding
The above algorithm is intended to provide an example by which the output of implementations can be tested -- implementation of the algorithm itself is not required. For example, some systems may find it more efficient to implement Step 6 as a pair of segment stacks being merged, rather than as a series of string pattern matches.
5. Examples and Recommended Practice
Within an object with a well-defined base URL of
Base:
the relative URLs would be resolved as follows:
5.1. Normal Examples
g:h =
5.2. Abnormal Examples
Although the following abnormal examples are unlikely to occur in normal practice, all URL parsers should be capable of resolving them consistently. Each example uses the same base as above.
Formatted and Indexed by: page 12 of 16 instructional media + magic, inc. RFC1808 Web Address: Relative Uniform Resource Locators http://sunsite.cnlab-switch.ch/ By: Fielding
An empty reference resolves to the complete base URL:
<> =
Parsers must be careful in handling the case where there are more relative path ".." segments than there are hierarchical levels in the base URL's path. Note that the ".." syntax cannot be used to change the
../../../g =
Similarly, parsers must avoid treating "." and ".." as special when they are not complete components of a relative path.
/./g =
Less likely are cases where the relative URL uses unnecessary or nonsensical forms of the "." and ".." complete path segments.
./../g =
Finally, some older parsers allow the scheme name to be present in a relative URL if it is the same as the base URL scheme. This is considered to be a loophole in prior specifications of partial URLs [1] and should be avoided by future parsers.
http:g =
5.3. Recommended Practice
Authors should be aware that path names which contain a colon ":" character cannot be used as the first component of a relative URL path (e.g., "this:that") because they will likely be mistaken for a scheme name. It is therefore necessary to precede such cases with other components (e.g., "./this:that"), or to escape the colon character (e.g., "this%3Athat"), in order for them to be correctly parsed. The former solution is preferred because it does not affect the absolute form of the URL.
Formatted and Indexed by: page 13 of 16 instructional media + magic, inc. RFC1808 Web Address: Relative Uniform Resource Locators http://sunsite.cnlab-switch.ch/ By: Fielding
There is an ambiguity in the semantics for the ftp URL scheme regarding the use of a trailing slash ("/") character and/or a parameter ";type=d" to indicate a resource that is an ftp directory. If the result of retrieving that directory includes embedded relative URLs, it is necessary that the base URL path for that result include a trailing slash. For this reason, we recommend that the ";type=d" parameter value not be used within contexts that allow relative URLs.
6. Security Considerations
There are no security considerations in the use or parsing of relative URLs. However, once a relative URL has been resolved to its absolute form, the same security considerations apply as those described in RFC 1738 [2].
7. Acknowledgements
This work is derived from concepts introduced by Tim Berners-Lee and the World-Wide Web global information initiative. Relative URLs are described as "Partial URLs" in RFC 1630 [1]. That description was expanded for inclusion as an appendix for an early draft of RFC 1738, "Uniform Resource Locators (URL)" [2]. However, after further discussion, the URI-WG decided to specify Relative URLs separately from the primary URL draft.
This document is intended to fulfill the recommendations for Internet Resource Locators as stated in [6]. It has benefited greatly from the comments of all those participating in the URI-WG. Particular thanks go to Larry Masinter, Michael A. Dolan, Guido van Rossum, Dave Kristol, David Robinson, and Brad Barber for identifying problems/deficiencies in earlier drafts.
8. References
[1] Berners-Lee, T., "Universal Resource Identifiers in WWW: A Unifying Syntax for the Expression of Names and Addresses of Objects on the Network as used in the World-Wide Web", RFC 1630, CERN, June 1994.
[2] Berners-Lee, T., Masinter, L., and M. McCahill, Editors, "Uniform Resource Locators (URL)", RFC 1738, CERN, Xerox Corporation, University of Minnesota, December 1994.
[3] Berners-Lee T., and D. Connolly, "HyperText Markup Language Specification -- 2.0", Work in Progress, MIT, HaL Computer Systems, February 1995.
Formatted and Indexed by: page 14 of 16 instructional media + magic, inc. RFC1808 Web Address: Relative Uniform Resource Locators http://sunsite.cnlab-switch.ch/ By: Fielding
[4] Borenstein, N., and N. Freed, "MIME (Multipurpose Internet Mail Extensions): Mechanisms for Specifying and Describing the Format of Internet Message Bodies", RFC 1521, Bellcore, Innosoft, September 1993.
[5] Crocker, D., "Standard for the Format of ARPA Internet Text Messages", STD 11, RFC 822, UDEL, August 1982.
[6] Kunze, J., "Functional Recommendations for Internet Resource Locators", RFC 1736, IS&T, UC Berkeley, February 1995.
9. Author's Address
Roy T. Fielding Department of Information and Computer Science University of California Irvine, CA 92717-3425 U.S.A.
Tel: +1 (714) 824-4049 Fax: +1 (714) 824-4056 EMail: [email protected]
10. Appendix - Embedding the Base URL in HTML documents
It is useful to consider an example of how the base URL of a document can be embedded within the document's content. In this appendix, we describe how documents written in the Hypertext Markup Language (HTML) [3] can include an embedded base URL. This appendix does not form a part of the relative URL specification and should not be considered as anything more than a descriptive example.
HTML defines a special element "BASE" which, when present in the "HEAD" portion of a document, signals that the parser should use the BASE element's "HREF" attribute as the base URL for resolving any relative URLs. The "HREF" attribute must be an absolute URL. Note that, in HTML, element and attribute names are case-insensitive. For example:
Formatted and Indexed by: page 15 of 16 instructional media + magic, inc. RFC1808 Web Address: Relative Uniform Resource Locators http://sunsite.cnlab-switch.ch/ By: Fielding
A parser reading the example document should interpret the given relative URL "../x" as representing the absolute URL
regardless of the context in which the example document was obtained.
_
Formatted and Indexed by: page 16 of 16 instructional media + magic, inc. RFC1945 Web Address: HTTP/1.0 http://sunsite.cnlab-switch.ch/ By: Berners-Lee, et al
Network Working Group T. Berners-Lee Request for Comments: 1945 MIT/LCS Category: Informational R. Fielding UC Irvine H. Frystyk MIT/LCS May 1996
Hypertext Transfer Protocol -- HTTP/1.0
Status of This Memo
This memo provides information for the Internet community. This memo does not specify an Internet standard of any kind. Distribution of this memo is unlimited.
IESG Note:
The IESG has concerns about this protocol, and expects this document to be replaced relatively soon by a standards track document.
Abstract
The Hypertext Transfer Protocol (HTTP) is an application-level protocol with the lightness and speed necessary for distributed, collaborative, hypermedia information systems. It is a generic, stateless, object-oriented protocol which can be used for many tasks, such as name servers and distributed object management systems, through extension of its request methods (commands). A feature of HTTP is the typing of data representation, allowing systems to be built independently of the data being transferred.
HTTP has been in use by the World-Wide Web global information initiative since 1990. This specification reflects common usage of the protocol referred to as "HTTP/1.0".
Table of Contents
1. Introduction ...... 4 1.1 Purpose ...... 4 1.2 Terminology ...... 4 1.3 Overall Operation ...... 6 1.4 HTTP and MIME ...... 8 2. Notational Conventions and Generic Grammar ...... 8 2.1 Augmented BNF ...... 8 2.2 Basic Rules ...... 10 3. Protocol Parameters ...... 12
Formatted and Indexed by: page 1 of 60 instructional media + magic, inc. RFC1945 Web Address: HTTP/1.0 http://sunsite.cnlab-switch.ch/ By: Berners-Lee, et al
3.1 HTTP Version ...... 12 3.2 Uniform Resource Identifiers ...... 14 3.2.1 General Syntax ...... 14 3.2.2 http URL ...... 15 3.3 Date/Time Formats ...... 15 3.4 Character Sets ...... 17 3.5 Content Codings ...... 18 3.6 Media Types ...... 19 3.6.1 Canonicalization and Text Defaults ...... 19 3.6.2 Multipart Types ...... 20 3.7 Product Tokens ...... 20 4. HTTP Message ...... 21 4.1 Message Types ...... 21 4.2 Message Headers ...... 22 4.3 General Header Fields ...... 23 5. Request ...... 23 5.1 Request-Line ...... 23 5.1.1 Method ...... 24 5.1.2 Request-URI ...... 24 5.2 Request Header Fields ...... 25 6. Response ...... 25 6.1 Status-Line ...... 26 6.1.1 Status Code and Reason Phrase ...... 26 6.2 Response Header Fields ...... 28 7. Entity ...... 28 7.1 Entity Header Fields ...... 29 7.2 Entity Body ...... 29 7.2.1 Type ...... 29 7.2.2 Length ...... 30 8. Method Definitions ...... 30 8.1 GET ...... 31 8.2 HEAD ...... 31 8.3 POST ...... 31 9. Status Code Definitions ...... 32 9.1 Informational 1xx ...... 32 9.2 Successful 2xx ...... 32 9.3 Redirection 3xx ...... 34 9.4 Client Error 4xx ...... 35 9.5 Server Error 5xx ...... 37 10. Header Field Definitions ...... 37 10.1 Allow ...... 38 10.2 Authorization ...... 38 10.3 Content-Encoding ...... 39 10.4 Content-Length ...... 39 10.5 Content-Type ...... 40 10.6 Date ...... 40 10.7 Expires ...... 41 10.8 From ...... 42
Formatted and Indexed by: page 2 of 60 instructional media + magic, inc. RFC1945 Web Address: HTTP/1.0 http://sunsite.cnlab-switch.ch/ By: Berners-Lee, et al
10.9 If-Modified-Since ...... 42 10.10 Last-Modified ...... 43 10.11 Location ...... 44 10.12 Pragma ...... 44 10.13 Referer ...... 44 10.14 Server ...... 45 10.15 User-Agent ...... 46 10.16 WWW-Authenticate ...... 46 11. Access Authentication ...... 47 11.1 Basic Authentication Scheme ...... 48 12. Security Considerations ...... 49 12.1 Authentication of Clients ...... 49 12.2 Safe Methods ...... 49 12.3 Abuse of Server Log Information ...... 50 12.4 Transfer of Sensitive Information ...... 50 12.5 Attacks Based On File and Path Names ...... 51 13. Acknowledgments ...... 51 14. References ...... 52 15. Authors' Addresses ...... 54 Appendix A. Internet Media Type message/http ...... 55 Appendix B. Tolerant Applications ...... 55 Appendix C. Relationship to MIME ...... 56 C.1 Conversion to Canonical Form ...... 56 C.2 Conversion of Date Formats ...... 57 C.3 Introduction of Content-Encoding ...... 57 C.4 No Content-Transfer-Encoding ...... 57 C.5 HTTP Header Fields in Multipart Body-Parts ...... 57 Appendix D. Additional Features ...... 57 D.1 Additional Request Methods ...... 58 D.1.1 PUT ...... 58 D.1.2 DELETE ...... 58 D.1.3 LINK ...... 58 D.1.4 UNLINK ...... 58 D.2 Additional Header Field Definitions ...... 58 D.2.1 Accept ...... 58 D.2.2 Accept-Charset ...... 59 D.2.3 Accept-Encoding ...... 59 D.2.4 Accept-Language ...... 59 D.2.5 Content-Language ...... 59 D.2.6 Link ...... 59 D.2.7 MIME-Version ...... 59 D.2.8 Retry-After ...... 60 D.2.9 Title ...... 60 D.2.10 URI ...... 60
Formatted and Indexed by: page 3 of 60 instructional media + magic, inc. RFC1945 Web Address: HTTP/1.0 http://sunsite.cnlab-switch.ch/ By: Berners-Lee, et al
1. Introduction
1.1 Purpose
The Hypertext Transfer Protocol (HTTP) is an application-level protocol with the lightness and speed necessary for distributed, collaborative, hypermedia information systems. HTTP has been in use by the World-Wide Web global information initiative since 1990. This specification reflects common usage of the protocol referred too as "HTTP/1.0". This specification describes the features that seem to be consistently implemented in most HTTP/1.0 clients and servers. The specification is split into two sections. Those features of HTTP for which implementations are usually consistent are described in the main body of this document. Those features which have few or inconsistent implementations are listed in Appendix D.
Practical information systems require more functionality than simple retrieval, including search, front-end update, and annotation. HTTP allows an open-ended set of methods to be used to indicate the purpose of a request. It builds on the discipline of reference provided by the Uniform Resource Identifier (URI) [2], as a location (URL) [4] or name (URN) [16], for indicating the resource on which a method is to be applied. Messages are passed in a format similar to that used by Internet Mail [7] and the Multipurpose Internet Mail Extensions (MIME) [5].
HTTP is also used as a generic protocol for communication between user agents and proxies/gateways to other Internet protocols, such as SMTP [12], NNTP [11], FTP [14], Gopher [1], and WAIS [8], allowing basic hypermedia access to resources available from diverse applications and simplifying the implementation of user agents.
1.2 Terminology
This specification uses a number of terms to refer to the roles played by participants in, and objects of, the HTTP communication.
connection
A transport layer virtual circuit established between two application programs for the purpose of communication.
message
The basic unit of HTTP communication, consisting of a structured sequence of octets matching the syntax defined in Section 4 and transmitted via the connection.
_
Formatted and Indexed by: page 4 of 60 instructional media + magic, inc. RFC1945 Web Address: HTTP/1.0 http://sunsite.cnlab-switch.ch/ By: Berners-Lee, et al
request
An HTTP request message (as defined in Section 5).
response
An HTTP response message (as defined in Section 6).
resource
A network data object or service which can be identified by a URI (Section 3.2).
entity
A particular representation or rendition of a data resource, or reply from a service resource, that may be enclosed within a request or response message. An entity consists of metainformation in the form of entity headers and content in the form of an entity body.
client
An application program that establishes connections for the purpose of sending requests.
user agent
The client which initiates a request. These are often browsers, editors, spiders (web-traversing robots), or other end user tools.
server
An application program that accepts connections in order to service requests by sending back responses.
origin server
The server on which a given resource resides or is to be created.
proxy
An intermediary program which acts as both a server and a client for the purpose of making requests on behalf of other clients. Requests are serviced internally or by passing them, with possible translation, on to other servers. A proxy must interpret and, if necessary, rewrite a request message before
Formatted and Indexed by: page 5 of 60 instructional media + magic, inc. RFC1945 Web Address: HTTP/1.0 http://sunsite.cnlab-switch.ch/ By: Berners-Lee, et al
forwarding it. Proxies are often used as client-side portals through network firewalls and as helper applications for handling requests via protocols not implemented by the user agent.
gateway
A server which acts as an intermediary for some other server. Unlike a proxy, a gateway receives requests as if it were the origin server for the requested resource; the requesting client may not be aware that it is communicating with a gateway. Gateways are often used as server-side portals through network firewalls and as protocol translators for access to resources stored on non-HTTP systems.
tunnel
A tunnel is an intermediary program which is acting as a blind relay between two connections. Once active, a tunnel is not considered a party to the HTTP communication, though the tunnel may have been initiated by an HTTP request. The tunnel ceases to exist when both ends of the relayed connections are closed. Tunnels are used when a portal is necessary and the intermediary cannot, or should not, interpret the relayed communication.
cache
A program's local store of response messages and the subsystem that controls its message storage, retrieval, and deletion. A cache stores cachable responses in order to reduce the response time and network bandwidth consumption on future, equivalent requests. Any client or server may include a cache, though a cache cannot be used by a server while it is acting as a tunnel.
Any given program may be capable of being both a client and a server; our use of these terms refers only to the role being performed by the program for a particular connection, rather than to the program's capabilities in general. Likewise, any server may act as an origin server, proxy, gateway, or tunnel, switching behavior based on the nature of each request.
1.3 Overall Operation
The HTTP protocol is based on a request/response paradigm. A client establishes a connection with a server and sends a request to the server in the form of a request method, URI, and protocol version, followed by a MIME-like message containing request modifiers, client information, and possible body content. The server responds with a
Formatted and Indexed by: page 6 of 60 instructional media + magic, inc. RFC1945 Web Address: HTTP/1.0 http://sunsite.cnlab-switch.ch/ By: Berners-Lee, et al
status line, including the message's protocol version and a success or error code, followed by a MIME-like message containing server information, entity metainformation, and possible body content.
Most HTTP communication is initiated by a user agent and consists of a request to be applied to a resource on some origin server. In the simplest case, this may be accomplished via a single connection (v) between the user agent (UA) and the origin server (O).
request chain ------> UA ------v------O <------response chain
A more complicated situation occurs when one or more intermediaries are present in the request/response chain. There are three common forms of intermediary: proxy, gateway, and tunnel. A proxy is a forwarding agent, receiving requests for a URI in its absolute form, rewriting all or parts of the message, and forwarding the reformatted request toward the server identified by the URI. A gateway is a receiving agent, acting as a layer above some other server(s) and, if necessary, translating the requests to the underlying server's protocol. A tunnel acts as a relay point between two connections without changing the messages; tunnels are used when the communication needs to pass through an intermediary (such as a firewall) even when the intermediary cannot understand the contents of the messages.
request chain ------> UA -----v----- A -----v----- B -----v----- C -----v----- O <------response chain
The figure above shows three intermediaries (A, B, and C) between the user agent and origin server. A request or response message that travels the whole chain must pass through four separate connections. This distinction is important because some HTTP communication options may apply only to the connection with the nearest, non-tunnel neighbor, only to the end-points of the chain, or to all connections along the chain. Although the diagram is linear, each participant may be engaged in multiple, simultaneous communications. For example, B may be receiving requests from many clients other than A, and/or forwarding requests to servers other than C, at the same time that it is handling A's request.
Any party to the communication which is not acting as a tunnel may employ an internal cache for handling requests. The effect of a cache is that the request/response chain is shortened if one of the participants along the chain has a cached response applicable to that request. The following illustrates the resulting chain if B has a
Formatted and Indexed by: page 7 of 60 instructional media + magic, inc. RFC1945 Web Address: HTTP/1.0 http://sunsite.cnlab-switch.ch/ By: Berners-Lee, et al
cached copy of an earlier response from O (via C) for a request which has not been cached by UA or A.
request chain ------> UA -----v----- A -----v----- B ------C ------O <------response chain
Not all responses are cachable, and some requests may contain modifiers which place special requirements on cache behavior. Some HTTP/1.0 applications use heuristics to describe what is or is not a "cachable" response, but these rules are not standardized.
On the Internet, HTTP communication generally takes place over TCP/IP connections. The default port is TCP 80 [15], but other ports can be used. This does not preclude HTTP from being implemented on top of any other protocol on the Internet, or on other networks. HTTP only presumes a reliable transport; any protocol that provides such guarantees can be used, and the mapping of the HTTP/1.0 request and response structures onto the transport data units of the protocol in question is outside the scope of this specification.
Except for experimental applications, current practice requires that the connection be established by the client prior to each request and closed by the server after sending the response. Both clients and servers should be aware that either party may close the connection prematurely, due to user action, automated time-out, or program failure, and should handle such closing in a predictable fashion. In any case, the closing of the connection by either or both parties always terminates the current request, regardless of its status.
1.4 HTTP and MIME
HTTP/1.0 uses many of the constructs defined for MIME, as defined in RFC 1521 [5]. Appendix C describes the ways in which the context of HTTP allows for different use of Internet Media Types than is typically found in Internet mail, and gives the rationale for those differences.
2. Notational Conventions and Generic Grammar
2.1 Augmented BNF
All of the mechanisms specified in this document are described in both prose and an augmented Backus-Naur Form (BNF) similar to that used by RFC 822 [7]. Implementors will need to be familiar with the notation in order to understand this specification. The augmented BNF includes the following constructs:
Formatted and Indexed by: page 8 of 60 instructional media + magic, inc. RFC1945 Web Address: HTTP/1.0 http://sunsite.cnlab-switch.ch/ By: Berners-Lee, et al
name = definition
The name of a rule is simply the name itself (without any enclosing "<" and ">") and is separated from its definition by the equal character "=". Whitespace is only significant in that indentation of continuation lines is used to indicate a rule definition that spans more than one line. Certain basic rules are in uppercase, such as SP, LWS, HT, CRLF, DIGIT, ALPHA, etc. Angle brackets are used within definitions whenever their presence will facilitate discerning the use of rule names.
"literal"
Quotation marks surround literal text. Unless stated otherwise, the text is case-insensitive.
rule1 | rule2
Elements separated by a bar ("I") are alternatives, e.g., "yes | no" will accept yes or no.
(rule1 rule2)
Elements enclosed in parentheses are treated as a single element. Thus, "(elem (foo | bar) elem)" allows the token sequences "elem foo elem" and "elem bar elem".
*rule
The character "*" preceding an element indicates repetition. The full form is "
[rule]
Square brackets enclose optional elements; "[foo bar]" is equivalent to "*1(foo bar)".
N rule
Specific repetition: "
Formatted and Indexed by: page 9 of 60 instructional media + magic, inc. RFC1945 Web Address: HTTP/1.0 http://sunsite.cnlab-switch.ch/ By: Berners-Lee, et al
#rule
A construct "#" is defined, similar to "*", for defining lists of elements. The full form is "
; comment
A semi-colon, set off some distance to the right of rule text, starts a comment that continues to the end of line. This is a simple way of including useful notes in parallel with the specifications.
implied *LWS
The grammar described by this specification is word-based. Except where noted otherwise, linear whitespace (LWS) can be included between any two adjacent words (token or quoted-string), and between adjacent tokens and delimiters (tspecials), without changing the interpretation of a field. At least one delimiter (tspecials) must exist between any two tokens, since they would otherwise be interpreted as a single token. However, applications should attempt to follow "common form" when generating HTTP constructs, since there exist some implementations that fail to accept anything beyond the common forms.
2.2 Basic Rules
The following rules are used throughout this specification to describe basic parsing constructs. The US-ASCII coded character set is defined by [17].
OCTET =
Formatted and Indexed by: page 10 of 60 instructional media + magic, inc. RFC1945 Web Address: HTTP/1.0 http://sunsite.cnlab-switch.ch/ By: Berners-Lee, et al
ALPHA = UPALPHA | LOALPHA DIGIT =
HTTP/1.0 defines the octet sequence CR LF as the end-of-line marker for all protocol elements except the Entity-Body (see Appendix B for tolerant applications). The end-of-line marker within an Entity-Body is defined by its associated media type, as described in Section 3.6.
CRLF = CR LF
HTTP/1.0 headers may be folded onto multiple lines if each continuation line begins with a space or horizontal tab. All linear whitespace, including folding, has the same semantics as SP.
LWS = [CRLF] 1*( SP | HT )
However, folding of header lines is not expected by some applications, and should not be generated by HTTP/1.0 applications.
The TEXT rule is only used for descriptive field contents and values that are not intended to be interpreted by the message parser. Words of *TEXT may contain octets from character sets other than US-ASCII.
TEXT =
Recipients of header field TEXT containing octets outside the US- ASCII character set may assume that they represent ISO-8859-1 characters.
Hexadecimal numeric characters are used in several protocol elements.
HEX = "A" | "B" | "C" | "D" | "E" | "F" | "a" | "b" | "c" | "d" | "e" | "f" | DIGIT
Many HTTP/1.0 header field values consist of words separated by LWS or special characters. These special characters must be in a quoted string to be used within a parameter value.
word = token | quoted-string
Formatted and Indexed by: page 11 of 60 instructional media + magic, inc. RFC1945 Web Address: HTTP/1.0 http://sunsite.cnlab-switch.ch/ By: Berners-Lee, et al
token = 1*
tspecials = "(" | ")" | "<" | ">" | "@" | "," | ";" | ":" | "\" | <"> | "/" | "[" | "]" | "?" | "=" | "{" | "}" | SP | HT
Comments may be included in some HTTP header fields by surrounding the comment text with parentheses. Comments are only allowed in fields containing "comment" as part of their field value definition. In all other fields, parentheses are considered part of the field value.
comment = "(" *( ctext | comment ) ")" ctext =
A string of text is parsed as a single word if it is quoted using double-quote marks.
quoted-string = ( <"> *(qdtext) <"> )
qdtext =
Single-character quoting using the backslash ("\") character is not permitted in HTTP/1.0.
3. Protocol Parameters
3.1 HTTP Version
HTTP uses a "
The version of an HTTP message is indicated by an HTTP-Version field in the first line of the message. If the protocol version is not specified, the recipient must assume that the message is in the
Formatted and Indexed by: page 12 of 60 instructional media + magic, inc. RFC1945 Web Address: HTTP/1.0 http://sunsite.cnlab-switch.ch/ By: Berners-Lee, et al
simple HTTP/0.9 format.
HTTP-Version = "HTTP" "/" 1*DIGIT "." 1*DIGIT
Note that the major and minor numbers should be treated as separate integers and that each may be incremented higher than a single digit. Thus, HTTP/2.4 is a lower version than HTTP/2.13, which in turn is lower than HTTP/12.3. Leading zeros should be ignored by recipients and never generated by senders.
This document defines both the 0.9 and 1.0 versions of the HTTP protocol. Applications sending Full-Request or Full-Response messages, as defined by this specification, must include an HTTP- Version of "HTTP/1.0".
HTTP/1.0 servers must:
o recognize the format of the Request-Line for HTTP/0.9 and HTTP/1.0 requests;
o understand any valid request in the format of HTTP/0.9 or HTTP/1.0;
o respond appropriately with a message in the same protocol version used by the client.
HTTP/1.0 clients must:
o recognize the format of the Status-Line for HTTP/1.0 responses;
o understand any valid response in the format of HTTP/0.9 or HTTP/1.0.
Proxy and gateway applications must be careful in forwarding requests that are received in a format different than that of the application's native HTTP version. Since the protocol version indicates the protocol capability of the sender, a proxy/gateway must never send a message with a version indicator which is greater than its native version; if a higher version request is received, the proxy/gateway must either downgrade the request version or respond with an error. Requests with a version lower than that of the application's native format may be upgraded before being forwarded; the proxy/gateway's response to that request must follow the server requirements listed above.
Formatted and Indexed by: page 13 of 60 instructional media + magic, inc. RFC1945 Web Address: HTTP/1.0 http://sunsite.cnlab-switch.ch/ By: Berners-Lee, et al
3.2 Uniform Resource Identifiers
URIs have been known by many names: WWW addresses, Universal Document Identifiers, Universal Resource Identifiers [2], and finally the combination of Uniform Resource Locators (URL) [4] and Names (URN) [16]. As far as HTTP is concerned, Uniform Resource Identifiers are simply formatted strings which identify--via name, location, or any other characteristic--a network resource.
3.2.1 General Syntax
URIs in HTTP can be represented in absolute form or relative to some known base URI [9], depending upon the context of their use. The two forms are differentiated by the fact that absolute URIs always begin with a scheme name followed by a colon.
URI = ( absoluteURI | relativeURI ) [ "#" fragment ]
absoluteURI = scheme ":" *( uchar | reserved )
relativeURI = net_path | abs_path | rel_path
net_path = "//" net_loc [ abs_path ] abs_path = "/" rel_path rel_path = [ path ] [ ";" params ] [ "?" query ]
path = fsegment *( "/" segment ) fsegment = 1*pchar segment = *pchar
params = param *( ";" param ) param = *( pchar | "/" )
scheme = 1*( ALPHA | DIGIT | "+" | "-" | "." ) net_loc = *( pchar | ";" | "?" ) query = *( uchar | reserved ) fragment = *( uchar | reserved )
pchar = uchar | ":" | "@" | "&" | "=" | "+" uchar = unreserved | escape unreserved = ALPHA | DIGIT | safe | extra | national
escape = "%" HEX HEX reserved = ";" | "/" | "?" | ":" | "@" | "&" | "=" | "+" extra = "!" | "*" | "'" | "(" | ")" | "," safe = "$" | "-" | "_" | "." unsafe = CTL | SP | <"> | "#" | "%" | "<" | ">" national = Formatted and Indexed by: page 14 of 60 instructional media + magic, inc. RFC1945 Web Address: HTTP/1.0 http://sunsite.cnlab-switch.ch/ By: Berners-Lee, et al reserved, extra, safe, and unsafe> For definitive information on URL syntax and semantics, see RFC 1738 [4] and RFC 1808 [9]. The BNF above includes national characters not allowed in valid URLs as specified by RFC 1738, since HTTP servers are not restricted in the set of unreserved characters allowed to represent the rel_path part of addresses, and HTTP proxies may receive requests for URIs not defined by RFC 1738. 3.2.2 http URL The "http" scheme is used to locate network resources via the HTTP protocol. This section defines the scheme-specific syntax and semantics for http URLs. http_URL = "http:" "//" host [ ":" port ] [ abs_path ] port = *DIGIT If the port is empty or not given, port 80 is assumed. The semantics are that the identified resource is located at the server listening for TCP connections on that port of that host, and the Request-URI for the resource is abs_path. If the abs_path is not present in the URL, it must be given as "/" when used as a Request-URI (Section 5.1.2). Note: Although the HTTP protocol is independent of the transport layer protocol, the http URL only identifies resources by their TCP location, and thus non-TCP resources must be identified by some other URI scheme. The canonical form for "http" URLs is obtained by converting any UPALPHA characters in host to their LOALPHA equivalent (hostnames are case-insensitive), eliding the [ ":" port ] if the port is 80, and replacing an empty abs_path with "/". 3.3 Date/Time Formats HTTP/1.0 applications have historically allowed three different formats for the representation of date/time stamps: Sun, 06 Nov 1994 08:49:37 GMT ; RFC 822, updated by RFC 1123 Sunday, 06-Nov-94 08:49:37 GMT ; RFC 850, obsoleted by RFC 1036 Sun Nov 6 08:49:37 1994 ; ANSI C's asctime() format Formatted and Indexed by: page 15 of 60 instructional media + magic, inc. RFC1945 Web Address: HTTP/1.0 http://sunsite.cnlab-switch.ch/ By: Berners-Lee, et al The first format is preferred as an Internet standard and represents a fixed-length subset of that defined by RFC 1123 [6] (an update to RFC 822 [7]). The second format is in common use, but is based on the obsolete RFC 850 [10] date format and lacks a four-digit year. HTTP/1.0 clients and servers that parse the date value should accept all three formats, though they must never generate the third (asctime) format. Note: Recipients of date values are encouraged to be robust in accepting date values that may have been generated by non-HTTP applications, as is sometimes the case when retrieving or posting messages via proxies/gateways to SMTP or NNTP. All HTTP/1.0 date/time stamps must be represented in Universal Time (UT), also known as Greenwich Mean Time (GMT), without exception. This is indicated in the first two formats by the inclusion of "GMT" as the three-letter abbreviation for time zone, and should be assumed when reading the asctime format. HTTP-date = rfc1123-date | rfc850-date | asctime-date rfc1123-date = wkday "," SP date1 SP time SP "GMT" rfc850-date = weekday "," SP date2 SP time SP "GMT" asctime-date = wkday SP date3 SP time SP 4DIGIT date1 = 2DIGIT SP month SP 4DIGIT ; day month year (e.g., 02 Jun 1982) date2 = 2DIGIT "-" month "-" 2DIGIT ; day-month-year (e.g., 02-Jun-82) date3 = month SP ( 2DIGIT | ( SP 1DIGIT )) ; month day (e.g., Jun 2) time = 2DIGIT ":" 2DIGIT ":" 2DIGIT ; 00:00:00 - 23:59:59 wkday = "Mon" | "Tue" | "Wed" | "Thu" | "Fri" | "Sat" | "Sun" weekday = "Monday" | "Tuesday" | "Wednesday" | "Thursday" | "Friday" | "Saturday" | "Sunday" month = "Jan" | "Feb" | "Mar" | "Apr" | "May" | "Jun" | "Jul" | "Aug" | "Sep" | "Oct" | "Nov" | "Dec" Note: HTTP requirements for the date/time stamp format apply only to their usage within the protocol stream. Clients and servers are not required to use these formats for user Formatted and Indexed by: page 16 of 60 instructional media + magic, inc. RFC1945 Web Address: HTTP/1.0 http://sunsite.cnlab-switch.ch/ By: Berners-Lee, et al presentation, request logging, etc. 3.4 Character Sets HTTP uses the same definition of the term "character set" as that described for MIME: The term "character set" is used in this document to refer to a method used with one or more tables to convert a sequence of octets into a sequence of characters. Note that unconditional conversion in the other direction is not required, in that not all characters may be available in a given character set and a character set may provide more than one sequence of octets to represent a particular character. This definition is intended to allow various kinds of character encodings, from simple single- table mappings such as US-ASCII to complex table switching methods such as those that use ISO 2022's techniques. However, the definition associated with a MIME character set name must fully specify the mapping to be performed from octets to characters. In particular, use of external profiling information to determine the exact mapping is not permitted. Note: This use of the term "character set" is more commonly referred to as a "character encoding." However, since HTTP and MIME share the same registry, it is important that the terminology also be shared. HTTP character sets are identified by case-insensitive tokens. The complete set of tokens are defined by the IANA Character Set registry [15]. However, because that registry does not define a single, consistent token for each character set, we define here the preferred names for those character sets most likely to be used with HTTP entities. These character sets include those registered by RFC 1521 [5] -- the US-ASCII [17] and ISO-8859 [18] character sets -- and other names specifically recommended for use within MIME charset parameters. charset = "US-ASCII" | "ISO-8859-1" | "ISO-8859-2" | "ISO-8859-3" | "ISO-8859-4" | "ISO-8859-5" | "ISO-8859-6" | "ISO-8859-7" | "ISO-8859-8" | "ISO-8859-9" | "ISO-2022-JP" | "ISO-2022-JP-2" | "ISO-2022-KR" | "UNICODE-1-1" | "UNICODE-1-1-UTF-7" | "UNICODE-1-1-UTF-8" | token Although HTTP allows an arbitrary token to be used as a charset value, any token that has a predefined value within the IANA Character Set registry [15] must represent the character set defined Formatted and Indexed by: page 17 of 60 instructional media + magic, inc. RFC1945 Web Address: HTTP/1.0 http://sunsite.cnlab-switch.ch/ By: Berners-Lee, et al by that registry. Applications should limit their use of character sets to those defined by the IANA registry. The character set of an entity body should be labelled as the lowest common denominator of the character codes used within that body, with the exception that no label is preferred over the labels US-ASCII or ISO-8859-1. 3.5 Content Codings Content coding values are used to indicate an encoding transformation that has been applied to a resource. Content codings are primarily used to allow a document to be compressed or encrypted without losing the identity of its underlying media type. Typically, the resource is stored in this encoding and only decoded before rendering or analogous usage. content-coding = "x-gzip" | "x-compress" | token Note: For future compatibility, HTTP/1.0 applications should consider "gzip" and "compress" to be equivalent to "x-gzip" and "x-compress", respectively. All content-coding values are case-insensitive. HTTP/1.0 uses content-coding values in the Content-Encoding (Section 10.3) header field. Although the value describes the content-coding, what is more important is that it indicates what decoding mechanism will be required to remove the encoding. Note that a single program may be capable of decoding multiple content-coding formats. Two values are defined by this specification: x-gzip An encoding format produced by the file compression program "gzip" (GNU zip) developed by Jean-loup Gailly. This format is typically a Lempel-Ziv coding (LZ77) with a 32 bit CRC. x-compress The encoding format produced by the file compression program "compress". This format is an adaptive Lempel-Ziv-Welch coding (LZW). Note: Use of program names for the identification of encoding formats is not desirable and should be discouraged for future encodings. Their use here is representative of historical practice, not good design. Formatted and Indexed by: page 18 of 60 instructional media + magic, inc. RFC1945 Web Address: HTTP/1.0 http://sunsite.cnlab-switch.ch/ By: Berners-Lee, et al 3.6 Media Types HTTP uses Internet Media Types [13] in the Content-Type header field (Section 10.5) in order to provide open and extensible data typing. media-type = type "/" subtype *( ";" parameter ) type = token subtype = token Parameters may follow the type/subtype in the form of attribute/value pairs. parameter = attribute "=" value attribute = token value = token | quoted-string The type, subtype, and parameter attribute names are case- insensitive. Parameter values may or may not be case-sensitive, depending on the semantics of the parameter name. LWS must not be generated between the type and subtype, nor between an attribute and its value. Upon receipt of a media type with an unrecognized parameter, a user agent should treat the media type as if the unrecognized parameter and its value were not present. Some older HTTP applications do not recognize media type parameters. HTTP/1.0 applications should only use media type parameters when they are necessary to define the content of a message. Media-type values are registered with the Internet Assigned Number Authority (IANA [15]). The media type registration process is outlined in RFC 1590 [13]. Use of non-registered media types is discouraged. 3.6.1 Canonicalization and Text Defaults Internet media types are registered with a canonical form. In general, an Entity-Body transferred via HTTP must be represented in the appropriate canonical form prior to its transmission. If the body has been encoded with a Content-Encoding, the underlying data should be in canonical form prior to being encoded. Media subtypes of the "text" type use CRLF as the text line break when in canonical form. However, HTTP allows the transport of text media with plain CR or LF alone representing a line break when used consistently within the Entity-Body. HTTP applications must accept CRLF, bare CR, and bare LF as being representative of a line break in text media received via HTTP. Formatted and Indexed by: page 19 of 60 instructional media + magic, inc. RFC1945 Web Address: HTTP/1.0 http://sunsite.cnlab-switch.ch/ By: Berners-Lee, et al In addition, if the text media is represented in a character set that does not use octets 13 and 10 for CR and LF respectively, as is the case for some multi-byte character sets, HTTP allows the use of whatever octet sequences are defined by that character set to represent the equivalent of CR and LF for line breaks. This flexibility regarding line breaks applies only to text media in the Entity-Body; a bare CR or LF should not be substituted for CRLF within any of the HTTP control structures (such as header fields and multipart boundaries). The "charset" parameter is used with some media types to define the character set (Section 3.4) of the data. When no explicit charset parameter is provided by the sender, media subtypes of the "text" type are defined to have a default charset value of "ISO-8859-1" when received via HTTP. Data in character sets other than "ISO-8859-1" or its subsets must be labelled with an appropriate charset value in order to be consistently interpreted by the recipient. Note: Many current HTTP servers provide data using charsets other than "ISO-8859-1" without proper labelling. This situation reduces interoperability and is not recommended. To compensate for this, some HTTP user agents provide a configuration option to allow the user to change the default interpretation of the media type character set when no charset parameter is given. 3.6.2 Multipart Types MIME provides for a number of "multipart" types -- encapsulations of several entities within a single message's Entity-Body. The multipart types registered by IANA [15] do not have any special meaning for HTTP/1.0, though user agents may need to understand each type in order to correctly interpret the purpose of each body-part. An HTTP user agent should follow the same or similar behavior as a MIME user agent does upon receipt of a multipart type. HTTP servers should not assume that all HTTP clients are prepared to handle multipart types. All multipart types share a common syntax and must include a boundary parameter as part of the media type value. The message body is itself a protocol element and must therefore use only CRLF to represent line breaks between body-parts. Multipart body-parts may contain HTTP header fields which are significant to the meaning of that part. 3.7 Product Tokens Product tokens are used to allow communicating applications to identify themselves via a simple product token, with an optional slash and version designator. Most fields using product tokens also allow subproducts which form a significant part of the application to Formatted and Indexed by: page 20 of 60 instructional media + magic, inc. RFC1945 Web Address: HTTP/1.0 http://sunsite.cnlab-switch.ch/ By: Berners-Lee, et al be listed, separated by whitespace. By convention, the products are listed in order of their significance for identifying the application. product = token ["/" product-version] product-version = token Examples: User-Agent: CERN-LineMode/2.15 libwww/2.17b3 Server: Apache/0.8.4 Product tokens should be short and to the point -- use of them for advertizing or other non-essential information is explicitly forbidden. Although any token character may appear in a product- version, this token should only be used for a version identifier (i.e., successive versions of the same product should only differ in the product-version portion of the product value). 4. HTTP Message 4.1 Message Types HTTP messages consist of requests from client to server and responses from server to client. HTTP-message = Simple-Request ; HTTP/0.9 messages | Simple-Response | Full-Request ; HTTP/1.0 messages | Full-Response Full-Request and Full-Response use the generic message format of RFC 822 [7] for transferring entities. Both messages may include optional header fields (also known as "headers") and an entity body. The entity body is separated from the headers by a null line (i.e., a line with nothing preceding the CRLF). Full-Request = Request-Line ; Section 5.1 *( General-Header ; Section 4.3 | Request-Header ; Section 5.2 | Entity-Header ) ; Section 7.1 CRLF [ Entity-Body ] ; Section 7.2 Full-Response = Status-Line ; Section 6.1 *( General-Header ; Section 4.3 | Response-Header ; Section 6.2 Formatted and Indexed by: page 21 of 60 instructional media + magic, inc. RFC1945 Web Address: HTTP/1.0 http://sunsite.cnlab-switch.ch/ By: Berners-Lee, et al | Entity-Header ) ; Section 7.1 CRLF [ Entity-Body ] ; Section 7.2 Simple-Request and Simple-Response do not allow the use of any header information and are limited to a single request method (GET). Simple-Request = "GET" SP Request-URI CRLF Simple-Response = [ Entity-Body ] Use of the Simple-Request format is discouraged because it prevents the server from identifying the media type of the returned entity. 4.2 Message Headers HTTP header fields, which include General-Header (Section 4.3), Request-Header (Section 5.2), Response-Header (Section 6.2), and Entity-Header (Section 7.1) fields, follow the same generic format as that given in Section 3.1 of RFC 822 [7]. Each header field consists of a name followed immediately by a colon (":"), a single space (SP) character, and the field value. Field names are case-insensitive. Header fields can be extended over multiple lines by preceding each extra line with at least one SP or HT, though this is not recommended. HTTP-header = field-name ":" [ field-value ] CRLF field-name = token field-value = *( field-content | LWS ) field-content = The order in which header fields are received is not significant. However, it is "good practice" to send General-Header fields first, followed by Request-Header or Response-Header fields prior to the Entity-Header fields. Multiple HTTP-header fields with the same field-name may be present in a message if and only if the entire field-value for that header field is defined as a comma-separated list [i.e., #(values)]. It must be possible to combine the multiple header fields into one "field- name: field-value" pair, without changing the semantics of the message, by appending each subsequent field-value to the first, each separated by a comma. Formatted and Indexed by: page 22 of 60 instructional media + magic, inc. RFC1945 Web Address: HTTP/1.0 http://sunsite.cnlab-switch.ch/ By: Berners-Lee, et al 4.3 General Header Fields There are a few header fields which have general applicability for both request and response messages, but which do not apply to the entity being transferred. These headers apply only to the message being transmitted. General-Header = Date ; Section 10.6 | Pragma ; Section 10.12 General header field names can be extended reliably only in combination with a change in the protocol version. However, new or experimental header fields may be given the semantics of general header fields if all parties in the communication recognize them to be general header fields. Unrecognized header fields are treated as Entity-Header fields. 5. Request A request message from a client to a server includes, within the first line of that message, the method to be applied to the resource, the identifier of the resource, and the protocol version in use. For backwards compatibility with the more limited HTTP/0.9 protocol, there are two valid formats for an HTTP request: Request = Simple-Request | Full-Request Simple-Request = "GET" SP Request-URI CRLF Full-Request = Request-Line ; Section 5.1 *( General-Header ; Section 4.3 | Request-Header ; Section 5.2 | Entity-Header ) ; Section 7.1 CRLF [ Entity-Body ] ; Section 7.2 If an HTTP/1.0 server receives a Simple-Request, it must respond with an HTTP/0.9 Simple-Response. An HTTP/1.0 client capable of receiving a Full-Response should never generate a Simple-Request. 5.1 Request-Line The Request-Line begins with a method token, followed by the Request-URI and the protocol version, and ending with CRLF. The elements are separated by SP characters. No CR or LF are allowed except in the final CRLF sequence. Request-Line = Method SP Request-URI SP HTTP-Version CRLF Formatted and Indexed by: page 23 of 60 instructional media + magic, inc. RFC1945 Web Address: HTTP/1.0 http://sunsite.cnlab-switch.ch/ By: Berners-Lee, et al Note that the difference between a Simple-Request and the Request- Line of a Full-Request is the presence of the HTTP-Version field and the availability of methods other than GET. 5.1.1 Method The Method token indicates the method to be performed on the resource identified by the Request-URI. The method is case-sensitive. Method = "GET" ; Section 8.1 | "HEAD" ; Section 8.2 | "POST" ; Section 8.3 | extension-method extension-method = token The list of methods acceptable by a specific resource can change dynamically; the client is notified through the return code of the response if a method is not allowed on a resource. Servers should return the status code 501 (not implemented) if the method is unrecognized or not implemented. The methods commonly used by HTTP/1.0 applications are fully defined in Section 8. 5.1.2 Request-URI The Request-URI is a Uniform Resource Identifier (Section 3.2) and identifies the resource upon which to apply the request. Request-URI = absoluteURI | abs_path The two options for Request-URI are dependent on the nature of the request. The absoluteURI form is only allowed when the request is being made to a proxy. The proxy is requested to forward the request and return the response. If the request is GET or HEAD and a prior response is cached, the proxy may use the cached message if it passes any restrictions in the Expires header field. Note that the proxy may forward the request on to another proxy or directly to the server specified by the absoluteURI. In order to avoid request loops, a proxy must be able to recognize all of its server names, including any aliases, local variations, and the numeric IP address. An example Request-Line would be: GET http://www.w3.org/pub/WWW/TheProject.html HTTP/1.0 Formatted and Indexed by: page 24 of 60 instructional media + magic, inc. RFC1945 Web Address: HTTP/1.0 http://sunsite.cnlab-switch.ch/ By: Berners-Lee, et al The most common form of Request-URI is that used to identify a resource on an origin server or gateway. In this case, only the absolute path of the URI is transmitted (see Section 3.2.1, abs_path). For example, a client wishing to retrieve the resource above directly from the origin server would create a TCP connection to port 80 of the host "www.w3.org" and send the line: GET /pub/WWW/TheProject.html HTTP/1.0 followed by the remainder of the Full-Request. Note that the absolute path cannot be empty; if none is present in the original URI, it must be given as "/" (the server root). The Request-URI is transmitted as an encoded string, where some characters may be escaped using the "% HEX HEX" encoding defined by RFC 1738 [4]. The origin server must decode the Request-URI in order to properly interpret the request. 5.2 Request Header Fields The request header fields allow the client to pass additional information about the request, and about the client itself, to the server. These fields act as request modifiers, with semantics equivalent to the parameters on a programming language method (procedure) invocation. Request-Header = Authorization ; Section 10.2 | From ; Section 10.8 | If-Modified-Since ; Section 10.9 | Referer ; Section 10.13 | User-Agent ; Section 10.15 Request-Header field names can be extended reliably only in combination with a change in the protocol version. However, new or experimental header fields may be given the semantics of request header fields if all parties in the communication recognize them to be request header fields. Unrecognized header fields are treated as Entity-Header fields. 6. Response After receiving and interpreting a request message, a server responds in the form of an HTTP response message. Response = Simple-Response | Full-Response Simple-Response = [ Entity-Body ] Formatted and Indexed by: page 25 of 60 instructional media + magic, inc. RFC1945 Web Address: HTTP/1.0 http://sunsite.cnlab-switch.ch/ By: Berners-Lee, et al Full-Response = Status-Line ; Section 6.1 *( General-Header ; Section 4.3 | Response-Header ; Section 6.2 | Entity-Header ) ; Section 7.1 CRLF [ Entity-Body ] ; Section 7.2 A Simple-Response should only be sent in response to an HTTP/0.9 Simple-Request or if the server only supports the more limited HTTP/0.9 protocol. If a client sends an HTTP/1.0 Full-Request and receives a response that does not begin with a Status-Line, it should assume that the response is a Simple-Response and parse it accordingly. Note that the Simple-Response consists only of the entity body and is terminated by the server closing the connection. 6.1 Status-Line The first line of a Full-Response message is the Status-Line, consisting of the protocol version followed by a numeric status code and its associated textual phrase, with each element separated by SP characters. No CR or LF is allowed except in the final CRLF sequence. Status-Line = HTTP-Version SP Status-Code SP Reason-Phrase CRLF Since a status line always begins with the protocol version and status code "HTTP/" 1*DIGIT "." 1*DIGIT SP 3DIGIT SP (e.g., "HTTP/1.0 200 "), the presence of that expression is sufficient to differentiate a Full-Response from a Simple-Response. Although the Simple-Response format may allow such an expression to occur at the beginning of an entity body, and thus cause a misinterpretation of the message if it was given in response to a Full-Request, most HTTP/0.9 servers are limited to responses of type "text/html" and therefore would never generate such a response. 6.1.1 Status Code and Reason Phrase The Status-Code element is a 3-digit integer result code of the attempt to understand and satisfy the request. The Reason-Phrase is intended to give a short textual description of the Status-Code. The Status-Code is intended for use by automata and the Reason-Phrase is intended for the human user. The client is not required to examine or display the Reason-Phrase. Formatted and Indexed by: page 26 of 60 instructional media + magic, inc. RFC1945 Web Address: HTTP/1.0 http://sunsite.cnlab-switch.ch/ By: Berners-Lee, et al The first digit of the Status-Code defines the class of response. The last two digits do not have any categorization role. There are 5 values for the first digit: o 1xx: Informational - Not used, but reserved for future use o 2xx: Success - The action was successfully received, understood, and accepted. o 3xx: Redirection - Further action must be taken in order to complete the request o 4xx: Client Error - The request contains bad syntax or cannot be fulfilled o 5xx: Server Error - The server failed to fulfill an apparently valid request The individual values of the numeric status codes defined for HTTP/1.0, and an example set of corresponding Reason-Phrase's, are presented below. The reason phrases listed here are only recommended -- they may be replaced by local equivalents without affecting the protocol. These codes are fully defined in Section 9. Status-Code = "200" ; OK | "201" ; Created | "202" ; Accepted | "204" ; No Content | "301" ; Moved Permanently | "302" ; Moved Temporarily | "304" ; Not Modified | "400" ; Bad Request | "401" ; Unauthorized | "403" ; Forbidden | "404" ; Not Found | "500" ; Internal Server Error | "501" ; Not Implemented | "502" ; Bad Gateway | "503" ; Service Unavailable | extension-code extension-code = 3DIGIT Reason-Phrase = * HTTP status codes are extensible, but the above codes are the only ones generally recognized in current practice. HTTP applications are not required to understand the meaning of all registered status Formatted and Indexed by: page 27 of 60 instructional media + magic, inc. RFC1945 Web Address: HTTP/1.0 http://sunsite.cnlab-switch.ch/ By: Berners-Lee, et al codes, though such understanding is obviously desirable. However, applications must understand the class of any status code, as indicated by the first digit, and treat any unrecognized response as being equivalent to the x00 status code of that class, with the exception that an unrecognized response must not be cached. For example, if an unrecognized status code of 431 is received by the client, it can safely assume that there was something wrong with its request and treat the response as if it had received a 400 status code. In such cases, user agents should present to the user the entity returned with the response, since that entity is likely to include human-readable information which will explain the unusual status. 6.2 Response Header Fields The response header fields allow the server to pass additional information about the response which cannot be placed in the Status- Line. These header fields give information about the server and about further access to the resource identified by the Request-URI. Response-Header = Location ; Section 10.11 | Server ; Section 10.14 | WWW-Authenticate ; Section 10.16 Response-Header field names can be extended reliably only in combination with a change in the protocol version. However, new or experimental header fields may be given the semantics of response header fields if all parties in the communication recognize them to be response header fields. Unrecognized header fields are treated as Entity-Header fields. 7. Entity Full-Request and Full-Response messages may transfer an entity within some requests and responses. An entity consists of Entity-Header fields and (usually) an Entity-Body. In this section, both sender and recipient refer to either the client or the server, depending on who sends and who receives the entity. Formatted and Indexed by: page 28 of 60 instructional media + magic, inc. RFC1945 Web Address: HTTP/1.0 http://sunsite.cnlab-switch.ch/ By: Berners-Lee, et al 7.1 Entity Header Fields Entity-Header fields define optional metainformation about the Entity-Body or, if no body is present, about the resource identified by the request. Entity-Header = Allow ; Section 10.1 | Content-Encoding ; Section 10.3 | Content-Length ; Section 10.4 | Content-Type ; Section 10.5 | Expires ; Section 10.7 | Last-Modified ; Section 10.10 | extension-header extension-header = HTTP-header The extension-header mechanism allows additional Entity-Header fields to be defined without changing the protocol, but these fields cannot be assumed to be recognizable by the recipient. Unrecognized header fields should be ignored by the recipient and forwarded by proxies. 7.2 Entity Body The entity body (if any) sent with an HTTP request or response is in a format and encoding defined by the Entity-Header fields. Entity-Body = *OCTET An entity body is included with a request message only when the request method calls for one. The presence of an entity body in a request is signaled by the inclusion of a Content-Length header field in the request message headers. HTTP/1.0 requests containing an entity body must include a valid Content-Length header field. For response messages, whether or not an entity body is included with a message is dependent on both the request method and the response code. All responses to the HEAD request method must not include a body, even though the presence of entity header fields may lead one to believe they do. All 1xx (informational), 204 (no content), and 304 (not modified) responses must not include a body. All other responses must include an entity body or a Content-Length header field defined with a value of zero (0). 7.2.1 Type When an Entity-Body is included with a message, the data type of that body is determined via the header fields Content-Type and Content- Encoding. These define a two-layer, ordered encoding model: Formatted and Indexed by: page 29 of 60 instructional media + magic, inc. RFC1945 Web Address: HTTP/1.0 http://sunsite.cnlab-switch.ch/ By: Berners-Lee, et al entity-body := Content-Encoding( Content-Type( data ) ) A Content-Type specifies the media type of the underlying data. A Content-Encoding may be used to indicate any additional content coding applied to the type, usually for the purpose of data compression, that is a property of the resource requested. The default for the content encoding is none (i.e., the identity function). Any HTTP/1.0 message containing an entity body should include a Content-Type header field defining the media type of that body. If and only if the media type is not given by a Content-Type header, as is the case for Simple-Response messages, the recipient may attempt to guess the media type via inspection of its content and/or the name extension(s) of the URL used to identify the resource. If the media type remains unknown, the recipient should treat it as type "application/octet-stream". 7.2.2 Length When an Entity-Body is included with a message, the length of that body may be determined in one of two ways. If a Content-Length header field is present, its value in bytes represents the length of the Entity-Body. Otherwise, the body length is determined by the closing of the connection by the server. Closing the connection cannot be used to indicate the end of a request body, since it leaves no possibility for the server to send back a response. Therefore, HTTP/1.0 requests containing an entity body must include a valid Content-Length header field. If a request contains an entity body and Content-Length is not specified, and the server does not recognize or cannot calculate the length from other fields, then the server should send a 400 (bad request) response. Note: Some older servers supply an invalid Content-Length when sending a document that contains server-side includes dynamically inserted into the data stream. It must be emphasized that this will not be tolerated by future versions of HTTP. Unless the client knows that it is receiving a response from a compliant server, it should not depend on the Content-Length value being correct. 8. Method Definitions The set of common methods for HTTP/1.0 is defined below. Although this set can be expanded, additional methods cannot be assumed to share the same semantics for separately extended clients and servers. Formatted and Indexed by: page 30 of 60 instructional media + magic, inc. RFC1945 Web Address: HTTP/1.0 http://sunsite.cnlab-switch.ch/ By: Berners-Lee, et al 8.1 GET The GET method means retrieve whatever information (in the form of an entity) is identified by the Request-URI. If the Request-URI refers to a data-producing process, it is the produced data which shall be returned as the entity in the response and not the source text of the process, unless that text happens to be the output of the process. The semantics of the GET method changes to a "conditional GET" if the request message includes an If-Modified-Since header field. A conditional GET method requests that the identified resource be transferred only if it has been modified since the date given by the If-Modified-Since header, as described in Section 10.9. The conditional GET method is intended to reduce network usage by allowing cached entities to be refreshed without requiring multiple requests or transferring unnecessary data. 8.2 HEAD The HEAD method is identical to GET except that the server must not return any Entity-Body in the response. The metainformation contained in the HTTP headers in response to a HEAD request should be identical to the information sent in response to a GET request. This method can be used for obtaining metainformation about the resource identified by the Request-URI without transferring the Entity-Body itself. This method is often used for testing hypertext links for validity, accessibility, and recent modification. There is no "conditional HEAD" request analogous to the conditional GET. If an If-Modified-Since header field is included with a HEAD request, it should be ignored. 8.3 POST The POST method is used to request that the destination server accept the entity enclosed in the request as a new subordinate of the resource identified by the Request-URI in the Request-Line. POST is designed to allow a uniform method to cover the following functions: o Annotation of existing resources; o Posting a message to a bulletin board, newsgroup, mailing list, or similar group of articles; o Providing a block of data, such as the result of submitting a form [3], to a data-handling process; o Extending a database through an append operation. Formatted and Indexed by: page 31 of 60 instructional media + magic, inc. RFC1945 Web Address: HTTP/1.0 http://sunsite.cnlab-switch.ch/ By: Berners-Lee, et al The actual function performed by the POST method is determined by the server and is usually dependent on the Request-URI. The posted entity is subordinate to that URI in the same way that a file is subordinate to a directory containing it, a news article is subordinate to a newsgroup to which it is posted, or a record is subordinate to a database. A successful POST does not require that the entity be created as a resource on the origin server or made accessible for future reference. That is, the action performed by the POST method might not result in a resource that can be identified by a URI. In this case, either 200 (ok) or 204 (no content) is the appropriate response status, depending on whether or not the response includes an entity that describes the result. If a resource has been created on the origin server, the response should be 201 (created) and contain an entity (preferably of type "text/html") which describes the status of the request and refers to the new resource. A valid Content-Length is required on all HTTP/1.0 POST requests. An HTTP/1.0 server should respond with a 400 (bad request) message if it cannot determine the length of the request message's content. Applications must not cache responses to a POST request because the application has no way of knowing that the server would return an equivalent response on some future request. 9. Status Code Definitions Each Status-Code is described below, including a description of which method(s) it can follow and any metainformation required in the response. 9.1 Informational 1xx This class of status code indicates a provisional response, consisting only of the Status-Line and optional headers, and is terminated by an empty line. HTTP/1.0 does not define any 1xx status codes and they are not a valid response to a HTTP/1.0 request. However, they may be useful for experimental applications which are outside the scope of this specification. 9.2 Successful 2xx This class of status code indicates that the client's request was successfully received, understood, and accepted. Formatted and Indexed by: page 32 of 60 instructional media + magic, inc. RFC1945 Web Address: HTTP/1.0 http://sunsite.cnlab-switch.ch/ By: Berners-Lee, et al 200 OK The request has succeeded. The information returned with the response is dependent on the method used in the request, as follows: GET an entity corresponding to the requested resource is sent in the response; HEAD the response must only contain the header information and no Entity-Body; POST an entity describing or containing the result of the action. 201 Created The request has been fulfilled and resulted in a new resource being created. The newly created resource can be referenced by the URI(s) returned in the entity of the response. The origin server should create the resource before using this Status-Code. If the action cannot be carried out immediately, the server must include in the response body a description of when the resource will be available; otherwise, the server should respond with 202 (accepted). Of the methods defined by this specification, only POST can create a resource. 202 Accepted The request has been accepted for processing, but the processing has not been completed. The request may or may not eventually be acted upon, as it may be disallowed when processing actually takes place. There is no facility for re-sending a status code from an asynchronous operation such as this. The 202 response is intentionally non-committal. Its purpose is to allow a server to accept a request for some other process (perhaps a batch-oriented process that is only run once per day) without requiring that the user agent's connection to the server persist until the process is completed. The entity returned with this response should include an indication of the request's current status and either a pointer to a status monitor or some estimate of when the user can expect the request to be fulfilled. 204 No Content The server has fulfilled the request but there is no new information to send back. If the client is a user agent, it should not change its document view from that which caused the request to Formatted and Indexed by: page 33 of 60 instructional media + magic, inc. RFC1945 Web Address: HTTP/1.0 http://sunsite.cnlab-switch.ch/ By: Berners-Lee, et al be generated. This response is primarily intended to allow input for scripts or other actions to take place without causing a change to the user agent's active document view. The response may include new metainformation in the form of entity headers, which should apply to the document currently in the user agent's active view. 9.3 Redirection 3xx This class of status code indicates that further action needs to be taken by the user agent in order to fulfill the request. The action required may be carried out by the user agent without interaction with the user if and only if the method used in the subsequent request is GET or HEAD. A user agent should never automatically redirect a request more than 5 times, since such redirections usually indicate an infinite loop. 300 Multiple Choices This response code is not directly used by HTTP/1.0 applications, but serves as the default for interpreting the 3xx class of responses. The requested resource is available at one or more locations. Unless it was a HEAD request, the response should include an entity containing a list of resource characteristics and locations from which the user or user agent can choose the one most appropriate. If the server has a preferred choice, it should include the URL in a Location field; user agents may use this field value for automatic redirection. 301 Moved Permanently The requested resource has been assigned a new permanent URL and any future references to this resource should be done using that URL. Clients with link editing capabilities should automatically relink references to the Request-URI to the new reference returned by the server, where possible. The new URL must be given by the Location field in the response. Unless it was a HEAD request, the Entity-Body of the response should contain a short note with a hyperlink to the new URL. If the 301 status code is received in response to a request using the POST method, the user agent must not automatically redirect the request unless it can be confirmed by the user, since this might change the conditions under which the request was issued. Formatted and Indexed by: page 34 of 60 instructional media + magic, inc. RFC1945 Web Address: HTTP/1.0 http://sunsite.cnlab-switch.ch/ By: Berners-Lee, et al Note: When automatically redirecting a POST request after receiving a 301 status code, some existing user agents will erroneously change it into a GET request. 302 Moved Temporarily The requested resource resides temporarily under a different URL. Since the redirection may be altered on occasion, the client should continue to use the Request-URI for future requests. The URL must be given by the Location field in the response. Unless it was a HEAD request, the Entity-Body of the response should contain a short note with a hyperlink to the new URI(s). If the 302 status code is received in response to a request using the POST method, the user agent must not automatically redirect the request unless it can be confirmed by the user, since this might change the conditions under which the request was issued. Note: When automatically redirecting a POST request after receiving a 302 status code, some existing user agents will erroneously change it into a GET request. 304 Not Modified If the client has performed a conditional GET request and access is allowed, but the document has not been modified since the date and time specified in the If-Modified-Since field, the server must respond with this status code and not send an Entity-Body to the client. Header fields contained in the response should only include information which is relevant to cache managers or which may have changed independently of the entity's Last-Modified date. Examples of relevant header fields include: Date, Server, and Expires. A cache should update its cached entity to reflect any new field values given in the 304 response. 9.4 Client Error 4xx The 4xx class of status code is intended for cases in which the client seems to have erred. If the client has not completed the request when a 4xx code is received, it should immediately cease sending data to the server. Except when responding to a HEAD request, the server should include an entity containing an explanation of the error situation, and whether it is a temporary or permanent condition. These status codes are applicable to any request method. Formatted and Indexed by: page 35 of 60 instructional media + magic, inc. RFC1945 Web Address: HTTP/1.0 http://sunsite.cnlab-switch.ch/ By: Berners-Lee, et al Note: If the client is sending data, server implementations on TCP should be careful to ensure that the client acknowledges receipt of the packet(s) containing the response prior to closing the input connection. If the client continues sending data to the server after the close, the server's controller will send a reset packet to the client, which may erase the client's unacknowledged input buffers before they can be read and interpreted by the HTTP application. 400 Bad Request The request could not be understood by the server due to malformed syntax. The client should not repeat the request without modifications. 401 Unauthorized The request requires user authentication. The response must include a WWW-Authenticate header field (Section 10.16) containing a challenge applicable to the requested resource. The client may repeat the request with a suitable Authorization header field (Section 10.2). If the request already included Authorization credentials, then the 401 response indicates that authorization has been refused for those credentials. If the 401 response contains the same challenge as the prior response, and the user agent has already attempted authentication at least once, then the user should be presented the entity that was given in the response, since that entity may include relevant diagnostic information. HTTP access authentication is explained in Section 11. 403 Forbidden The server understood the request, but is refusing to fulfill it. Authorization will not help and the request should not be repeated. If the request method was not HEAD and the server wishes to make public why the request has not been fulfilled, it should describe the reason for the refusal in the entity body. This status code is commonly used when the server does not wish to reveal exactly why the request has been refused, or when no other response is applicable. 404 Not Found The server has not found anything matching the Request-URI. No indication is given of whether the condition is temporary or permanent. If the server does not wish to make this information available to the client, the status code 403 (forbidden) can be used instead. Formatted and Indexed by: page 36 of 60 instructional media + magic, inc. RFC1945 Web Address: HTTP/1.0 http://sunsite.cnlab-switch.ch/ By: Berners-Lee, et al 9.5 Server Error 5xx Response status codes beginning with the digit "5" indicate cases in which the server is aware that it has erred or is incapable of performing the request. If the client has not completed the request when a 5xx code is received, it should immediately cease sending data to the server. Except when responding to a HEAD request, the server should include an entity containing an explanation of the error situation, and whether it is a temporary or permanent condition. These response codes are applicable to any request method and there are no required header fields. 500 Internal Server Error The server encountered an unexpected condition which prevented it from fulfilling the request. 501 Not Implemented The server does not support the functionality required to fulfill the request. This is the appropriate response when the server does not recognize the request method and is not capable of supporting it for any resource. 502 Bad Gateway The server, while acting as a gateway or proxy, received an invalid response from the upstream server it accessed in attempting to fulfill the request. 503 Service Unavailable The server is currently unable to handle the request due to a temporary overloading or maintenance of the server. The implication is that this is a temporary condition which will be alleviated after some delay. Note: The existence of the 503 status code does not imply that a server must use it when becoming overloaded. Some servers may wish to simply refuse the connection. 10. Header Field Definitions This section defines the syntax and semantics of all commonly used HTTP/1.0 header fields. For general and entity header fields, both sender and recipient refer to either the client or the server, depending on who sends and who receives the message. Formatted and Indexed by: page 37 of 60 instructional media + magic, inc. RFC1945 Web Address: HTTP/1.0 http://sunsite.cnlab-switch.ch/ By: Berners-Lee, et al 10.1 Allow The Allow entity-header field lists the set of methods supported by the resource identified by the Request-URI. The purpose of this field is strictly to inform the recipient of valid methods associated with the resource. The Allow header field is not permitted in a request using the POST method, and thus should be ignored if it is received as part of a POST entity. Allow = "Allow" ":" 1#method Example of use: Allow: GET, HEAD This field cannot prevent a client from trying other methods. However, the indications given by the Allow header field value should be followed. The actual set of allowed methods is defined by the origin server at the time of each request. A proxy must not modify the Allow header field even if it does not understand all the methods specified, since the user agent may have other means of communicating with the origin server. The Allow header field does not indicate what methods are implemented by the server. 10.2 Authorization A user agent that wishes to authenticate itself with a server-- usually, but not necessarily, after receiving a 401 response--may do so by including an Authorization request-header field with the request. The Authorization field value consists of credentials containing the authentication information of the user agent for the realm of the resource being requested. Authorization = "Authorization" ":" credentials HTTP access authentication is described in Section 11. If a request is authenticated and a realm specified, the same credentials should be valid for all other requests within this realm. Responses to requests containing an Authorization field are not cachable. Formatted and Indexed by: page 38 of 60 instructional media + magic, inc. RFC1945 Web Address: HTTP/1.0 http://sunsite.cnlab-switch.ch/ By: Berners-Lee, et al 10.3 Content-Encoding The Content-Encoding entity-header field is used as a modifier to the media-type. When present, its value indicates what additional content coding has been applied to the resource, and thus what decoding mechanism must be applied in order to obtain the media-type referenced by the Content-Type header field. The Content-Encoding is primarily used to allow a document to be compressed without losing the identity of its underlying media type. Content-Encoding = "Content-Encoding" ":" content-coding Content codings are defined in Section 3.5. An example of its use is Content-Encoding: x-gzip The Content-Encoding is a characteristic of the resource identified by the Request-URI. Typically, the resource is stored with this encoding and is only decoded before rendering or analogous usage. 10.4 Content-Length The Content-Length entity-header field indicates the size of the Entity-Body, in decimal number of octets, sent to the recipient or, in the case of the HEAD method, the size of the Entity-Body that would have been sent had the request been a GET. Content-Length = "Content-Length" ":" 1*DIGIT An example is Content-Length: 3495 Applications should use this field to indicate the size of the Entity-Body to be transferred, regardless of the media type of the entity. A valid Content-Length field value is required on all HTTP/1.0 request messages containing an entity body. Any Content-Length greater than or equal to zero is a valid value. Section 7.2.2 describes how to determine the length of a response entity body if a Content-Length is not given. Note: The meaning of this field is significantly different from the corresponding definition in MIME, where it is an optional field used within the "message/external-body" content-type. In HTTP, it should be used whenever the entity's length can be determined prior to being transferred. Formatted and Indexed by: page 39 of 60 instructional media + magic, inc. RFC1945 Web Address: HTTP/1.0 http://sunsite.cnlab-switch.ch/ By: Berners-Lee, et al 10.5 Content-Type The Content-Type entity-header field indicates the media type of the Entity-Body sent to the recipient or, in the case of the HEAD method, the media type that would have been sent had the request been a GET. Content-Type = "Content-Type" ":" media-type Media types are defined in Section 3.6. An example of the field is Content-Type: text/html Further discussion of methods for identifying the media type of an entity is provided in Section 7.2.1. 10.6 Date The Date general-header field represents the date and time at which the message was originated, having the same semantics as orig-date in RFC 822. The field value is an HTTP-date, as described in Section 3.3. Date = "Date" ":" HTTP-date An example is Date: Tue, 15 Nov 1994 08:12:31 GMT If a message is received via direct connection with the user agent (in the case of requests) or the origin server (in the case of responses), then the date can be assumed to be the current date at the receiving end. However, since the date--as it is believed by the origin--is important for evaluating cached responses, origin servers should always include a Date header. Clients should only send a Date header field in messages that include an entity body, as in the case of the POST request, and even then it is optional. A received message which does not have a Date header field should be assigned one by the recipient if the message will be cached by that recipient or gatewayed via a protocol which requires a Date. In theory, the date should represent the moment just before the entity is generated. In practice, the date can be generated at any time during the message origination without affecting its semantic value. Note: An earlier version of this document incorrectly specified that this field should contain the creation date of the enclosed Entity-Body. This has been changed to reflect actual (and proper) Formatted and Indexed by: page 40 of 60 instructional media + magic, inc. RFC1945 Web Address: HTTP/1.0 http://sunsite.cnlab-switch.ch/ By: Berners-Lee, et al usage. 10.7 Expires The Expires entity-header field gives the date/time after which the entity should be considered stale. This allows information providers to suggest the volatility of the resource, or a date after which the information may no longer be valid. Applications must not cache this entity beyond the date given. The presence of an Expires field does not imply that the original resource will change or cease to exist at, before, or after that time. However, information providers that know or even suspect that a resource will change by a certain date should include an Expires header with that date. The format is an absolute date and time as defined by HTTP-date in Section 3.3. Expires = "Expires" ":" HTTP-date An example of its use is Expires: Thu, 01 Dec 1994 16:00:00 GMT If the date given is equal to or earlier than the value of the Date header, the recipient must not cache the enclosed entity. If a resource is dynamic by nature, as is the case with many data- producing processes, entities from that resource should be given an appropriate Expires value which reflects that dynamism. The Expires field cannot be used to force a user agent to refresh its display or reload a resource; its semantics apply only to caching mechanisms, and such mechanisms need only check a resource's expiration status when a new request for that resource is initiated. User agents often have history mechanisms, such as "Back" buttons and history lists, which can be used to redisplay an entity retrieved earlier in a session. By default, the Expires field does not apply to history mechanisms. If the entity is still in storage, a history mechanism should display it even if the entity has expired, unless the user has specifically configured the agent to refresh expired history documents. Note: Applications are encouraged to be tolerant of bad or misinformed implementations of the Expires header. A value of zero (0) or an invalid date format should be considered equivalent to an "expires immediately." Although these values are not legitimate for HTTP/1.0, a robust implementation is always desirable. Formatted and Indexed by: page 41 of 60 instructional media + magic, inc. RFC1945 Web Address: HTTP/1.0 http://sunsite.cnlab-switch.ch/ By: Berners-Lee, et al 10.8 From The From request-header field, if given, should contain an Internet e-mail address for the human user who controls the requesting user agent. The address should be machine-usable, as defined by mailbox in RFC 822 [7] (as updated by RFC 1123 [6]): From = "From" ":" mailbox An example is: From: [email protected] This header field may be used for logging purposes and as a means for identifying the source of invalid or unwanted requests. It should not be used as an insecure form of access protection. The interpretation of this field is that the request is being performed on behalf of the person given, who accepts responsibility for the method performed. In particular, robot agents should include this header so that the person responsible for running the robot can be contacted if problems occur on the receiving end. The Internet e-mail address in this field may be separate from the Internet host which issued the request. For example, when a request is passed through a proxy, the original issuer's address should be used. Note: The client should not send the From header field without the user's approval, as it may conflict with the user's privacy interests or their site's security policy. It is strongly recommended that the user be able to disable, enable, and modify the value of this field at any time prior to a request. 10.9 If-Modified-Since The If-Modified-Since request-header field is used with the GET method to make it conditional: if the requested resource has not been modified since the time specified in this field, a copy of the resource will not be returned from the server; instead, a 304 (not modified) response will be returned without any Entity-Body. If-Modified-Since = "If-Modified-Since" ":" HTTP-date An example of the field is: If-Modified-Since: Sat, 29 Oct 1994 19:43:31 GMT Formatted and Indexed by: page 42 of 60 instructional media + magic, inc. RFC1945 Web Address: HTTP/1.0 http://sunsite.cnlab-switch.ch/ By: Berners-Lee, et al A conditional GET method requests that the identified resource be transferred only if it has been modified since the date given by the If-Modified-Since header. The algorithm for determining this includes the following cases: a) If the request would normally result in anything other than a 200 (ok) status, or if the passed If-Modified-Since date is invalid, the response is exactly the same as for a normal GET. A date which is later than the server's current time is invalid. b) If the resource has been modified since the If-Modified-Since date, the response is exactly the same as for a normal GET. c) If the resource has not been modified since a valid If-Modified-Since date, the server shall return a 304 (not modified) response. The purpose of this feature is to allow efficient updates of cached information with a minimum amount of transaction overhead. 10.10 Last-Modified The Last-Modified entity-header field indicates the date and time at which the sender believes the resource was last modified. The exact semantics of this field are defined in terms of how the recipient should interpret it: if the recipient has a copy of this resource which is older than the date given by the Last-Modified field, that copy should be considered stale. Last-Modified = "Last-Modified" ":" HTTP-date An example of its use is Last-Modified: Tue, 15 Nov 1994 12:45:26 GMT The exact meaning of this header field depends on the implementation of the sender and the nature of the original resource. For files, it may be just the file system last-modified time. For entities with dynamically included parts, it may be the most recent of the set of last-modify times for its component parts. For database gateways, it may be the last-update timestamp of the record. For virtual objects, it may be the last time the internal state changed. An origin server must not send a Last-Modified date which is later than the server's time of message origination. In such cases, where the resource's last modification would indicate some time in the Formatted and Indexed by: page 43 of 60 instructional media + magic, inc. RFC1945 Web Address: HTTP/1.0 http://sunsite.cnlab-switch.ch/ By: Berners-Lee, et al future, the server must replace that date with the message origination date. 10.11 Location The Location response-header field defines the exact location of the resource that was identified by the Request-URI. For 3xx responses, the location must indicate the server's preferred URL for automatic redirection to the resource. Only one absolute URL is allowed. Location = "Location" ":" absoluteURI An example is Location: http://www.w3.org/hypertext/WWW/NewLocation.html 10.12 Pragma The Pragma general-header field is used to include implementation- specific directives that may apply to any recipient along the request/response chain. All pragma directives specify optional behavior from the viewpoint of the protocol; however, some systems may require that behavior be consistent with the directives. Pragma = "Pragma" ":" 1#pragma-directive pragma-directive = "no-cache" | extension-pragma extension-pragma = token [ "=" word ] When the "no-cache" directive is present in a request message, an application should forward the request toward the origin server even if it has a cached copy of what is being requested. This allows a client to insist upon receiving an authoritative response to its request. It also allows a client to refresh a cached copy which is known to be corrupted or stale. Pragma directives must be passed through by a proxy or gateway application, regardless of their significance to that application, since the directives may be applicable to all recipients along the request/response chain. It is not possible to specify a pragma for a specific recipient; however, any pragma directive not relevant to a recipient should be ignored by that recipient. 10.13 Referer The Referer request-header field allows the client to specify, for the server's benefit, the address (URI) of the resource from which the Request-URI was obtained. This allows a server to generate lists Formatted and Indexed by: page 44 of 60 instructional media + magic, inc. RFC1945 Web Address: HTTP/1.0 http://sunsite.cnlab-switch.ch/ By: Berners-Lee, et al of back-links to resources for interest, logging, optimized caching, etc. It also allows obsolete or mistyped links to be traced for maintenance. The Referer field must not be sent if the Request-URI was obtained from a source that does not have its own URI, such as input from the user keyboard. Referer = "Referer" ":" ( absoluteURI | relativeURI ) Example: Referer: http://www.w3.org/hypertext/DataSources/Overview.html If a partial URI is given, it should be interpreted relative to the Request-URI. The URI must not include a fragment. Note: Because the source of a link may be private information or may reveal an otherwise private information source, it is strongly recommended that the user be able to select whether or not the Referer field is sent. For example, a browser client could have a toggle switch for browsing openly/anonymously, which would respectively enable/disable the sending of Referer and From information. 10.14 Server The Server response-header field contains information about the software used by the origin server to handle the request. The field can contain multiple product tokens (Section 3.7) and comments identifying the server and any significant subproducts. By convention, the product tokens are listed in order of their significance for identifying the application. Server = "Server" ":" 1*( product | comment ) Example: Server: CERN/3.0 libwww/2.17 If the response is being forwarded through a proxy, the proxy application must not add its data to the product list. Note: Revealing the specific software version of the server may allow the server machine to become more vulnerable to attacks against software that is known to contain security holes. Server implementors are encouraged to make this field a configurable option. Formatted and Indexed by: page 45 of 60 instructional media + magic, inc. RFC1945 Web Address: HTTP/1.0 http://sunsite.cnlab-switch.ch/ By: Berners-Lee, et al Note: Some existing servers fail to restrict themselves to the product token syntax within the Server field. 10.15 User-Agent The User-Agent request-header field contains information about the user agent originating the request. This is for statistical purposes, the tracing of protocol violations, and automated recognition of user agents for the sake of tailoring responses to avoid particular user agent limitations. Although it is not required, user agents should include this field with requests. The field can contain multiple product tokens (Section 3.7) and comments identifying the agent and any subproducts which form a significant part of the user agent. By convention, the product tokens are listed in order of their significance for identifying the application. User-Agent = "User-Agent" ":" 1*( product | comment ) Example: User-Agent: CERN-LineMode/2.15 libwww/2.17b3 Note: Some current proxy applications append their product information to the list in the User-Agent field. This is not recommended, since it makes machine interpretation of these fields ambiguous. Note: Some existing clients fail to restrict themselves to the product token syntax within the User-Agent field. 10.16 WWW-Authenticate The WWW-Authenticate response-header field must be included in 401 (unauthorized) response messages. The field value consists of at least one challenge that indicates the authentication scheme(s) and parameters applicable to the Request-URI. WWW-Authenticate = "WWW-Authenticate" ":" 1#challenge The HTTP access authentication process is described in Section 11. User agents must take special care in parsing the WWW-Authenticate field value if it contains more than one challenge, or if more than one WWW-Authenticate header field is provided, since the contents of a challenge may itself contain a comma-separated list of authentication parameters. Formatted and Indexed by: page 46 of 60 instructional media + magic, inc. RFC1945 Web Address: HTTP/1.0 http://sunsite.cnlab-switch.ch/ By: Berners-Lee, et al 11. Access Authentication HTTP provides a simple challenge-response authentication mechanism which may be used by a server to challenge a client request and by a client to provide authentication information. It uses an extensible, case-insensitive token to identify the authentication scheme, followed by a comma-separated list of attribute-value pairs which carry the parameters necessary for achieving authentication via that scheme. auth-scheme = token auth-param = token "=" quoted-string The 401 (unauthorized) response message is used by an origin server to challenge the authorization of a user agent. This response must include a WWW-Authenticate header field containing at least one challenge applicable to the requested resource. challenge = auth-scheme 1*SP realm *( "," auth-param ) realm = "realm" "=" realm-value realm-value = quoted-string The realm attribute (case-insensitive) is required for all authentication schemes which issue a challenge. The realm value (case-sensitive), in combination with the canonical root URL of the server being accessed, defines the protection space. These realms allow the protected resources on a server to be partitioned into a set of protection spaces, each with its own authentication scheme and/or authorization database. The realm value is a string, generally assigned by the origin server, which may have additional semantics specific to the authentication scheme. A user agent that wishes to authenticate itself with a server-- usually, but not necessarily, after receiving a 401 response--may do so by including an Authorization header field with the request. The Authorization field value consists of credentials containing the authentication information of the user agent for the realm of the resource being requested. credentials = basic-credentials | ( auth-scheme #auth-param ) The domain over which credentials can be automatically applied by a user agent is determined by the protection space. If a prior request has been authorized, the same credentials may be reused for all other requests within that protection space for a period of time determined Formatted and Indexed by: page 47 of 60 instructional media + magic, inc. RFC1945 Web Address: HTTP/1.0 http://sunsite.cnlab-switch.ch/ By: Berners-Lee, et al by the authentication scheme, parameters, and/or user preference. Unless otherwise defined by the authentication scheme, a single protection space cannot extend outside the scope of its server. If the server does not wish to accept the credentials sent with a request, it should return a 403 (forbidden) response. The HTTP protocol does not restrict applications to this simple challenge-response mechanism for access authentication. Additional mechanisms may be used, such as encryption at the transport level or via message encapsulation, and with additional header fields specifying authentication information. However, these additional mechanisms are not defined by this specification. Proxies must be completely transparent regarding user agent authentication. That is, they must forward the WWW-Authenticate and Authorization headers untouched, and must not cache the response to a request containing Authorization. HTTP/1.0 does not provide a means for a client to be authenticated with a proxy. 11.1 Basic Authentication Scheme The "basic" authentication scheme is based on the model that the user agent must authenticate itself with a user-ID and a password for each realm. The realm value should be considered an opaque string which can only be compared for equality with other realms on that server. The server will authorize the request only if it can validate the user-ID and password for the protection space of the Request-URI. There are no optional authentication parameters. Upon receipt of an unauthorized request for a URI within the protection space, the server should respond with a challenge like the following: WWW-Authenticate: Basic realm="WallyWorld" where "WallyWorld" is the string assigned by the server to identify the protection space of the Request-URI. To receive authorization, the client sends the user-ID and password, separated by a single colon (":") character, within a base64 [5] encoded string in the credentials. basic-credentials = "Basic" SP basic-cookie basic-cookie = Formatted and Indexed by: page 48 of 60 instructional media + magic, inc. RFC1945 Web Address: HTTP/1.0 http://sunsite.cnlab-switch.ch/ By: Berners-Lee, et al userid-password = [ token ] ":" *TEXT If the user agent wishes to send the user-ID "Aladdin" and password "open sesame", it would use the following header field: Authorization: Basic QWxhZGRpbjpvcGVuIHNlc2FtZQ== The basic authentication scheme is a non-secure method of filtering unauthorized access to resources on an HTTP server. It is based on the assumption that the connection between the client and the server can be regarded as a trusted carrier. As this is not generally true on an open network, the basic authentication scheme should be used accordingly. In spite of this, clients should implement the scheme in order to communicate with servers that use it. 12. Security Considerations This section is meant to inform application developers, information providers, and users of the security limitations in HTTP/1.0 as described by this document. The discussion does not include definitive solutions to the problems revealed, though it does make some suggestions for reducing security risks. 12.1 Authentication of Clients As mentioned in Section 11.1, the Basic authentication scheme is not a secure method of user authentication, nor does it prevent the Entity-Body from being transmitted in clear text across the physical network used as the carrier. HTTP/1.0 does not prevent additional authentication schemes and encryption mechanisms from being employed to increase security. 12.2 Safe Methods The writers of client software should be aware that the software represents the user in their interactions over the Internet, and should be careful to allow the user to be aware of any actions they may take which may have an unexpected significance to themselves or others. In particular, the convention has been established that the GET and HEAD methods should never have the significance of taking an action other than retrieval. These methods should be considered "safe." This allows user agents to represent other methods, such as POST, in a special way, so that the user is made aware of the fact that a possibly unsafe action is being requested. Formatted and Indexed by: page 49 of 60 instructional media + magic, inc. RFC1945 Web Address: HTTP/1.0 http://sunsite.cnlab-switch.ch/ By: Berners-Lee, et al Naturally, it is not possible to ensure that the server does not generate side-effects as a result of performing a GET request; in fact, some dynamic resources consider that a feature. The important distinction here is that the user did not request the side-effects, so therefore cannot be held accountable for them. 12.3 Abuse of Server Log Information A server is in the position to save personal data about a user's requests which may identify their reading patterns or subjects of interest. This information is clearly confidential in nature and its handling may be constrained by law in certain countries. People using the HTTP protocol to provide data are responsible for ensuring that such material is not distributed without the permission of any individuals that are identifiable by the published results. 12.4 Transfer of Sensitive Information Like any generic data transfer protocol, HTTP cannot regulate the content of the data that is transferred, nor is there any a priori method of determining the sensitivity of any particular piece of information within the context of any given request. Therefore, applications should supply as much control over this information as possible to the provider of that information. Three header fields are worth special mention in this context: Server, Referer and From. Revealing the specific software version of the server may allow the server machine to become more vulnerable to attacks against software that is known to contain security holes. Implementors should make the Server header field a configurable option. The Referer field allows reading patterns to be studied and reverse links drawn. Although it can be very useful, its power can be abused if user details are not separated from the information contained in the Referer. Even when the personal information has been removed, the Referer field may indicate a private document's URI whose publication would be inappropriate. The information sent in the From field might conflict with the user's privacy interests or their site's security policy, and hence it should not be transmitted without the user being able to disable, enable, and modify the contents of the field. The user must be able to set the contents of this field within a user preference or application defaults configuration. We suggest, though do not require, that a convenient toggle interface be provided for the user to enable or disable the sending of From and Referer information. Formatted and Indexed by: page 50 of 60 instructional media + magic, inc. RFC1945 Web Address: HTTP/1.0 http://sunsite.cnlab-switch.ch/ By: Berners-Lee, et al 12.5 Attacks Based On File and Path Names Implementations of HTTP origin servers should be careful to restrict the documents returned by HTTP requests to be only those that were intended by the server administrators. If an HTTP server translates HTTP URIs directly into file system calls, the server must take special care not to serve files that were not intended to be delivered to HTTP clients. For example, Unix, Microsoft Windows, and other operating systems use ".." as a path component to indicate a directory level above the current one. On such a system, an HTTP server must disallow any such construct in the Request-URI if it would otherwise allow access to a resource outside those intended to be accessible via the HTTP server. Similarly, files intended for reference only internally to the server (such as access control files, configuration files, and script code) must be protected from inappropriate retrieval, since they might contain sensitive information. Experience has shown that minor bugs in such HTTP server implementations have turned into security risks. 13. Acknowledgments This specification makes heavy use of the augmented BNF and generic constructs defined by David H. Crocker for RFC 822 [7]. Similarly, it reuses many of the definitions provided by Nathaniel Borenstein and Ned Freed for MIME [5]. We hope that their inclusion in this specification will help reduce past confusion over the relationship between HTTP/1.0 and Internet mail message formats. The HTTP protocol has evolved considerably over the past four years. It has benefited from a large and active developer community--the many people who have participated on the www-talk mailing list--and it is that community which has been most responsible for the success of HTTP and of the World-Wide Web in general. Marc Andreessen, Robert Cailliau, Daniel W. Connolly, Bob Denny, Jean-Francois Groff, Phillip M. Hallam-Baker, Hakon W. Lie, Ari Luotonen, Rob McCool, Lou Montulli, Dave Raggett, Tony Sanders, and Marc VanHeyningen deserve special recognition for their efforts in defining aspects of the protocol for early versions of this specification. Paul Hoffman contributed sections regarding the informational status of this document and Appendices C and D. Formatted and Indexed by: page 51 of 60 instructional media + magic, inc. RFC1945 Web Address: HTTP/1.0 http://sunsite.cnlab-switch.ch/ By: Berners-Lee, et al This document has benefited greatly from the comments of all those participating in the HTTP-WG. In addition to those already mentioned, the following individuals have contributed to this specification: Gary Adams Harald Tveit Alvestrand Keith Ball Brian Behlendorf Paul Burchard Maurizio Codogno Mike Cowlishaw Roman Czyborra Michael A. Dolan John Franks Jim Gettys Marc Hedlund Koen Holtman Alex Hopmann Bob Jernigan Shel Kaphan Martijn Koster Dave Kristol Daniel LaLiberte Paul Leach Albert Lunde John C. Mallery Larry Masinter Mitra Jeffrey Mogul Gavin Nicol Bill Perry Jeffrey Perry Owen Rees Luigi Rizzo David Robinson Marc Salomon Rich Salz Jim Seidman Chuck Shotton Eric W. Sink Simon E. Spero Robert S. Thau Francois Yergeau Mary Ellen Zurko Jean-Philippe Martin-Flatin 14. References [1] Anklesaria, F., McCahill, M., Lindner, P., Johnson, D., Torrey, D., and B. Alberti, "The Internet Gopher Protocol: A Distributed Document Search and Retrieval Protocol", RFC 1436, University of Minnesota, March 1993. [2] Berners-Lee, T., "Universal Resource Identifiers in WWW: A Unifying Syntax for the Expression of Names and Addresses of Objects on the Network as used in the World-Wide Web", RFC 1630, CERN, June 1994. [3] Berners-Lee, T., and D. Connolly, "Hypertext Markup Language - 2.0", RFC 1866, MIT/W3C, November 1995. [4] Berners-Lee, T., Masinter, L., and M. McCahill, "Uniform Resource Locators (URL)", RFC 1738, CERN, Xerox PARC, University of Minnesota, December 1994. Formatted and Indexed by: page 52 of 60 instructional media + magic, inc. RFC1945 Web Address: HTTP/1.0 http://sunsite.cnlab-switch.ch/ By: Berners-Lee, et al [5] Borenstein, N., and N. Freed, "MIME (Multipurpose Internet Mail Extensions) Part One: Mechanisms for Specifying and Describing the Format of Internet Message Bodies", RFC 1521, Bellcore, Innosoft, September 1993. [6] Braden, R., "Requirements for Internet hosts - Application and Support", STD 3, RFC 1123, IETF, October 1989. [7] Crocker, D., "Standard for the Format of ARPA Internet Text Messages", STD 11, RFC 822, UDEL, August 1982. [8] F. Davis, B. Kahle, H. Morris, J. Salem, T. Shen, R. Wang, J. Sui, and M. Grinbaum. "WAIS Interface Protocol Prototype Functional Specification." (v1.5), Thinking Machines Corporation, April 1990. [9] Fielding, R., "Relative Uniform Resource Locators", RFC 1808, UC Irvine, June 1995. [10] Horton, M., and R. Adams, "Standard for interchange of USENET Messages", RFC 1036 (Obsoletes RFC 850), AT&T Bell Laboratories, Center for Seismic Studies, December 1987. [11] Kantor, B., and P. Lapsley, "Network News Transfer Protocol: A Proposed Standard for the Stream-Based Transmission of News", RFC 977, UC San Diego, UC Berkeley, February 1986. [12] Postel, J., "Simple Mail Transfer Protocol." STD 10, RFC 821, USC/ISI, August 1982. [13] Postel, J., "Media Type Registration Procedure." RFC 1590, USC/ISI, March 1994. [14] Postel, J., and J. Reynolds, "File Transfer Protocol (FTP)", STD 9, RFC 959, USC/ISI, October 1985. [15] Reynolds, J., and J. Postel, "Assigned Numbers", STD 2, RFC 1700, USC/ISI, October 1994. [16] Sollins, K., and L. Masinter, "Functional Requirements for Uniform Resource Names", RFC 1737, MIT/LCS, Xerox Corporation, December 1994. [17] US-ASCII. Coded Character Set - 7-Bit American Standard Code for Information Interchange. Standard ANSI X3.4-1986, ANSI, 1986. Formatted and Indexed by: page 53 of 60 instructional media + magic, inc. RFC1945 Web Address: HTTP/1.0 http://sunsite.cnlab-switch.ch/ By: Berners-Lee, et al [18] ISO-8859. International Standard -- Information Processing -- 8-bit Single-Byte Coded Graphic Character Sets -- Part 1: Latin alphabet No. 1, ISO 8859-1:1987. Part 2: Latin alphabet No. 2, ISO 8859-2, 1987. Part 3: Latin alphabet No. 3, ISO 8859-3, 1988. Part 4: Latin alphabet No. 4, ISO 8859-4, 1988. Part 5: Latin/Cyrillic alphabet, ISO 8859-5, 1988. Part 6: Latin/Arabic alphabet, ISO 8859-6, 1987. Part 7: Latin/Greek alphabet, ISO 8859-7, 1987. Part 8: Latin/Hebrew alphabet, ISO 8859-8, 1988. Part 9: Latin alphabet No. 5, ISO 8859-9, 1990. 15. Authors' Addresses Tim Berners-Lee Director, W3 Consortium MIT Laboratory for Computer Science 545 Technology Square Cambridge, MA 02139, U.S.A. Fax: +1 (617) 258 8682 EMail: [email protected] Roy T. Fielding Department of Information and Computer Science University of California Irvine, CA 92717-3425, U.S.A. Fax: +1 (714) 824-4056 EMail: [email protected] Henrik Frystyk Nielsen W3 Consortium MIT Laboratory for Computer Science 545 Technology Square Cambridge, MA 02139, U.S.A. Fax: +1 (617) 258 8682 EMail: [email protected] Formatted and Indexed by: page 54 of 60 instructional media + magic, inc. RFC1945 Web Address: HTTP/1.0 http://sunsite.cnlab-switch.ch/ By: Berners-Lee, et al Appendices These appendices are provided for informational reasons only -- they do not form a part of the HTTP/1.0 specification. A. Internet Media Type message/http In addition to defining the HTTP/1.0 protocol, this document serves as the specification for the Internet media type "message/http". The following is to be registered with IANA [13]. Media Type name: message Media subtype name: http Required parameters: none Optional parameters: version, msgtype version: The HTTP-Version number of the enclosed message (e.g., "1.0"). If not present, the version can be determined from the first line of the body. msgtype: The message type -- "request" or "response". If not present, the type can be determined from the first line of the body. Encoding considerations: only "7bit", "8bit", or "binary" are permitted Security considerations: none B. Tolerant Applications Although this document specifies the requirements for the generation of HTTP/1.0 messages, not all applications will be correct in their implementation. We therefore recommend that operational applications be tolerant of deviations whenever those deviations can be interpreted unambiguously. Clients should be tolerant in parsing the Status-Line and servers tolerant when parsing the Request-Line. In particular, they should accept any amount of SP or HT characters between fields, even though only a single SP is required. The line terminator for HTTP-header fields is the sequence CRLF. However, we recommend that applications, when parsing such headers, recognize a single LF as a line terminator and ignore the leading CR. _ Formatted and Indexed by: page 55 of 60 instructional media + magic, inc. RFC1945 Web Address: HTTP/1.0 http://sunsite.cnlab-switch.ch/ By: Berners-Lee, et al C. Relationship to MIME HTTP/1.0 uses many of the constructs defined for Internet Mail (RFC 822 [7]) and the Multipurpose Internet Mail Extensions (MIME [5]) to allow entities to be transmitted in an open variety of representations and with extensible mechanisms. However, RFC 1521 discusses mail, and HTTP has a few features that are different than those described in RFC 1521. These differences were carefully chosen to optimize performance over binary connections, to allow greater freedom in the use of new media types, to make date comparisons easier, and to acknowledge the practice of some early HTTP servers and clients. At the time of this writing, it is expected that RFC 1521 will be revised. The revisions may include some of the practices found in HTTP/1.0 but not in RFC 1521. This appendix describes specific areas where HTTP differs from RFC 1521. Proxies and gateways to strict MIME environments should be aware of these differences and provide the appropriate conversions where necessary. Proxies and gateways from MIME environments to HTTP also need to be aware of the differences because some conversions may be required. C.1 Conversion to Canonical Form RFC 1521 requires that an Internet mail entity be converted to canonical form prior to being transferred, as described in Appendix G of RFC 1521 [5]. Section 3.6.1 of this document describes the forms allowed for subtypes of the "text" media type when transmitted over HTTP. RFC 1521 requires that content with a Content-Type of "text" represent line breaks as CRLF and forbids the use of CR or LF outside of line break sequences. HTTP allows CRLF, bare CR, and bare LF to indicate a line break within text content when a message is transmitted over HTTP. Where it is possible, a proxy or gateway from HTTP to a strict RFC 1521 environment should translate all line breaks within the text media types described in Section 3.6.1 of this document to the RFC 1521 canonical form of CRLF. Note, however, that this may be complicated by the presence of a Content-Encoding and by the fact that HTTP allows the use of some character sets which do not use octets 13 and 10 to represent CR and LF, as is the case for some multi-byte character sets. Formatted and Indexed by: page 56 of 60 instructional media + magic, inc. RFC1945 Web Address: HTTP/1.0 http://sunsite.cnlab-switch.ch/ By: Berners-Lee, et al C.2 Conversion of Date Formats HTTP/1.0 uses a restricted set of date formats (Section 3.3) to simplify the process of date comparison. Proxies and gateways from other protocols should ensure that any Date header field present in a message conforms to one of the HTTP/1.0 formats and rewrite the date if necessary. C.3 Introduction of Content-Encoding RFC 1521 does not include any concept equivalent to HTTP/1.0's Content-Encoding header field. Since this acts as a modifier on the media type, proxies and gateways from HTTP to MIME-compliant protocols must either change the value of the Content-Type header field or decode the Entity-Body before forwarding the message. (Some experimental applications of Content-Type for Internet mail have used a media-type parameter of ";conversions= C.4 No Content-Transfer-Encoding HTTP does not use the Content-Transfer-Encoding (CTE) field of RFC 1521. Proxies and gateways from MIME-compliant protocols to HTTP must remove any non-identity CTE ("quoted-printable" or "base64") encoding prior to delivering the response message to an HTTP client. Proxies and gateways from HTTP to MIME-compliant protocols are responsible for ensuring that the message is in the correct format and encoding for safe transport on that protocol, where "safe transport" is defined by the limitations of the protocol being used. Such a proxy or gateway should label the data with an appropriate Content-Transfer-Encoding if doing so will improve the likelihood of safe transport over the destination protocol. C.5 HTTP Header Fields in Multipart Body-Parts In RFC 1521, most header fields in multipart body-parts are generally ignored unless the field name begins with "Content-". In HTTP/1.0, multipart body-parts may contain any HTTP header fields which are significant to the meaning of that part. D. Additional Features This appendix documents protocol elements used by some existing HTTP implementations, but not consistently and correctly across most HTTP/1.0 applications. Implementors should be aware of these features, but cannot rely upon their presence in, or interoperability Formatted and Indexed by: page 57 of 60 instructional media + magic, inc. RFC1945 Web Address: HTTP/1.0 http://sunsite.cnlab-switch.ch/ By: Berners-Lee, et al with, other HTTP/1.0 applications. D.1 Additional Request Methods D.1.1 PUT The PUT method requests that the enclosed entity be stored under the supplied Request-URI. If the Request-URI refers to an already existing resource, the enclosed entity should be considered as a modified version of the one residing on the origin server. If the Request-URI does not point to an existing resource, and that URI is capable of being defined as a new resource by the requesting user agent, the origin server can create the resource with that URI. The fundamental difference between the POST and PUT requests is reflected in the different meaning of the Request-URI. The URI in a POST request identifies the resource that will handle the enclosed entity as data to be processed. That resource may be a data-accepting process, a gateway to some other protocol, or a separate entity that accepts annotations. In contrast, the URI in a PUT request identifies the entity enclosed with the request -- the user agent knows what URI is intended and the server should not apply the request to some other resource. D.1.2 DELETE The DELETE method requests that the origin server delete the resource identified by the Request-URI. D.1.3 LINK The LINK method establishes one or more Link relationships between the existing resource identified by the Request-URI and other existing resources. D.1.4 UNLINK The UNLINK method removes one or more Link relationships from the existing resource identified by the Request-URI. D.2 Additional Header Field Definitions D.2.1 Accept The Accept request-header field can be used to indicate a list of media ranges which are acceptable as a response to the request. The asterisk "*" character is used to group media types into ranges, with "*/*" indicating all media types and "type/*" indicating all subtypes Formatted and Indexed by: page 58 of 60 instructional media + magic, inc. RFC1945 Web Address: HTTP/1.0 http://sunsite.cnlab-switch.ch/ By: Berners-Lee, et al of that type. The set of ranges given by the client should represent what types are acceptable given the context of the request. D.2.2 Accept-Charset The Accept-Charset request-header field can be used to indicate a list of preferred character sets other than the default US-ASCII and ISO-8859-1. This field allows clients capable of understanding more comprehensive or special-purpose character sets to signal that capability to a server which is capable of representing documents in those character sets. D.2.3 Accept-Encoding The Accept-Encoding request-header field is similar to Accept, but restricts the content-coding values which are acceptable in the response. D.2.4 Accept-Language The Accept-Language request-header field is similar to Accept, but restricts the set of natural languages that are preferred as a response to the request. D.2.5 Content-Language The Content-Language entity-header field describes the natural language(s) of the intended audience for the enclosed entity. Note that this may not be equivalent to all the languages used within the entity. D.2.6 Link The Link entity-header field provides a means for describing a relationship between the entity and some other resource. An entity may include multiple Link values. Links at the metainformation level typically indicate relationships like hierarchical structure and navigation paths. D.2.7 MIME-Version HTTP messages may include a single MIME-Version general-header field to indicate what version of the MIME protocol was used to construct the message. Use of the MIME-Version header field, as defined by RFC 1521 [5], should indicate that the message is MIME-conformant. Unfortunately, some older HTTP/1.0 servers send it indiscriminately, and thus this field should be ignored. Formatted and Indexed by: page 59 of 60 instructional media + magic, inc. RFC1945 Web Address: HTTP/1.0 http://sunsite.cnlab-switch.ch/ By: Berners-Lee, et al D.2.8 Retry-After The Retry-After response-header field can be used with a 503 (service unavailable) response to indicate how long the service is expected to be unavailable to the requesting client. The value of this field can be either an HTTP-date or an integer number of seconds (in decimal) after the time of the response. D.2.9 Title The Title entity-header field indicates the title of the entity. D.2.10 URI The URI entity-header field may contain some or all of the Uniform Resource Identifiers (Section 3.2) by which the Request-URI resource can be identified. There is no guarantee that the resource can be accessed using the URI(s) specified. Formatted and Indexed by: page 60 of 60 instructional media + magic, inc. RFC2046 Web Address: Media Types http://sunsite.cnlab-switch.ch/ By: Freed & Borenstein Network Working Group N. Freed Request for Comments: 2046 Innosoft Obsoletes: 1521, 1522, 1590 N. Borenstein Category: Standards Track First Virtual November 1996 Multipurpose Internet Mail Extensions (MIME) Part Two: Media Types Status of this Memo This document specifies an Internet standards track protocol for the Internet community, and requests discussion and suggestions for improvements. Please refer to the current edition of the "Internet Official Protocol Standards" (STD 1) for the standardization state and status of this protocol. Distribution of this memo is unlimited. Abstract STD 11, RFC 822 defines a message representation protocol specifying considerable detail about US-ASCII message headers, but which leaves the message content, or message body, as flat US-ASCII text. This set of documents, collectively called the Multipurpose Internet Mail Extensions, or MIME, redefines the format of messages to allow for (1) textual message bodies in character sets other than US-ASCII, (2) an extensible set of different formats for non-textual message bodies, (3) multi-part message bodies, and (4) textual header information in character sets other than US-ASCII. These documents are based on earlier work documented in RFC 934, STD 11, and RFC 1049, but extends and revises them. Because RFC 822 said so little about message bodies, these documents are largely orthogonal to (rather than a revision of) RFC 822. The initial document in this set, RFC 2045, specifies the various headers used to describe the structure of MIME messages. This second document defines the general structure of the MIME media typing system and defines an initial set of media types. The third document, RFC 2047, describes extensions to RFC 822 to allow non-US-ASCII text Formatted and Indexed by: page 1 of 44 instructional media + magic, inc. RFC2046 Web Address: Media Types http://sunsite.cnlab-switch.ch/ By: Freed & Borenstein data in Internet mail header fields. The fourth document, RFC 2048, specifies various IANA registration procedures for MIME-related facilities. The fifth and final document, RFC 2049, describes MIME conformance criteria as well as providing some illustrative examples of MIME message formats, acknowledgements, and the bibliography. These documents are revisions of RFCs 1521 and 1522, which themselves were revisions of RFCs 1341 and 1342. An appendix in RFC 2049 describes differences and changes from previous versions. Table of Contents 1. Introduction ...... 3 2. Definition of a Top-Level Media Type ...... 4 3. Overview Of The Initial Top-Level Media Types ...... 4 4. Discrete Media Type Values ...... 6 4.1 Text Media Type ...... 6 4.1.1 Representation of Line Breaks ...... 7 4.1.2 Charset Parameter ...... 7 4.1.3 Plain Subtype ...... 11 4.1.4 Unrecognized Subtypes ...... 11 4.2 Image Media Type ...... 11 4.3 Audio Media Type ...... 11 4.4 Video Media Type ...... 12 4.5 Application Media Type ...... 12 4.5.1 Octet-Stream Subtype ...... 13 4.5.2 PostScript Subtype ...... 14 4.5.3 Other Application Subtypes ...... 17 5. Composite Media Type Values ...... 17 5.1 Multipart Media Type ...... 17 5.1.1 Common Syntax ...... 19 5.1.2 Handling Nested Messages and Multiparts ...... 24 5.1.3 Mixed Subtype ...... 24 5.1.4 Alternative Subtype ...... 24 5.1.5 Digest Subtype ...... 26 5.1.6 Parallel Subtype ...... 27 5.1.7 Other Multipart Subtypes ...... 28 5.2 Message Media Type ...... 28 5.2.1 RFC822 Subtype ...... 28 5.2.2 Partial Subtype ...... 29 5.2.2.1 Message Fragmentation and Reassembly ...... 30 5.2.2.2 Fragmentation and Reassembly Example ...... 31 5.2.3 External-Body Subtype ...... 33 5.2.4 Other Message Subtypes ...... 40 6. Experimental Media Type Values ...... 40 7. Summary ...... 41 8. Security Considerations ...... 41 9. Authors' Addresses ...... 42 A. Collected Grammar ...... 43 Formatted and Indexed by: page 2 of 44 instructional media + magic, inc. RFC2046 Web Address: Media Types http://sunsite.cnlab-switch.ch/ By: Freed & Borenstein 1. Introduction The first document in this set, RFC 2045, defines a number of header fields, including Content-Type. The Content-Type field is used to specify the nature of the data in the body of a MIME entity, by giving media type and subtype identifiers, and by providing auxiliary information that may be required for certain media types. After the type and subtype names, the remainder of the header field is simply a set of parameters, specified in an attribute/value notation. The ordering of parameters is not significant. In general, the top-level media type is used to declare the general type of data, while the subtype specifies a specific format for that type of data. Thus, a media type of "image/xyz" is enough to tell a user agent that the data is an image, even if the user agent has no knowledge of the specific image format "xyz". Such information can be used, for example, to decide whether or not to show a user the raw data from an unrecognized subtype -- such an action might be reasonable for unrecognized subtypes of "text", but not for unrecognized subtypes of "image" or "audio". For this reason, registered subtypes of "text", "image", "audio", and "video" should not contain embedded information that is really of a different type. Such compound formats should be represented using the "multipart" or "application" types. Parameters are modifiers of the media subtype, and as such do not fundamentally affect the nature of the content. The set of meaningful parameters depends on the media type and subtype. Most parameters are associated with a single specific subtype. However, a given top-level media type may define parameters which are applicable to any subtype of that type. Parameters may be required by their defining media type or subtype or they may be optional. MIME implementations must also ignore any parameters whose names they do not recognize. MIME's Content-Type header field and media type mechanism has been carefully designed to be extensible, and it is expected that the set of media type/subtype pairs and their associated parameters will grow significantly over time. Several other MIME facilities, such as transfer encodings and "message/external-body" access types, are likely to have new values defined over time. In order to ensure that the set of such values is developed in an orderly, well-specified, and public manner, MIME sets up a registration process which uses the Internet Assigned Numbers Authority (IANA) as a central registry for MIME's various areas of extensibility. The registration process for these areas is described in a companion document, RFC 2048. Formatted and Indexed by: page 3 of 44 instructional media + magic, inc. RFC2046 Web Address: Media Types http://sunsite.cnlab-switch.ch/ By: Freed & Borenstein The initial seven standard top-level media type are defined and described in the remainder of this document. 2. Definition of a Top-Level Media Type The definition of a top-level media type consists of: (1) a name and a description of the type, including criteria for whether a particular type would qualify under that type, (2) the names and definitions of parameters, if any, which are defined for all subtypes of that type (including whether such parameters are required or optional), (3) how a user agent and/or gateway should handle unknown subtypes of this type, (4) general considerations on gatewaying entities of this top-level type, if any, and (5) any restrictions on content-transfer-encodings for entities of this top-level type. 3. Overview Of The Initial Top-Level Media Types The five discrete top-level media types are: (1) text -- textual information. The subtype "plain" in particular indicates plain text containing no formatting commands or directives of any sort. Plain text is intended to be displayed "as-is". No special software is required to get the full meaning of the text, aside from support for the indicated character set. Other subtypes are to be used for enriched text in forms where application software may enhance the appearance of the text, but such software must not be required in order to get the general idea of the content. Possible subtypes of "text" thus include any word processor format that can be read without resorting to software that understands the format. In particular, formats that employ embeddded binary formatting information are not considered directly readable. A very simple and portable subtype, "richtext", was defined in RFC 1341, with a further revision in RFC 1896 under the name "enriched". Formatted and Indexed by: page 4 of 44 instructional media + magic, inc. RFC2046 Web Address: Media Types http://sunsite.cnlab-switch.ch/ By: Freed & Borenstein (2) image -- image data. "Image" requires a display device (such as a graphical display, a graphics printer, or a FAX machine) to view the information. An initial subtype is defined for the widely-used image format JPEG. . subtypes are defined for two widely-used image formats, jpeg and gif. (3) audio -- audio data. "Audio" requires an audio output device (such as a speaker or a telephone) to "display" the contents. An initial subtype "basic" is defined in this document. (4) video -- video data. "Video" requires the capability to display moving images, typically including specialized hardware and software. An initial subtype "mpeg" is defined in this document. (5) application -- some other kind of data, typically either uninterpreted binary data or information to be processed by an application. The subtype "octet- stream" is to be used in the case of uninterpreted binary data, in which case the simplest recommended action is to offer to write the information into a file for the user. The "PostScript" subtype is also defined for the transport of PostScript material. Other expected uses for "application" include spreadsheets, data for mail-based scheduling systems, and languages for "active" (computational) messaging, and word processing formats that are not directly readable. Note that security considerations may exist for some types of application data, most notably "application/PostScript" and any form of active messaging. These issues are discussed later in this document. The two composite top-level media types are: (1) multipart -- data consisting of multiple entities of independent data types. Four subtypes are initially defined, including the basic "mixed" subtype specifying a generic mixed set of parts, "alternative" for representing the same data in multiple formats, "parallel" for parts intended to be viewed simultaneously, and "digest" for multipart entities in which each part has a default type of "message/rfc822". Formatted and Indexed by: page 5 of 44 instructional media + magic, inc. RFC2046 Web Address: Media Types http://sunsite.cnlab-switch.ch/ By: Freed & Borenstein (2) message -- an encapsulated message. A body of media type "message" is itself all or a portion of some kind of message object. Such objects may or may not in turn contain other entities. The "rfc822" subtype is used when the encapsulated content is itself an RFC 822 message. The "partial" subtype is defined for partial RFC 822 messages, to permit the fragmented transmission of bodies that are thought to be too large to be passed through transport facilities in one piece. Another subtype, "external-body", is defined for specifying large bodies by reference to an external data source. It should be noted that the list of media type values given here may be augmented in time, via the mechanisms described above, and that the set of subtypes is expected to grow substantially. 4. Discrete Media Type Values Five of the seven initial media type values refer to discrete bodies. The content of these types must be handled by non-MIME mechanisms; they are opaque to MIME processors. 4.1. Text Media Type The "text" media type is intended for sending material which is principally textual in form. A "charset" parameter may be used to indicate the character set of the body text for "text" subtypes, notably including the subtype "text/plain", which is a generic subtype for plain text. Plain text does not provide for or allow formatting commands, font attribute specifications, processing instructions, interpretation directives, or content markup. Plain text is seen simply as a linear sequence of characters, possibly interrupted by line breaks or page breaks. Plain text may allow the stacking of several characters in the same position in the text. Plain text in scripts like Arabic and Hebrew may also include facilitites that allow the arbitrary mixing of text segments with opposite writing directions. Beyond plain text, there are many formats for representing what might be known as "rich text". An interesting characteristic of many such representations is that they are to some extent readable even without the software that interprets them. It is useful, then, to distinguish them, at the highest level, from such unreadable data as images, audio, or text represented in an unreadable form. In the absence of appropriate interpretation software, it is reasonable to show subtypes of "text" to the user, while it is not reasonable to do so with most nontextual data. Such formatted textual data should be represented using subtypes of "text". Formatted and Indexed by: page 6 of 44 instructional media + magic, inc. RFC2046 Web Address: Media Types http://sunsite.cnlab-switch.ch/ By: Freed & Borenstein 4.1.1. Representation of Line Breaks The canonical form of any MIME "text" subtype MUST always represent a line break as a CRLF sequence. Similarly, any occurrence of CRLF in MIME "text" MUST represent a line break. Use of CR and LF outside of line break sequences is also forbidden. This rule applies regardless of format or character set or sets involved. NOTE: The proper interpretation of line breaks when a body is displayed depends on the media type. In particular, while it is appropriate to treat a line break as a transition to a new line when displaying a "text/plain" body, this treatment is actually incorrect for other subtypes of "text" like "text/enriched" [RFC-1896]. Similarly, whether or not line breaks should be added during display operations is also a function of the media type. It should not be necessary to add any line breaks to display "text/plain" correctly, whereas proper display of "text/enriched" requires the appropriate addition of line breaks. NOTE: Some protocols defines a maximum line length. E.g. SMTP [RFC- 821] allows a maximum of 998 octets before the next CRLF sequence. To be transported by such protocols, data which includes too long segments without CRLF sequences must be encoded with a suitable content-transfer-encoding. 4.1.2. Charset Parameter A critical parameter that may be specified in the Content-Type field for "text/plain" data is the character set. This is specified with a "charset" parameter, as in: Content-type: text/plain; charset=iso-8859-1 Unlike some other parameter values, the values of the charset parameter are NOT case sensitive. The default character set, which must be assumed in the absence of a charset parameter, is US-ASCII. The specification for any future subtypes of "text" must specify whether or not they will also utilize a "charset" parameter, and may possibly restrict its values as well. For other subtypes of "text" than "text/plain", the semantics of the "charset" parameter should be defined to be identical to those specified here for "text/plain", i.e., the body consists entirely of characters in the given charset. In particular, definers of future "text" subtypes should pay close attention to the implications of multioctet character sets for their subtype definitions. Formatted and Indexed by: page 7 of 44 instructional media + magic, inc. RFC2046 Web Address: Media Types http://sunsite.cnlab-switch.ch/ By: Freed & Borenstein The charset parameter for subtypes of "text" gives a name of a character set, as "character set" is defined in RFC 2045. The rules regarding line breaks detailed in the previous section must also be observed -- a character set whose definition does not conform to these rules cannot be used in a MIME "text" subtype. An initial list of predefined character set names can be found at the end of this section. Additional character sets may be registered with IANA. Other media types than subtypes of "text" might choose to employ the charset parameter as defined here, but with the CRLF/line break restriction removed. Therefore, all character sets that conform to the general definition of "character set" in RFC 2045 can be registered for MIME use. Note that if the specified character set includes 8-bit characters and such characters are used in the body, a Content-Transfer-Encoding header field and a corresponding encoding on the data are required in order to transmit the body via some mail transfer protocols, such as SMTP [RFC-821]. The default character set, US-ASCII, has been the subject of some confusion and ambiguity in the past. Not only were there some ambiguities in the definition, there have been wide variations in practice. In order to eliminate such ambiguity and variations in the future, it is strongly recommended that new user agents explicitly specify a character set as a media type parameter in the Content-Type header field. "US-ASCII" does not indicate an arbitrary 7-bit character set, but specifies that all octets in the body must be interpreted as characters according to the US-ASCII character set. National and application-oriented versions of ISO 646 [ISO-646] are usually NOT identical to US-ASCII, and in that case their use in Internet mail is explicitly discouraged. The omission of the ISO 646 character set from this document is deliberate in this regard. The character set name of "US-ASCII" explicitly refers to the character set defined in ANSI X3.4-1986 [US- ASCII]. The new international reference version (IRV) of the 1991 edition of ISO 646 is identical to US-ASCII. The character set name "ASCII" is reserved and must not be used for any purpose. NOTE: RFC 821 explicitly specifies "ASCII", and references an earlier version of the American Standard. Insofar as one of the purposes of specifying a media type and character set is to permit the receiver to unambiguously determine how the sender intended the coded message to be interpreted, assuming anything other than "strict ASCII" as the default would risk unintentional and incompatible changes to the semantics of messages now being transmitted. This also implies that Formatted and Indexed by: page 8 of 44 instructional media + magic, inc. RFC2046 Web Address: Media Types http://sunsite.cnlab-switch.ch/ By: Freed & Borenstein messages containing characters coded according to other versions of ISO 646 than US-ASCII and the 1991 IRV, or using code-switching procedures (e.g., those of ISO 2022), as well as 8bit or multiple octet character encodings MUST use an appropriate character set specification to be consistent with MIME. The complete US-ASCII character set is listed in ANSI X3.4- 1986. Note that the control characters including DEL (0-31, 127) have no defined meaning in apart from the combination CRLF (US-ASCII values 13 and 10) indicating a new line. Two of the characters have de facto meanings in wide use: FF (12) often means "start subsequent text on the beginning of a new page"; and TAB or HT (9) often (though not always) means "move the cursor to the next available column after the current position where the column number is a multiple of 8 (counting the first column as column 0)." Aside from these conventions, any use of the control characters or DEL in a body must either occur (1) because a subtype of text other than "plain" specifically assigns some additional meaning, or (2) within the context of a private agreement between the sender and recipient. Such private agreements are discouraged and should be replaced by the other capabilities of this document. NOTE: An enormous proliferation of character sets exist beyond US- ASCII. A large number of partially or totally overlapping character sets is NOT a good thing. A SINGLE character set that can be used universally for representing all of the world's languages in Internet mail would be preferrable. Unfortunately, existing practice in several communities seems to point to the continued use of multiple character sets in the near future. A small number of standard character sets are, therefore, defined for Internet use in this document. The defined charset values are: (1) US-ASCII -- as defined in ANSI X3.4-1986 [US-ASCII]. (2) ISO-8859-X -- where "X" is to be replaced, as necessary, for the parts of ISO-8859 [ISO-8859]. Note that the ISO 646 character sets have deliberately been omitted in favor of their 8859 replacements, which are the designated character sets for Internet mail. As of the publication of this document, the legitimate values for "X" are the digits 1 through 10. Formatted and Indexed by: page 9 of 44 instructional media + magic, inc. RFC2046 Web Address: Media Types http://sunsite.cnlab-switch.ch/ By: Freed & Borenstein Characters in the range 128-159 has no assigned meaning in ISO-8859- X. Characters with values below 128 in ISO-8859-X have the same assigned meaning as they do in US-ASCII. Part 6 of ISO 8859 (Latin/Arabic alphabet) and part 8 (Latin/Hebrew alphabet) includes both characters for which the normal writing direction is right to left and characters for which it is left to right, but do not define a canonical ordering method for representing bi-directional text. The charset values "ISO-8859-6" and "ISO-8859- 8", however, specify that the visual method is used [RFC-1556]. All of these character sets are used as pure 7bit or 8bit sets without any shift or escape functions. The meaning of shift and escape sequences in these character sets is not defined. The character sets specified above are the ones that were relatively uncontroversial during the drafting of MIME. This document does not endorse the use of any particular character set other than US-ASCII, and recognizes that the future evolution of world character sets remains unclear. Note that the character set used, if anything other than US- ASCII, must always be explicitly specified in the Content-Type field. No character set name other than those defined above may be used in Internet mail without the publication of a formal specification and its registration with IANA, or by private agreement, in which case the character set name must begin with "X-". Implementors are discouraged from defining new character sets unless absolutely necessary. The "charset" parameter has been defined primarily for the purpose of textual data, and is described in this section for that reason. However, it is conceivable that non-textual data might also wish to specify a charset value for some purpose, in which case the same syntax and values should be used. In general, composition software should always use the "lowest common denominator" character set possible. For example, if a body contains only US-ASCII characters, it SHOULD be marked as being in the US- ASCII character set, not ISO-8859-1, which, like all the ISO-8859 family of character sets, is a superset of US-ASCII. More generally, if a widely-used character set is a subset of another character set, and a body contains only characters in the widely-used subset, it should be labelled as being in that subset. This will increase the chances that the recipient will be able to view the resulting entity correctly. Formatted and Indexed by: page 10 of 44 instructional media + magic, inc. RFC2046 Web Address: Media Types http://sunsite.cnlab-switch.ch/ By: Freed & Borenstein 4.1.3. Plain Subtype The simplest and most important subtype of "text" is "plain". This indicates plain text that does not contain any formatting commands or directives. Plain text is intended to be displayed "as-is", that is, no interpretation of embedded formatting commands, font attribute specifications, processing instructions, interpretation directives, or content markup should be necessary for proper display. The default media type of "text/plain; charset=us-ascii" for Internet mail describes existing Internet practice. That is, it is the type of body defined by RFC 822. No other "text" subtype is defined by this document. 4.1.4. Unrecognized Subtypes Unrecognized subtypes of "text" should be treated as subtype "plain" as long as the MIME implementation knows how to handle the charset. Unrecognized subtypes which also specify an unrecognized charset should be treated as "application/octet- stream". 4.2. Image Media Type A media type of "image" indicates that the body contains an image. The subtype names the specific image format. These names are not case sensitive. An initial subtype is "jpeg" for the JPEG format using JFIF encoding [JPEG]. The list of "image" subtypes given here is neither exclusive nor exhaustive, and is expected to grow as more types are registered with IANA, as described in RFC 2048. Unrecognized subtypes of "image" should at a miniumum be treated as "application/octet-stream". Implementations may optionally elect to pass subtypes of "image" that they do not specifically recognize to a secure and robust general-purpose image viewing application, if such an application is available. NOTE: Using of a generic-purpose image viewing application this way inherits the security problems of the most dangerous type supported by the application. 4.3. Audio Media Type A media type of "audio" indicates that the body contains audio data. Although there is not yet a consensus on an "ideal" audio format for use with computers, there is a pressing need for a format capable of providing interoperable behavior. Formatted and Indexed by: page 11 of 44 instructional media + magic, inc. RFC2046 Web Address: Media Types http://sunsite.cnlab-switch.ch/ By: Freed & Borenstein The initial subtype of "basic" is specified to meet this requirement by providing an absolutely minimal lowest common denominator audio format. It is expected that richer formats for higher quality and/or lower bandwidth audio will be defined by a later document. The content of the "audio/basic" subtype is single channel audio encoded using 8bit ISDN mu-law [PCM] at a sample rate of 8000 Hz. Unrecognized subtypes of "audio" should at a miniumum be treated as "application/octet-stream". Implementations may optionally elect to pass subtypes of "audio" that they do not specifically recognize to a robust general-purpose audio playing application, if such an application is available. 4.4. Video Media Type A media type of "video" indicates that the body contains a time- varying-picture image, possibly with color and coordinated sound. The term 'video' is used in its most generic sense, rather than with reference to any particular technology or format, and is not meant to preclude subtypes such as animated drawings encoded compactly. The subtype "mpeg" refers to video coded according to the MPEG standard [MPEG]. Note that although in general this document strongly discourages the mixing of multiple media in a single body, it is recognized that many so-called video formats include a representation for synchronized audio, and this is explicitly permitted for subtypes of "video". Unrecognized subtypes of "video" should at a minumum be treated as "application/octet-stream". Implementations may optionally elect to pass subtypes of "video" that they do not specifically recognize to a robust general-purpose video display application, if such an application is available. 4.5. Application Media Type The "application" media type is to be used for discrete data which do not fit in any of the other categories, and particularly for data to be processed by some type of application program. This is information which must be processed by an application before it is viewable or usable by a user. Expected uses for the "application" media type include file transfer, spreadsheets, data for mail-based scheduling systems, and languages for "active" (computational) material. (The latter, in particular, can pose security problems which must be understood by implementors, and are considered in detail in the discussion of the "application/PostScript" media type.) Formatted and Indexed by: page 12 of 44 instructional media + magic, inc. RFC2046 Web Address: Media Types http://sunsite.cnlab-switch.ch/ By: Freed & Borenstein For example, a meeting scheduler might define a standard representation for information about proposed meeting dates. An intelligent user agent would use this information to conduct a dialog with the user, and might then send additional material based on that dialog. More generally, there have been several "active" messaging languages developed in which programs in a suitably specialized language are transported to a remote location and automatically run in the recipient's environment. Such applications may be defined as subtypes of the "application" media type. This document defines two subtypes: octet-stream, and PostScript. The subtype of "application" will often be either the name or include part of the name of the application for which the data are intended. This does not mean, however, that any application program name may be used freely as a subtype of "application". 4.5.1. Octet-Stream Subtype The "octet-stream" subtype is used to indicate that a body contains arbitrary binary data. The set of currently defined parameters is: (1) TYPE -- the general type or category of binary data. This is intended as information for the human recipient rather than for any automatic processing. (2) PADDING -- the number of bits of padding that were appended to the bit-stream comprising the actual contents to produce the enclosed 8bit byte-oriented data. This is useful for enclosing a bit-stream in a body when the total number of bits is not a multiple of 8. Both of these parameters are optional. An additional parameter, "CONVERSIONS", was defined in RFC 1341 but has since been removed. RFC 1341 also defined the use of a "NAME" parameter which gave a suggested file name to be used if the data were to be written to a file. This has been deprecated in anticipation of a separate Content-Disposition header field, to be defined in a subsequent RFC. The recommended action for an implementation that receives an "application/octet-stream" entity is to simply offer to put the data in a file, with any Content-Transfer-Encoding undone, or perhaps to use it as input to a user-specified process. Formatted and Indexed by: page 13 of 44 instructional media + magic, inc. RFC2046 Web Address: Media Types http://sunsite.cnlab-switch.ch/ By: Freed & Borenstein To reduce the danger of transmitting rogue programs, it is strongly recommended that implementations NOT implement a path-search mechanism whereby an arbitrary program named in the Content-Type parameter (e.g., an "interpreter=" parameter) is found and executed using the message body as input. 4.5.2. PostScript Subtype A media type of "application/postscript" indicates a PostScript program. Currently two variants of the PostScript language are allowed; the original level 1 variant is described in [POSTSCRIPT] and the more recent level 2 variant is described in [POSTSCRIPT2]. PostScript is a registered trademark of Adobe Systems, Inc. Use of the MIME media type "application/postscript" implies recognition of that trademark and all the rights it entails. The PostScript language definition provides facilities for internal labelling of the specific language features a given program uses. This labelling, called the PostScript document structuring conventions, or DSC, is very general and provides substantially more information than just the language level. The use of document structuring conventions, while not required, is strongly recommended as an aid to interoperability. Documents which lack proper structuring conventions cannot be tested to see whether or not they will work in a given environment. As such, some systems may assume the worst and refuse to process unstructured documents. The execution of general-purpose PostScript interpreters entails serious security risks, and implementors are discouraged from simply sending PostScript bodies to "off- the-shelf" interpreters. While it is usually safe to send PostScript to a printer, where the potential for harm is greatly constrained by typical printer environments, implementors should consider all of the following before they add interactive display of PostScript bodies to their MIME readers. The remainder of this section outlines some, though probably not all, of the possible problems with the transport of PostScript entities. (1) Dangerous operations in the PostScript language include, but may not be limited to, the PostScript operators "deletefile", "renamefile", "filenameforall", and "file". "File" is only dangerous when applied to something other than standard input or output. Implementations may also define additional nonstandard file operators; these may also pose a threat to security. "Filenameforall", the wildcard file search operator, may appear at first glance to be harmless. Formatted and Indexed by: page 14 of 44 instructional media + magic, inc. RFC2046 Web Address: Media Types http://sunsite.cnlab-switch.ch/ By: Freed & Borenstein Note, however, that this operator has the potential to reveal information about what files the recipient has access to, and this information may itself be sensitive. Message senders should avoid the use of potentially dangerous file operators, since these operators are quite likely to be unavailable in secure PostScript implementations. Message receiving and displaying software should either completely disable all potentially dangerous file operators or take special care not to delegate any special authority to their operation. These operators should be viewed as being done by an outside agency when interpreting PostScript documents. Such disabling and/or checking should be done completely outside of the reach of the PostScript language itself; care should be taken to insure that no method exists for re-enabling full- function versions of these operators. (2) The PostScript language provides facilities for exiting the normal interpreter, or server, loop. Changes made in this "outer" environment are customarily retained across documents, and may in some cases be retained semipermanently in nonvolatile memory. The operators associated with exiting the interpreter loop have the potential to interfere with subsequent document processing. As such, their unrestrained use constitutes a threat of service denial. PostScript operators that exit the interpreter loop include, but may not be limited to, the exitserver and startjob operators. Message sending software should not generate PostScript that depends on exiting the interpreter loop to operate, since the ability to exit will probably be unavailable in secure PostScript implementations. Message receiving and displaying software should completely disable the ability to make retained changes to the PostScript environment by eliminating or disabling the "startjob" and "exitserver" operations. If these operations cannot be eliminated or completely disabled the password associated with them should at least be set to a hard- to-guess value. (3) PostScript provides operators for setting system-wide and device-specific parameters. These parameter settings may be retained across jobs and may potentially pose a threat to the correct operation of the interpreter. The PostScript operators that set system and device parameters include, but may not be Formatted and Indexed by: page 15 of 44 instructional media + magic, inc. RFC2046 Web Address: Media Types http://sunsite.cnlab-switch.ch/ By: Freed & Borenstein limited to, the "setsystemparams" and "setdevparams" operators. Message sending software should not generate PostScript that depends on the setting of system or device parameters to operate correctly. The ability to set these parameters will probably be unavailable in secure PostScript implementations. Message receiving and displaying software should disable the ability to change system and device parameters. If these operators cannot be completely disabled the password associated with them should at least be set to a hard-to-guess value. (4) Some PostScript implementations provide nonstandard facilities for the direct loading and execution of machine code. Such facilities are quite obviously open to substantial abuse. Message sending software should not make use of such features. Besides being totally hardware-specific, they are also likely to be unavailable in secure implementations of PostScript. Message receiving and displaying software should not allow such operators to be used if they exist. (5) PostScript is an extensible language, and many, if not most, implementations of it provide a number of their own extensions. This document does not deal with such extensions explicitly since they constitute an unknown factor. Message sending software should not make use of nonstandard extensions; they are likely to be missing from some implementations. Message receiving and displaying software should make sure that any nonstandard PostScript operators are secure and don't present any kind of threat. (6) It is possible to write PostScript that consumes huge amounts of various system resources. It is also possible to write PostScript programs that loop indefinitely. Both types of programs have the potential to cause damage if sent to unsuspecting recipients. Message-sending software should avoid the construction and dissemination of such programs, which is antisocial. Message receiving and displaying software should provide appropriate mechanisms to abort processing after a reasonable amount of time has elapsed. In addition, PostScript interpreters should be limited to the consumption of only a reasonable amount of any given system resource. Formatted and Indexed by: page 16 of 44 instructional media + magic, inc. RFC2046 Web Address: Media Types http://sunsite.cnlab-switch.ch/ By: Freed & Borenstein (7) It is possible to include raw binary information inside PostScript in various forms. This is not recommended for use in Internet mail, both because it is not supported by all PostScript interpreters and because it significantly complicates the use of a MIME Content- Transfer-Encoding. (Without such binary, PostScript may typically be viewed as line-oriented data. The treatment of CRLF sequences becomes extremely problematic if binary and line-oriented data are mixed in a single Postscript data stream.) (8) Finally, bugs may exist in some PostScript interpreters which could possibly be exploited to gain unauthorized access to a recipient's system. Apart from noting this possibility, there is no specific action to take to prevent this, apart from the timely correction of such bugs if any are found. 4.5.3. Other Application Subtypes It is expected that many other subtypes of "application" will be defined in the future. MIME implementations must at a minimum treat any unrecognized subtypes as being equivalent to "application/octet- stream". 5. Composite Media Type Values The remaining two of the seven initial Content-Type values refer to composite entities. Composite entities are handled using MIME mechanisms -- a MIME processor typically handles the body directly. 5.1. Multipart Media Type In the case of multipart entities, in which one or more different sets of data are combined in a single body, a "multipart" media type field must appear in the entity's header. The body must then contain one or more body parts, each preceded by a boundary delimiter line, and the last one followed by a closing boundary delimiter line. After its boundary delimiter line, each body part then consists of a header area, a blank line, and a body area. Thus a body part is similar to an RFC 822 message in syntax, but different in meaning. A body part is an entity and hence is NOT to be interpreted as actually being an RFC 822 message. To begin with, NO header fields are actually required in body parts. A body part that starts with a blank line, therefore, is allowed and is a body part for which all default values are to be assumed. In such a case, the absence of a Content-Type header usually indicates that the corresponding body has Formatted and Indexed by: page 17 of 44 instructional media + magic, inc. RFC2046 Web Address: Media Types http://sunsite.cnlab-switch.ch/ By: Freed & Borenstein a content-type of "text/plain; charset=US-ASCII". The only header fields that have defined meaning for body parts are those the names of which begin with "Content-". All other header fields may be ignored in body parts. Although they should generally be retained if at all possible, they may be discarded by gateways if necessary. Such other fields are permitted to appear in body parts but must not be depended on. "X-" fields may be created for experimental or private purposes, with the recognition that the information they contain may be lost at some gateways. NOTE: The distinction between an RFC 822 message and a body part is subtle, but important. A gateway between Internet and X.400 mail, for example, must be able to tell the difference between a body part that contains an image and a body part that contains an encapsulated message, the body of which is a JPEG image. In order to represent the latter, the body part must have "Content-Type: message/rfc822", and its body (after the blank line) must be the encapsulated message, with its own "Content-Type: image/jpeg" header field. The use of similar syntax facilitates the conversion of messages to body parts, and vice versa, but the distinction between the two must be understood by implementors. (For the special case in which parts actually are messages, a "digest" subtype is also defined.) As stated previously, each body part is preceded by a boundary delimiter line that contains the boundary delimiter. The boundary delimiter MUST NOT appear inside any of the encapsulated parts, on a line by itself or as the prefix of any line. This implies that it is crucial that the composing agent be able to choose and specify a unique boundary parameter value that does not contain the boundary parameter value of an enclosing multipart as a prefix. All present and future subtypes of the "multipart" type must use an identical syntax. Subtypes may differ in their semantics, and may impose additional restrictions on syntax, but must conform to the required syntax for the "multipart" type. This requirement ensures that all conformant user agents will at least be able to recognize and separate the parts of any multipart entity, even those of an unrecognized subtype. As stated in the definition of the Content-Transfer-Encoding field [RFC 2045], no encoding other than "7bit", "8bit", or "binary" is permitted for entities of type "multipart". The "multipart" boundary delimiters and header fields are always represented as 7bit US-ASCII in any case (though the header fields may encode non-US-ASCII header text as per RFC 2047) and data within the body parts can be encoded on a part-by-part basis, with Content-Transfer-Encoding fields for each appropriate body part. Formatted and Indexed by: page 18 of 44 instructional media + magic, inc. RFC2046 Web Address: Media Types http://sunsite.cnlab-switch.ch/ By: Freed & Borenstein 5.1.1. Common Syntax This section defines a common syntax for subtypes of "multipart". All subtypes of "multipart" must use this syntax. A simple example of a multipart message also appears in this section. An example of a more complex multipart message is given in RFC 2049. The Content-Type field for multipart entities requires one parameter, "boundary". The boundary delimiter line is then defined as a line consisting entirely of two hyphen characters ("-", decimal value 45) followed by the boundary parameter value from the Content-Type header field, optional linear whitespace, and a terminating CRLF. NOTE: The hyphens are for rough compatibility with the earlier RFC 934 method of message encapsulation, and for ease of searching for the boundaries in some implementations. However, it should be noted that multipart messages are NOT completely compatible with RFC 934 encapsulations; in particular, they do not obey RFC 934 quoting conventions for embedded lines that begin with hyphens. This mechanism was chosen over the RFC 934 mechanism because the latter causes lines to grow with each level of quoting. The combination of this growth with the fact that SMTP implementations sometimes wrap long lines made the RFC 934 mechanism unsuitable for use in the event that deeply-nested multipart structuring is ever desired. WARNING TO IMPLEMENTORS: The grammar for parameters on the Content- type field is such that it is often necessary to enclose the boundary parameter values in quotes on the Content-type line. This is not always necessary, but never hurts. Implementors should be sure to study the grammar carefully in order to avoid producing invalid Content-type fields. Thus, a typical "multipart" Content-Type header field might look like this: Content-Type: multipart/mixed; boundary=gc0p4Jq0M2Yt08j34c0p But the following is not valid: Content-Type: multipart/mixed; boundary=gc0pJq0M:08jU534c0p (because of the colon) and must instead be represented as Content-Type: multipart/mixed; boundary="gc0pJq0M:08jU534c0p" This Content-Type value indicates that the content consists of one or more parts, each with a structure that is syntactically identical to an RFC 822 message, except that the header area is allowed to be completely empty, and that the parts are each preceded by the line Formatted and Indexed by: page 19 of 44 instructional media + magic, inc. RFC2046 Web Address: Media Types http://sunsite.cnlab-switch.ch/ By: Freed & Borenstein --gc0pJq0M:08jU534c0p The boundary delimiter MUST occur at the beginning of a line, i.e., following a CRLF, and the initial CRLF is considered to be attached to the boundary delimiter line rather than part of the preceding part. The boundary may be followed by zero or more characters of linear whitespace. It is then terminated by either another CRLF and the header fields for the next part, or by two CRLFs, in which case there are no header fields for the next part. If no Content-Type field is present it is assumed to be "message/rfc822" in a "multipart/digest" and "text/plain" otherwise. NOTE: The CRLF preceding the boundary delimiter line is conceptually attached to the boundary so that it is possible to have a part that does not end with a CRLF (line break). Body parts that must be considered to end with line breaks, therefore, must have two CRLFs preceding the boundary delimiter line, the first of which is part of the preceding body part, and the second of which is part of the encapsulation boundary. Boundary delimiters must not appear within the encapsulated material, and must be no longer than 70 characters, not counting the two leading hyphens. The boundary delimiter line following the last body part is a distinguished delimiter that indicates that no further body parts will follow. Such a delimiter line is identical to the previous delimiter lines, with the addition of two more hyphens after the boundary parameter value. --gc0pJq0M:08jU534c0p-- NOTE TO IMPLEMENTORS: Boundary string comparisons must compare the boundary value with the beginning of each candidate line. An exact match of the entire candidate line is not required; it is sufficient that the boundary appear in its entirety following the CRLF. There appears to be room for additional information prior to the first boundary delimiter line and following the final boundary delimiter line. These areas should generally be left blank, and implementations must ignore anything that appears before the first boundary delimiter line or after the last one. NOTE: These "preamble" and "epilogue" areas are generally not used because of the lack of proper typing of these parts and the lack of clear semantics for handling these areas at gateways, particularly X.400 gateways. However, rather than leaving the preamble area blank, many MIME implementations have found this to be a convenient Formatted and Indexed by: page 20 of 44 instructional media + magic, inc. RFC2046 Web Address: Media Types http://sunsite.cnlab-switch.ch/ By: Freed & Borenstein place to insert an explanatory note for recipients who read the message with pre-MIME software, since such notes will be ignored by MIME-compliant software. NOTE: Because boundary delimiters must not appear in the body parts being encapsulated, a user agent must exercise care to choose a unique boundary parameter value. The boundary parameter value in the example above could have been the result of an algorithm designed to produce boundary delimiters with a very low probability of already existing in the data to be encapsulated without having to prescan the data. Alternate algorithms might result in more "readable" boundary delimiters for a recipient with an old user agent, but would require more attention to the possibility that the boundary delimiter might appear at the beginning of some line in the encapsulated part. The simplest boundary delimiter line possible is something like "---", with a closing boundary delimiter line of "-----". As a very simple example, the following multipart message has two parts, both of them plain text, one of them explicitly typed and one of them implicitly typed: From: Nathaniel Borenstein This is the preamble. It is to be ignored, though it is a handy place for composition agents to include an explanatory note to non-MIME conformant readers. --simple boundary This is implicitly typed plain US-ASCII text. It does NOT end with a linebreak. --simple boundary Content-type: text/plain; charset=us-ascii This is explicitly typed plain US-ASCII text. It DOES end with a linebreak. --simple boundary-- This is the epilogue. It is also to be ignored. Formatted and Indexed by: page 21 of 44 instructional media + magic, inc. RFC2046 Web Address: Media Types http://sunsite.cnlab-switch.ch/ By: Freed & Borenstein The use of a media type of "multipart" in a body part within another "multipart" entity is explicitly allowed. In such cases, for obvious reasons, care must be taken to ensure that each nested "multipart" entity uses a different boundary delimiter. See RFC 2049 for an example of nested "multipart" entities. The use of the "multipart" media type with only a single body part may be useful in certain contexts, and is explicitly permitted. NOTE: Experience has shown that a "multipart" media type with a single body part is useful for sending non-text media types. It has the advantage of providing the preamble as a place to include decoding instructions. In addition, a number of SMTP gateways move or remove the MIME headers, and a clever MIME decoder can take a good guess at multipart boundaries even in the absence of the Content-Type header and thereby successfully decode the message. The only mandatory global parameter for the "multipart" media type is the boundary parameter, which consists of 1 to 70 characters from a set of characters known to be very robust through mail gateways, and NOT ending with white space. (If a boundary delimiter line appears to end with white space, the white space must be presumed to have been added by a gateway, and must be deleted.) It is formally specified by the following BNF: boundary := 0*69 bchars := bcharsnospace / " " bcharsnospace := DIGIT / ALPHA / "'" / "(" / ")" / "+" / "_" / "," / "-" / "." / "/" / ":" / "=" / "?" Overall, the body of a "multipart" entity may be specified as follows: dash-boundary := "--" boundary ; boundary taken from the value of ; boundary parameter of the ; Content-Type field. multipart-body := [preamble CRLF] dash-boundary transport-padding CRLF body-part *encapsulation close-delimiter transport-padding [CRLF epilogue] Formatted and Indexed by: page 22 of 44 instructional media + magic, inc. RFC2046 Web Address: Media Types http://sunsite.cnlab-switch.ch/ By: Freed & Borenstein transport-padding := *LWSP-char ; Composers MUST NOT generate ; non-zero length transport ; padding, but receivers MUST ; be able to handle padding ; added by message transports. encapsulation := delimiter transport-padding CRLF body-part delimiter := CRLF dash-boundary close-delimiter := delimiter "--" preamble := discard-text epilogue := discard-text discard-text := *(*text CRLF) *text ; May be ignored or discarded. body-part := MIME-part-headers [CRLF *OCTET] ; Lines in a body-part must not start ; with the specified dash-boundary and ; the delimiter must not appear anywhere ; in the body part. Note that the ; semantics of a body-part differ from ; the semantics of a message, as ; described in the text. OCTET := IMPORTANT: The free insertion of linear-white-space and RFC 822 comments between the elements shown in this BNF is NOT allowed since this BNF does not specify a structured header field. NOTE: In certain transport enclaves, RFC 822 restrictions such as the one that limits bodies to printable US-ASCII characters may not be in force. (That is, the transport domains may exist that resemble standard Internet mail transport as specified in RFC 821 and assumed by RFC 822, but without certain restrictions.) The relaxation of these restrictions should be construed as locally extending the definition of bodies, for example to include octets outside of the US-ASCII range, as long as these extensions are supported by the transport and adequately documented in the Content- Transfer-Encoding header field. However, in no event are headers (either message headers or body part headers) allowed to contain anything other than US-ASCII characters. Formatted and Indexed by: page 23 of 44 instructional media + magic, inc. RFC2046 Web Address: Media Types http://sunsite.cnlab-switch.ch/ By: Freed & Borenstein NOTE: Conspicuously missing from the "multipart" type is a notion of structured, related body parts. It is recommended that those wishing to provide more structured or integrated multipart messaging facilities should define subtypes of multipart that are syntactically identical but define relationships between the various parts. For example, subtypes of multipart could be defined that include a distinguished part which in turn is used to specify the relationships between the other parts, probably referring to them by their Content-ID field. Old implementations will not recognize the new subtype if this approach is used, but will treat it as multipart/mixed and will thus be able to show the user the parts that are recognized. 5.1.2. Handling Nested Messages and Multiparts The "message/rfc822" subtype defined in a subsequent section of this document has no terminating condition other than running out of data. Similarly, an improperly truncated "multipart" entity may not have any terminating boundary marker, and can turn up operationally due to mail system malfunctions. It is essential that such entities be handled correctly when they are themselves imbedded inside of another "multipart" structure. MIME implementations are therefore required to recognize outer level boundary markers at ANY level of inner nesting. It is not sufficient to only check for the next expected marker or other terminating condition. 5.1.3. Mixed Subtype The "mixed" subtype of "multipart" is intended for use when the body parts are independent and need to be bundled in a particular order. Any "multipart" subtypes that an implementation does not recognize must be treated as being of subtype "mixed". 5.1.4. Alternative Subtype The "multipart/alternative" type is syntactically identical to "multipart/mixed", but the semantics are different. In particular, each of the body parts is an "alternative" version of the same information. Systems should recognize that the content of the various parts are interchangeable. Systems should choose the "best" type based on the local environment and references, in some cases even through user interaction. As with "multipart/mixed", the order of body parts is significant. In this case, the alternatives appear in an order of increasing faithfulness to the original content. In general, the Formatted and Indexed by: page 24 of 44 instructional media + magic, inc. RFC2046 Web Address: Media Types http://sunsite.cnlab-switch.ch/ By: Freed & Borenstein best choice is the LAST part of a type supported by the recipient system's local environment. "Multipart/alternative" may be used, for example, to send a message in a fancy text format in such a way that it can easily be displayed anywhere: From: Nathaniel Borenstein --boundary42 Content-Type: text/plain; charset=us-ascii ... plain text version of message goes here ... --boundary42 Content-Type: text/enriched ... RFC 1896 text/enriched version of same message goes here ... --boundary42 Content-Type: application/x-whatever ... fanciest version of same message goes here ... --boundary42-- In this example, users whose mail systems understood the "application/x-whatever" format would see only the fancy version, while other users would see only the enriched or plain text version, depending on the capabilities of their system. In general, user agents that compose "multipart/alternative" entities must place the body parts in increasing order of preference, that is, with the preferred format last. For fancy text, the sending user agent should put the plainest format first and the richest format last. Receiving user agents should pick and display the last format they are capable of displaying. In the case where one of the alternatives is itself of type "multipart" and contains unrecognized sub-parts, the user agent may choose either to show that alternative, an earlier alternative, or both. Formatted and Indexed by: page 25 of 44 instructional media + magic, inc. RFC2046 Web Address: Media Types http://sunsite.cnlab-switch.ch/ By: Freed & Borenstein NOTE: From an implementor's perspective, it might seem more sensible to reverse this ordering, and have the plainest alternative last. However, placing the plainest alternative first is the friendliest possible option when "multipart/alternative" entities are viewed using a non-MIME-conformant viewer. While this approach does impose some burden on conformant MIME viewers, interoperability with older mail readers was deemed to be more important in this case. It may be the case that some user agents, if they can recognize more than one of the formats, will prefer to offer the user the choice of which format to view. This makes sense, for example, if a message includes both a nicely- formatted image version and an easily-edited text version. What is most critical, however, is that the user not automatically be shown multiple versions of the same data. Either the user should be shown the last recognized version or should be given the choice. THE SEMANTICS OF CONTENT-ID IN MULTIPART/ALTERNATIVE: Each part of a "multipart/alternative" entity represents the same data, but the mappings between the two are not necessarily without information loss. For example, information is lost when translating ODA to PostScript or plain text. It is recommended that each part should have a different Content-ID value in the case where the information content of the two parts is not identical. And when the information content is identical -- for example, where several parts of type "message/external-body" specify alternate ways to access the identical data -- the same Content-ID field value should be used, to optimize any caching mechanisms that might be present on the recipient's end. However, the Content-ID values used by the parts should NOT be the same Content-ID value that describes the "multipart/alternative" as a whole, if there is any such Content-ID field. That is, one Content-ID value will refer to the "multipart/alternative" entity, while one or more other Content-ID values will refer to the parts inside it. 5.1.5. Digest Subtype This document defines a "digest" subtype of the "multipart" Content- Type. This type is syntactically identical to "multipart/mixed", but the semantics are different. In particular, in a digest, the default Content-Type value for a body part is changed from "text/plain" to "message/rfc822". This is done to allow a more readable digest format that is largely compatible (except for the quoting convention) with RFC 934. Note: Though it is possible to specify a Content-Type value for a body part in a digest which is other than "message/rfc822", such as a "text/plain" part containing a description of the material in the Formatted and Indexed by: page 26 of 44 instructional media + magic, inc. RFC2046 Web Address: Media Types http://sunsite.cnlab-switch.ch/ By: Freed & Borenstein digest, actually doing so is undesireble. The "multipart/digest" Content-Type is intended to be used to send collections of messages. If a "text/plain" part is needed, it should be included as a seperate part of a "multipart/mixed" message. A digest in this format might, then, look something like this: From: Moderator-Address To: Recipient-List Date: Mon, 22 Mar 1994 13:34:51 +0000 Subject: Internet Digest, volume 42 MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="---- main boundary ----" ------main boundary ---- ...Introductory text or table of contents... ------main boundary ---- Content-Type: multipart/digest; boundary="---- next message ----" ------next message ---- From: someone-else Date: Fri, 26 Mar 1993 11:13:32 +0200 Subject: my opinion ...body goes here ... ------next message ---- From: someone-else-again Date: Fri, 26 Mar 1993 10:07:13 -0500 Subject: my different opinion ... another body goes here ... ------next message ------ ------main boundary ------ 5.1.6. Parallel Subtype This document defines a "parallel" subtype of the "multipart" Content-Type. This type is syntactically identical to "multipart/mixed", but the semantics are different. In particular, Formatted and Indexed by: page 27 of 44 instructional media + magic, inc. RFC2046 Web Address: Media Types http://sunsite.cnlab-switch.ch/ By: Freed & Borenstein in a parallel entity, the order of body parts is not significant. A common presentation of this type is to display all of the parts simultaneously on hardware and software that are capable of doing so. However, composing agents should be aware that many mail readers will lack this capability and will show the parts serially in any event. 5.1.7. Other Multipart Subtypes Other "multipart" subtypes are expected in the future. MIME implementations must in general treat unrecognized subtypes of "multipart" as being equivalent to "multipart/mixed". 5.2. Message Media Type It is frequently desirable, in sending mail, to encapsulate another mail message. A special media type, "message", is defined to facilitate this. In particular, the "rfc822" subtype of "message" is used to encapsulate RFC 822 messages. NOTE: It has been suggested that subtypes of "message" might be defined for forwarded or rejected messages. However, forwarded and rejected messages can be handled as multipart messages in which the first part contains any control or descriptive information, and a second part, of type "message/rfc822", is the forwarded or rejected message. Composing rejection and forwarding messages in this manner will preserve the type information on the original message and allow it to be correctly presented to the recipient, and hence is strongly encouraged. Subtypes of "message" often impose restrictions on what encodings are allowed. These restrictions are described in conjunction with each specific subtype. Mail gateways, relays, and other mail handling agents are commonly known to alter the top-level header of an RFC 822 message. In particular, they frequently add, remove, or reorder header fields. These operations are explicitly forbidden for the encapsulated headers embedded in the bodies of messages of type "message." 5.2.1. RFC822 Subtype A media type of "message/rfc822" indicates that the body contains an encapsulated message, with the syntax of an RFC 822 message. However, unlike top-level RFC 822 messages, the restriction that each "message/rfc822" body must include a "From", "Date", and at least one destination header is removed and replaced with the requirement that at least one of "From", "Subject", or "Date" must be present. Formatted and Indexed by: page 28 of 44 instructional media + magic, inc. RFC2046 Web Address: Media Types http://sunsite.cnlab-switch.ch/ By: Freed & Borenstein It should be noted that, despite the use of the numbers "822", a "message/rfc822" entity isn't restricted to material in strict conformance to RFC822, nor are the semantics of "message/rfc822" objects restricted to the semantics defined in RFC822. More specifically, a "message/rfc822" message could well be a News article or a MIME message. No encoding other than "7bit", "8bit", or "binary" is permitted for the body of a "message/rfc822" entity. The message header fields are always US-ASCII in any case, and data within the body can still be encoded, in which case the Content-Transfer-Encoding header field in the encapsulated message will reflect this. Non-US-ASCII text in the headers of an encapsulated message can be specified using the mechanisms described in RFC 2047. 5.2.2. Partial Subtype The "partial" subtype is defined to allow large entities to be delivered as several separate pieces of mail and automatically reassembled by a receiving user agent. (The concept is similar to IP fragmentation and reassembly in the basic Internet Protocols.) This mechanism can be used when intermediate transport agents limit the size of individual messages that can be sent. The media type "message/partial" thus indicates that the body contains a fragment of a larger entity. Because data of type "message" may never be encoded in base64 or quoted-printable, a problem might arise if "message/partial" entities are constructed in an environment that supports binary or 8bit transport. The problem is that the binary data would be split into multiple "message/partial" messages, each of them requiring binary transport. If such messages were encountered at a gateway into a 7bit transport environment, there would be no way to properly encode them for the 7bit world, aside from waiting for all of the fragments, reassembling the inner message, and then encoding the reassembled data in base64 or quoted-printable. Since it is possible that different fragments might go through different gateways, even this is not an acceptable solution. For this reason, it is specified that entities of type "message/partial" must always have a content- transfer-encoding of 7bit (the default). In particular, even in environments that support binary or 8bit transport, the use of a content- transfer-encoding of "8bit" or "binary" is explicitly prohibited for MIME entities of type "message/partial". This in turn implies that the inner message must not use "8bit" or "binary" encoding. Formatted and Indexed by: page 29 of 44 instructional media + magic, inc. RFC2046 Web Address: Media Types http://sunsite.cnlab-switch.ch/ By: Freed & Borenstein Because some message transfer agents may choose to automatically fragment large messages, and because such agents may use very different fragmentation thresholds, it is possible that the pieces of a partial message, upon reassembly, may prove themselves to comprise a partial message. This is explicitly permitted. Three parameters must be specified in the Content-Type field of type "message/partial": The first, "id", is a unique identifier, as close to a world-unique identifier as possible, to be used to match the fragments together. (In general, the identifier is essentially a message-id; if placed in double quotes, it can be ANY message-id, in accordance with the BNF for "parameter" given in RFC 2045.) The second, "number", an integer, is the fragment number, which indicates where this fragment fits into the sequence of fragments. The third, "total", another integer, is the total number of fragments. This third subfield is required on the final fragment, and is optional (though encouraged) on the earlier fragments. Note also that these parameters may be given in any order. Thus, the second piece of a 3-piece message may have either of the following header fields: Content-Type: Message/Partial; number=2; total=3; id="[email protected]" Content-Type: Message/Partial; id="[email protected]"; number=2 But the third piece MUST specify the total number of fragments: Content-Type: Message/Partial; number=3; total=3; id="[email protected]" Note that fragment numbering begins with 1, not 0. When the fragments of an entity broken up in this manner are put together, the result is always a complete MIME entity, which may have its own Content-Type header field, and thus may contain any other data type. 5.2.2.1. Message Fragmentation and Reassembly The semantics of a reassembled partial message must be those of the "inner" message, rather than of a message containing the inner message. This makes it possible, for example, to send a large audio message as several partial messages, and still have it appear to the recipient as a simple audio message rather than as an encapsulated Formatted and Indexed by: page 30 of 44 instructional media + magic, inc. RFC2046 Web Address: Media Types http://sunsite.cnlab-switch.ch/ By: Freed & Borenstein message containing an audio message. That is, the encapsulation of the message is considered to be "transparent". When generating and reassembling the pieces of a "message/partial" message, the headers of the encapsulated message must be merged with the headers of the enclosing entities. In this process the following rules must be observed: (1) Fragmentation agents must split messages at line boundaries only. This restriction is imposed because splits at points other than the ends of lines in turn depends on message transports being able to preserve the semantics of messages that don't end with a CRLF sequence. Many transports are incapable of preserving such semantics. (2) All of the header fields from the initial enclosing message, except those that start with "Content-" and the specific header fields "Subject", "Message-ID", "Encrypted", and "MIME-Version", must be copied, in order, to the new message. (3) The header fields in the enclosed message which start with "Content-", plus the "Subject", "Message-ID", "Encrypted", and "MIME-Version" fields, must be appended, in order, to the header fields of the new message. Any header fields in the enclosed message which do not start with "Content-" (except for the "Subject", "Message-ID", "Encrypted", and "MIME- Version" fields) will be ignored and dropped. (4) All of the header fields from the second and any subsequent enclosing messages are discarded by the reassembly process. 5.2.2.2. Fragmentation and Reassembly Example If an audio message is broken into two pieces, the first piece might look something like this: X-Weird-Header-1: Foo From: [email protected] To: [email protected] Date: Fri, 26 Mar 1993 12:59:38 -0500 (EST) Subject: Audio mail (part 1 of 2) Message-ID: Formatted and Indexed by: page 31 of 44 instructional media + magic, inc. RFC2046 Web Address: Media Types http://sunsite.cnlab-switch.ch/ By: Freed & Borenstein number=1; total=2 X-Weird-Header-1: Bar X-Weird-Header-2: Hello Message-ID: ... first half of encoded audio data goes here ... and the second half might look something like this: From: [email protected] To: [email protected] Date: Fri, 26 Mar 1993 12:59:38 -0500 (EST) Subject: Audio mail (part 2 of 2) MIME-Version: 1.0 Message-ID: ... second half of encoded audio data goes here ... Then, when the fragmented message is reassembled, the resulting message to be displayed to the user should look something like this: X-Weird-Header-1: Foo From: [email protected] To: [email protected] Date: Fri, 26 Mar 1993 12:59:38 -0500 (EST) Subject: Audio mail Message-ID: ... first half of encoded audio data goes here ...... second half of encoded audio data goes here ... The inclusion of a "References" field in the headers of the second and subsequent pieces of a fragmented message that references the Message-Id on the previous piece may be of benefit to mail readers that understand and track references. However, the generation of such "References" fields is entirely optional. Formatted and Indexed by: page 32 of 44 instructional media + magic, inc. RFC2046 Web Address: Media Types http://sunsite.cnlab-switch.ch/ By: Freed & Borenstein Finally, it should be noted that the "Encrypted" header field has been made obsolete by Privacy Enhanced Messaging (PEM) [RFC-1421, RFC-1422, RFC-1423, RFC-1424], but the rules above are nevertheless believed to describe the correct way to treat it if it is encountered in the context of conversion to and from "message/partial" fragments. 5.2.3. External-Body Subtype The external-body subtype indicates that the actual body data are not included, but merely referenced. In this case, the parameters describe a mechanism for accessing the external data. When a MIME entity is of type "message/external-body", it consists of a header, two consecutive CRLFs, and the message header for the encapsulated message. If another pair of consecutive CRLFs appears, this of course ends the message header for the encapsulated message. However, since the encapsulated message's body is itself external, it does NOT appear in the area that follows. For example, consider the following message: Content-type: message/external-body; access-type=local-file; name="/u/nsb/Me.jpeg" Content-type: image/jpeg Content-ID: THIS IS NOT REALLY THE BODY! The area at the end, which might be called the "phantom body", is ignored for most external-body messages. However, it may be used to contain auxiliary information for some such messages, as indeed it is when the access-type is "mail- server". The only access-type defined in this document that uses the phantom body is "mail-server", but other access-types may be defined in the future in other specifications that use this area. The encapsulated headers in ALL "message/external-body" entities MUST include a Content-ID header field to give a unique identifier by which to reference the data. This identifier may be used for caching mechanisms, and for recognizing the receipt of the data when the access-type is "mail-server". Note that, as specified here, the tokens that describe external-body data, such as file names and mail server commands, are required to be in the US-ASCII character set. Formatted and Indexed by: page 33 of 44 instructional media + magic, inc. RFC2046 Web Address: Media Types http://sunsite.cnlab-switch.ch/ By: Freed & Borenstein If this proves problematic in practice, a new mechanism may be required as a future extension to MIME, either as newly defined access-types for "message/external-body" or by some other mechanism. As with "message/partial", MIME entities of type "message/external- body" MUST have a content-transfer-encoding of 7bit (the default). In particular, even in environments that support binary or 8bit transport, the use of a content- transfer-encoding of "8bit" or "binary" is explicitly prohibited for entities of type "message/external-body". 5.2.3.1. General External-Body Parameters The parameters that may be used with any "message/external- body" are: (1) ACCESS-TYPE -- A word indicating the supported access mechanism by which the file or data may be obtained. This word is not case sensitive. Values include, but are not limited to, "FTP", "ANON-FTP", "TFTP", "LOCAL- FILE", and "MAIL-SERVER". Future values, except for experimental values beginning with "X-", must be registered with IANA, as described in RFC 2048. This parameter is unconditionally mandatory and MUST be present on EVERY "message/external-body". (2) EXPIRATION -- The date (in the RFC 822 "date-time" syntax, as extended by RFC 1123 to permit 4 digits in the year field) after which the existence of the external data is not guaranteed. This parameter may be used with ANY access-type and is ALWAYS optional. (3) SIZE -- The size (in octets) of the data. The intent of this parameter is to help the recipient decide whether or not to expend the necessary resources to retrieve the external data. Note that this describes the size of the data in its canonical form, that is, before any Content-Transfer-Encoding has been applied or after the data have been decoded. This parameter may be used with ANY access-type and is ALWAYS optional. (4) PERMISSION -- A case-insensitive field that indicates whether or not it is expected that clients might also attempt to overwrite the data. By default, or if permission is "read", the assumption is that they are not, and that if the data is retrieved once, it is never needed again. If PERMISSION is "read-write", Formatted and Indexed by: page 34 of 44 instructional media + magic, inc. RFC2046 Web Address: Media Types http://sunsite.cnlab-switch.ch/ By: Freed & Borenstein this assumption is invalid, and any local copy must be considered no more than a cache. "Read" and "Read- write" are the only defined values of permission. This parameter may be used with ANY access-type and is ALWAYS optional. The precise semantics of the access-types defined here are described in the sections that follow. 5.2.3.2. The 'ftp' and 'tftp' Access-Types An access-type of FTP or TFTP indicates that the message body is accessible as a file using the FTP [RFC-959] or TFTP [RFC- 783] protocols, respectively. For these access-types, the following additional parameters are mandatory: (1) NAME -- The name of the file that contains the actual body data. (2) SITE -- A machine from which the file may be obtained, using the given protocol. This must be a fully qualified domain name, not a nickname. (3) Before any data are retrieved, using FTP, the user will generally need to be asked to provide a login id and a password for the machine named by the site parameter. For security reasons, such an id and password are not specified as content-type parameters, but must be obtained from the user. In addition, the following parameters are optional: (1) DIRECTORY -- A directory from which the data named by NAME should be retrieved. (2) MODE -- A case-insensitive string indicating the mode to be used when retrieving the information. The valid values for access-type "TFTP" are "NETASCII", "OCTET", and "MAIL", as specified by the TFTP protocol [RFC- 783]. The valid values for access-type "FTP" are "ASCII", "EBCDIC", "IMAGE", and "LOCALn" where "n" is a decimal integer, typically 8. These correspond to the representation types "A" "E" "I" and "L n" as specified by the FTP protocol [RFC-959]. Note that "BINARY" and "TENEX" are not valid values for MODE and that "OCTET" or "IMAGE" or "LOCAL8" should be used instead. IF MODE is not specified, the default value is "NETASCII" for TFTP and "ASCII" otherwise. Formatted and Indexed by: page 35 of 44 instructional media + magic, inc. RFC2046 Web Address: Media Types http://sunsite.cnlab-switch.ch/ By: Freed & Borenstein 5.2.3.3. The 'anon-ftp' Access-Type The "anon-ftp" access-type is identical to the "ftp" access type, except that the user need not be asked to provide a name and password for the specified site. Instead, the ftp protocol will be used with login "anonymous" and a password that corresponds to the user's mail address. 5.2.3.4. The 'local-file' Access-Type An access-type of "local-file" indicates that the actual body is accessible as a file on the local machine. Two additional parameters are defined for this access type: (1) NAME -- The name of the file that contains the actual body data. This parameter is mandatory for the "local-file" access-type. (2) SITE -- A domain specifier for a machine or set of machines that are known to have access to the data file. This optional parameter is used to describe the locality of reference for the data, that is, the site or sites at which the file is expected to be visible. Asterisks may be used for wildcard matching to a part of a domain name, such as "*.bellcore.com", to indicate a set of machines on which the data should be directly visible, while a single asterisk may be used to indicate a file that is expected to be universally available, e.g., via a global file system. 5.2.3.5. The 'mail-server' Access-Type The "mail-server" access-type indicates that the actual body is available from a mail server. Two additional parameters are defined for this access-type: (1) SERVER -- The addr-spec of the mail server from which the actual body data can be obtained. This parameter is mandatory for the "mail-server" access-type. (2) SUBJECT -- The subject that is to be used in the mail that is sent to obtain the data. Note that keying mail servers on Subject lines is NOT recommended, but such mail servers are known to exist. This is an optional parameter. Formatted and Indexed by: page 36 of 44 instructional media + magic, inc. RFC2046 Web Address: Media Types http://sunsite.cnlab-switch.ch/ By: Freed & Borenstein Because mail servers accept a variety of syntaxes, some of which is multiline, the full command to be sent to a mail server is not included as a parameter in the content-type header field. Instead, it is provided as the "phantom body" when the media type is "message/external-body" and the access-type is mail-server. Note that MIME does not define a mail server syntax. Rather, it allows the inclusion of arbitrary mail server commands in the phantom body. Implementations must include the phantom body in the body of the message it sends to the mail server address to retrieve the relevant data. Unlike other access-types, mail-server access is asynchronous and will happen at an unpredictable time in the future. For this reason, it is important that there be a mechanism by which the returned data can be matched up with the original "message/external-body" entity. MIME mail servers must use the same Content-ID field on the returned message that was used in the original "message/external-body" entities, to facilitate such matching. 5.2.3.6. External-Body Security Issues "Message/external-body" entities give rise to two important security issues: (1) Accessing data via a "message/external-body" reference effectively results in the message recipient performing an operation that was specified by the message originator. It is therefore possible for the message originator to trick a recipient into doing something they would not have done otherwise. For example, an originator could specify a action that attempts retrieval of material that the recipient is not authorized to obtain, causing the recipient to unwittingly violate some security policy. For this reason, user agents capable of resolving external references must always take steps to describe the action they are to take to the recipient and ask for explicit permisssion prior to performing it. The 'mail-server' access-type is particularly vulnerable, in that it causes the recipient to send a new message whose contents are specified by the original message's originator. Given the potential for abuse, any such request messages that are constructed should contain a clear indication that they were generated automatically (e.g. in a Comments: header field) in an attempt to resolve a MIME Formatted and Indexed by: page 37 of 44 instructional media + magic, inc. RFC2046 Web Address: Media Types http://sunsite.cnlab-switch.ch/ By: Freed & Borenstein "message/external-body" reference. (2) MIME will sometimes be used in environments that provide some guarantee of message integrity and authenticity. If present, such guarantees may apply only to the actual direct content of messages -- they may or may not apply to data accessed through MIME's "message/external-body" mechanism. In particular, it may be possible to subvert certain access mechanisms even when the messaging system itself is secure. It should be noted that this problem exists either with or without the availabilty of MIME mechanisms. A casual reference to an FTP site containing a document in the text of a secure message brings up similar issues -- the only difference is that MIME provides for automatic retrieval of such material, and users may place unwarranted trust is such automatic retrieval mechanisms. 5.2.3.7. Examples and Further Explanations When the external-body mechanism is used in conjunction with the "multipart/alternative" media type it extends the functionality of "multipart/alternative" to include the case where the same entity is provided in the same format but via different accces mechanisms. When this is done the originator of the message must order the parts first in terms of preferred formats and then by preferred access mechanisms. The recipient's viewer should then evaluate the list both in terms of format and access mechanisms. With the emerging possibility of very wide-area file systems, it becomes very hard to know in advance the set of machines where a file will and will not be accessible directly from the file system. Therefore it may make sense to provide both a file name, to be tried directly, and the name of one or more sites from which the file is known to be accessible. An implementation can try to retrieve remote files using FTP or any other protocol, using anonymous file retrieval or prompting the user for the necessary name and password. If an external body is accessible via multiple mechanisms, the sender may include multiple entities of type "message/external-body" within the body parts of an enclosing "multipart/alternative" entity. However, the external-body mechanism is not intended to be limited to file retrieval, as shown by the mail-server access-type. Beyond this, one can imagine, for example, using a video server for external references to video clips. Formatted and Indexed by: page 38 of 44 instructional media + magic, inc. RFC2046 Web Address: Media Types http://sunsite.cnlab-switch.ch/ By: Freed & Borenstein The embedded message header fields which appear in the body of the "message/external-body" data must be used to declare the media type of the external body if it is anything other than plain US-ASCII text, since the external body does not have a header section to declare its type. Similarly, any Content-transfer-encoding other than "7bit" must also be declared here. Thus a complete "message/external-body" message, referring to an object in PostScript format, might look like this: From: Whomever To: Someone Date: Whenever Subject: whatever MIME-Version: 1.0 Message-ID: --42 Content-Type: message/external-body; name="BodyFormats.ps"; site="thumper.bellcore.com"; mode="image"; access-type=ANON-FTP; directory="pub"; expiration="Fri, 14 Jun 1991 19:13:14 -0400 (EDT)" Content-type: application/postscript Content-ID: --42 Content-Type: message/external-body; access-type=local-file; name="/u/nsb/writing/rfcs/RFC-MIME.ps"; site="thumper.bellcore.com"; expiration="Fri, 14 Jun 1991 19:13:14 -0400 (EDT)" Content-type: application/postscript Content-ID: --42 Content-Type: message/external-body; access-type=mail-server server="[email protected]"; expiration="Fri, 14 Jun 1991 19:13:14 -0400 (EDT)" Content-type: application/postscript Content-ID: get RFC-MIME.DOC --42-- Formatted and Indexed by: page 39 of 44 instructional media + magic, inc. RFC2046 Web Address: Media Types http://sunsite.cnlab-switch.ch/ By: Freed & Borenstein Note that in the above examples, the default Content-transfer- encoding of "7bit" is assumed for the external postscript data. Like the "message/partial" type, the "message/external-body" media type is intended to be transparent, that is, to convey the data type in the external body rather than to convey a message with a body of that type. Thus the headers on the outer and inner parts must be merged using the same rules as for "message/partial". In particular, this means that the Content-type and Subject fields are overridden, but the From field is preserved. Note that since the external bodies are not transported along with the external body reference, they need not conform to transport limitations that apply to the reference itself. In particular, Internet mail transports may impose 7bit and line length limits, but these do not automatically apply to binary external body references. Thus a Content-Transfer-Encoding is not generally necessary, though it is permitted. Note that the body of a message of type "message/external-body" is governed by the basic syntax for an RFC 822 message. In particular, anything before the first consecutive pair of CRLFs is header information, while anything after it is body information, which is ignored for most access-types. 5.2.4. Other Message Subtypes MIME implementations must in general treat unrecognized subtypes of "message" as being equivalent to "application/octet-stream". Future subtypes of "message" intended for use with email should be restricted to "7bit" encoding. A type other than "message" should be used if restriction to "7bit" is not possible. 6. Experimental Media Type Values A media type value beginning with the characters "X-" is a private value, to be used by consenting systems by mutual agreement. Any format without a rigorous and public definition must be named with an "X-" prefix, and publicly specified values shall never begin with "X-". (Older versions of the widely used Andrew system use the "X- BE2" name, so new systems should probably choose a different name.) In general, the use of "X-" top-level types is strongly discouraged. Implementors should invent subtypes of the existing types whenever possible. In many cases, a subtype of "application" will be more appropriate than a new top-level type. Formatted and Indexed by: page 40 of 44 instructional media + magic, inc. RFC2046 Web Address: Media Types http://sunsite.cnlab-switch.ch/ By: Freed & Borenstein 7. Summary The five discrete media types provide provide a standardized mechanism for tagging entities as "audio", "image", or several other kinds of data. The composite "multipart" and "message" media types allow mixing and hierarchical structuring of entities of different types in a single message. A distinguished parameter syntax allows further specification of data format details, particularly the specification of alternate character sets. Additional optional header fields provide mechanisms for certain extensions deemed desirable by many implementors. Finally, a number of useful media types are defined for general use by consenting user agents, notably "message/partial" and "message/external-body". 9. Security Considerations Security issues are discussed in the context of the "application/postscript" type, the "message/external-body" type, and in RFC 2048. Implementors should pay special attention to the security implications of any media types that can cause the remote execution of any actions in the recipient's environment. In such cases, the discussion of the "application/postscript" type may serve as a model for considering other media types with remote execution capabilities. Formatted and Indexed by: page 41 of 44 instructional media + magic, inc. RFC2046 Web Address: Media Types http://sunsite.cnlab-switch.ch/ By: Freed & Borenstein 9. Authors' Addresses For more information, the authors of this document are best contacted via Internet mail: Ned Freed Innosoft International, Inc. 1050 East Garvey Avenue South West Covina, CA 91790 USA Phone: +1 818 919 3600 Fax: +1 818 919 3614 EMail: [email protected] Nathaniel S. Borenstein First Virtual Holdings 25 Washington Avenue Morristown, NJ 07960 USA Phone: +1 201 540 8967 Fax: +1 201 993 3032 EMail: [email protected] MIME is a result of the work of the Internet Engineering Task Force Working Group on RFC 822 Extensions. The chairman of that group, Greg Vaudreuil, may be reached at: Gregory M. Vaudreuil Octel Network Services 17080 Dallas Parkway Dallas, TX 75248-1905 USA EMail: [email protected] Formatted and Indexed by: page 42 of 44 instructional media + magic, inc. RFC2046 Web Address: Media Types http://sunsite.cnlab-switch.ch/ By: Freed & Borenstein Appendix A -- Collected Grammar This appendix contains the complete BNF grammar for all the syntax specified by this document. By itself, however, this grammar is incomplete. It refers by name to several syntax rules that are defined by RFC 822. Rather than reproduce those definitions here, and risk unintentional differences between the two, this document simply refers the reader to RFC 822 for the remaining definitions. Wherever a term is undefined, it refers to the RFC 822 definition. boundary := 0*69 bchars := bcharsnospace / " " bcharsnospace := DIGIT / ALPHA / "'" / "(" / ")" / "+" / "_" / "," / "-" / "." / "/" / ":" / "=" / "?" body-part := <"message" as defined in RFC 822, with all header fields optional, not starting with the specified dash-boundary, and with the delimiter not occurring anywhere in the body part. Note that the semantics of a part differ from the semantics of a message, as described in the text.> close-delimiter := delimiter "--" dash-boundary := "--" boundary ; boundary taken from the value of ; boundary parameter of the ; Content-Type field. delimiter := CRLF dash-boundary discard-text := *(*text CRLF) ; May be ignored or discarded. encapsulation := delimiter transport-padding CRLF body-part epilogue := discard-text multipart-body := [preamble CRLF] dash-boundary transport-padding CRLF body-part *encapsulation Formatted and Indexed by: page 43 of 44 instructional media + magic, inc. RFC2046 Web Address: Media Types http://sunsite.cnlab-switch.ch/ By: Freed & Borenstein close-delimiter transport-padding [CRLF epilogue] preamble := discard-text transport-padding := *LWSP-char ; Composers MUST NOT generate ; non-zero length transport ; padding, but receivers MUST ; be able to handle padding ; added by message transports. Formatted and Indexed by: page 44 of 44 instructional media + magic, inc. RFC2056 Web Address: URLs for Z39.50 http://sunsite.cnlab-switch.ch/ By: Denenberg, et. al. Network Working Group R. Denenberg Request for Comments: 2056 Library of Congress Category: Standards Track J. Kunze University of California, San Francisco D. Lynch SilverPlatter Information Ltd. Editors November 1996 Uniform Resource Locators for Z39.50 Status of this Memo This document specifies an Internet standards track protocol for the Internet community, and requests discussion and suggestions for improvements. Please refer to the current edition of the "Internet Official Protocol Standards" (STD 1) for the standardization state and status of this protocol. Distribution of this memo is unlimited. 1. Introduction Z39.50 is an information retrieval protocol that does not fit neatly into a retrieval model designed primarily around the stateless fetch of data. Instead, it models a general user inquiry as a session- oriented, multi-step task, any step of which may be suspended temporarily while the server requests additional parameters from the client before continuing. Some, none, or all of these client/server interactions may require participation of the client user, depending only on the client software (the protocol itself makes no such requirements). On the other hand, retrieval of "well-known" data may be performed in a single step, that is, with a degenerate Z39.50 session consisting of exactly one protocol search request and response. Besides the basic search sub-service, there are several ancillary sub-services (e.g., Scan, Result Set Delete). Among the functions covered by combinations of the sub-services, two core functions emerge as appropriately handled by two separate URL schemes: the Session URL and the Retrieval URL. Using two schemes instead of one makes a critical distinction between a Z39.50 Session URL, which opens a client session initialized for interactive use by the user, and a Z39.50 Retrieval URL, which opens and closes a client session to retrieve a specific information item. Making this distinction at the scheme level allows the user interface to reflect it on to the user, without requiring the user interface to Formatted and Indexed by: page 1 of 7 instructional media + magic, inc. RFC2056 Web Address: URLs for Z39.50 http://sunsite.cnlab-switch.ch/ By: Denenberg, et. al. parse otherwise opaque parts of the URL (consistent with current practice). 2. Some Basic Concepts This section briefly describes the usage of Z39.50-specific terminology within the URL definitions below: specifically, the terms database, elementset, recordsyntax, and docid. The Z39.50 protocol specifies various information retrieval operations, the two most basic of which are Search and Present. In a Search operation a client provides search criteria and indicates a database (or several databases) on the server to search. The essential result of a Search operation is that a result set is created at the server, consisting of pointers to the selected database records. Z39.50 models database records, abstract database records, and retrieval records. A database record is a unit of information in a database, represented in a data structure local to the server. An abstract database record is an abstract representation of that information, where the client and server share a common understanding of the representation. This allows logical elements to be addressed and selected for transfer, via an element set specification, or, as used below, an "elementset". A retrieval record is the set of selected elements packaged in an exportable structure, by the application of a "recordsyntax". Thus a Search operation results in entries pointing to database records; via a Present operation, a client requests a retrieval record, corresponding to a database record, corresponding to an entry in the result set. The client indicates the composition and format of the retrieval record by specifying an elementset and recordsyntax, respectively. A special case of a Z39.50 search is a "known-item" search, when a client intends that a search identify a single, known database record, or "document" (for purposes of illustration, assume that a database record corresponds to a document), and further, the client knows an identifier for the document that can be used to effect this known-item search. In this case, this identifier is often referred to as a document identifier, or "docid". Formatted and Indexed by: page 2 of 7 instructional media + magic, inc. RFC2056 Web Address: URLs for Z39.50 http://sunsite.cnlab-switch.ch/ By: Denenberg, et. al. 3. The Z39.50 Session URL The Z39.50 Session URL may be informally described as providing the mechanism to switch the user to a Z39.50 client application. - Host is required. - Port is optional, and defaults to 210. - All other parameters are optional. - The Z39.50 client will start a session to the specified host/port (alternatively, it need not explicitly start a session, but may instead utilize an already open session to the same host/port). - A database must be included if docid is included. - If docid is included, the client will perform the specified search (in the same manner as for the retrieval URL, specified below). - If docid is not included, and other parameters (besides host/port) are specified, the client may use those parameters as "hints". Various clients may choose to treat them as requirements, or as preferences, or ignore them. - In any case (whether a search is performed or not), the client will leave the Z39.50 session open for the user, to do retrievals, new searches, etc. (This is the main distinction from the Retrieval URL which leaves it up to the client whether or not to keep the Z39.50 session open.) 4. The Z39.50 Retrieval URL The Z39.50 Retrieval URL is intended to allow a Z39.50 session to be used as a transparent transfer mechanism to retrieve a specific information object. A Z39.50 client uses information in the URL to formulate a Search Request. The server's Search Response indicates how many records match the Request. If the number of matching records does not equal one, the retrieval is considered unsuccessful, and the client application's behavior is not defined. If the number of matching records equals one, the server may have included the desired record in the Search Response. If not, the client requests transmission of the record with a Present Request. After the client has received the specified record it may close the Z39.50 session immediately, or keep it open for subsequent retrievals. - Host is required. - Port is optional, and defaults to 210. - A database is required. - The meaning of a retrieval URL with no docid is undefined. - The docid is placed into a type-1 query, as the single term, in the general format (tag 45), using the Bib-1 attribute set, with a Use attribute value of docid, and a structure attribute of URx. The docid string is server-defined and completely opaque to the client. Formatted and Indexed by: page 3 of 7 instructional media + magic, inc. RFC2056 Web Address: URLs for Z39.50 http://sunsite.cnlab-switch.ch/ By: Denenberg, et. al. - If element set name (esn) is not specified, it is the client's choice. If esn is specified, it should be used either in the Search request for the value of small- and/or medium- set-element-set-names or in a Present request following a Search. These terms and their use are defined within the Z39.50 Standard [2]. - If record syntax (rs) is not specified, it is the client's choice. If one or more record syntaxes are specified, the client should select one (preferably the first in the list that it supports) and use it in a Search or Present request as the value of PreferredRecordSyntax. 5. BNF for Z39.50 URLs The Z39.50 Session and Retrieval URLs follow the Common Internet Scheme Syntax as defined in RFC 1738, "Uniform Resource Locators (URL)" [1]. In the definition, literals are quoted with "", optional elements are enclosed in [brackets], "|" is used to designate alternatives, and elements may be preceded with Future extensions to these URLs will be of the form of [;keyword=value]. The following definitions are from RFC 1738. Between the Internet Draft version and RFC 1738 two relevant changes were made: '=' was moved from the Formatted and Indexed by: page 4 of 7 instructional media + magic, inc. RFC2056 Web Address: URLs for Z39.50 http://sunsite.cnlab-switch.ch/ By: Denenberg, et. al. lowalpha = "a" | "b" | "c" | "d" | "e" | "f" | "g" | "h" | "i" | "j" | "k" | "l" | "m" | "n" | "o" | "p" | "q" | "r" | "s" | "t" | "u" | "v" | "w" | "x" | "y" | "z" hialpha = "A" | "B" | "C" | "D" | "E" | "F" | "G" | "H" | "I" | "J" | "K" | "L" | "M" | "N" | "O" | "P" | "Q" | "R" | "S" | "T" | "U" | "V" | "W" | "X" | "Y" | "Z" alpha = lowalpha | hialpha digit = "0" | "1" | "2" | "3" | "4" | "5" | "6" | "7" | "8" | "9" safe = "$" | "-" | "_" | "." | "+" extra = "!" | "*" | "'" | "(" | ")" | "," national = "{" | "}" | "|" | "\" | "^" | "~" | "[" | "]" | "`" punctuation = "<" | ">" | "#" | "%" | <"> reserved = ";" | "/" | "?" | ":" | "@" | "&" | "=" hex = digit | "A" | "B" | "C" | "D" | "E" | "F" | "a" | "b" | "c" | "d" | "e" | "f" escape = "%" hex hex unreserved = alpha | digit | safe | extra uchar = unreserved | escape xchar = unreserved | reserved | escape digits = 1*digit 6. Security Considerations The two Z39.50 URL schemes are subject to the same security implications as the general URL scheme [1], so the usual precautions apply. This means, for example, that a locator might no longer point to the object that was originally intended. It also means that it may be possible to construct a URL so that an attempt to perform a harmless idempotent operation such as the retrieval of an object will in fact cause a possibly damaging remote operation to occur. 7. Acknowledgements The Z39.50 Implementors Group contributed the substance of this document. 8. References [1] Berners-Lee, T., Masinter, L., McCahill, M. (editors), "Uniform Resource Locators (URL)", RFC 1738, December 1994. ftp://ds.internic.net/rfc/rfc1738.txt [2] ANSI/NISO Z39.50-1995, "ANSI Z39.50: Information Retrieval Service and Protocol", 1995. ftp://ftp.loc.gov/pub/z3950/ Formatted and Indexed by: page 5 of 7 instructional media + magic, inc. RFC2056 Web Address: URLs for Z39.50 http://sunsite.cnlab-switch.ch/ By: Denenberg, et. al. [3] ANSI/NISO Z39.50-1992, "ANSI Z39.50: Information Retrieval Service and Protocol", 1992. ftp://ftp.cni.org/pub/NISO/docs/Z39.50-1992/www/Z39.50.toc.html (also available in hard copy from Omnicom Information Service, 115 Park St., SE, Vienna, VA 22180). 9. Editors' Addresses Ray Denenberg Library of Congress Collections Services Network Development/MSO Washington DC 20540 Phone: (202) 707-5795 Fax: (202) 707-0115 EMail: [email protected] John A. Kunze Center for Knowledge Management University of California, San Francisco 530 Parnassus Ave, Box 0840 San Francisco, CA 94143-0840 Phone: (415) 502-6660 Fax: (415) 476-4653 EMail: [email protected] Denis Lynch SilverPlatter Information Ltd. 10 Barely Mow Passage Chiswick, London W4 4PH U.K. Voice: +44 (0)181-995-8242 Fax: +44 (0)181-995-5159 EMail: [email protected] Formatted and Indexed by: page 6 of 7 instructional media + magic, inc. RFC2056 Web Address: URLs for Z39.50 http://sunsite.cnlab-switch.ch/ By: Denenberg, et. al. Appendix. Examples of Z39.50 URLs A basic Z39.50 session URL that a client might use to open a connection to the MELVYL union catalog "cat" at the University of California is z39.50s://melvyl.ucop.edu/cat A URL that would open the MELVYL magazine database just long enough to fetch an article from volume 30, number 19 of a hypothetical periodical might look like z39.50r://melvyl.ucop.edu/mags?elecworld.v30.n19 As a final example, here is another retrieval URL that a client could use to request a full record (element set "f") in the MARC syntax from a hypothetical database called TMF at CNIDR: z39.50r://cnidr.org:2100/tmf?bkirch_rules__a1;esn=f;rs=marc As in the previous example, the part of the string after the `?' is determined by the server. In this example, the server is running on non-standard port 2100. _ Formatted and Indexed by: page 7 of 7 instructional media + magic, inc. RFC2083 Web Address: PNG: Portable Network Graphics http://sunsite.cnlab-switch.ch/ By: Boutell, et. al. Network Working Group T. Boutell, et. al. Request for Comments: 2083 Boutell.Com, Inc. Category: Informational March 1997 PNG (Portable Network Graphics) Specification Version 1.0 Status of this Memo This memo provides information for the Internet community. This memo does not specify an Internet standard of any kind. Distribution of this memo is unlimited. IESG Note: The IESG takes no position on the validity of any Intellectual Property Rights statements contained in this document. Abstract This document describes PNG (Portable Network Graphics), an extensible file format for the lossless, portable, well-compressed storage of raster images. PNG provides a patent-free replacement for GIF and can also replace many common uses of TIFF. Indexed-color, grayscale, and truecolor images are supported, plus an optional alpha channel. Sample depths range from 1 to 16 bits. PNG is designed to work well in online viewing applications, such as the World Wide Web, so it is fully streamable with a progressive display option. PNG is robust, providing both full file integrity checking and simple detection of common transmission errors. Also, PNG can store gamma and chromaticity data for improved color matching on heterogeneous platforms. This specification defines the Internet Media Type image/png. Table of Contents 1. Introduction ...... 4 2. Data Representation ...... 5 2.1. Integers and byte order ...... 5 2.2. Color values ...... 6 2.3. Image layout ...... 6 2.4. Alpha channel ...... 7 2.5. Filtering ...... 8 2.6. Interlaced data order ...... 8 2.7. Gamma correction ...... 10 Formatted and Indexed by: page 1 of 102 instructional media + magic, inc. RFC2083 Web Address: PNG: Portable Network Graphics http://sunsite.cnlab-switch.ch/ By: Boutell, et. al. 2.8. Text strings ...... 10 3. File Structure ...... 11 3.1. PNG file signature ...... 11 3.2. Chunk layout ...... 11 3.3. Chunk naming conventions ...... 12 3.4. CRC algorithm ...... 15 4. Chunk Specifications ...... 15 4.1. Critical chunks ...... 15 4.1.1. IHDR Image header ...... 15 4.1.2. PLTE Palette ...... 17 4.1.3. IDAT Image data ...... 18 4.1.4. IEND Image trailer ...... 19 4.2. Ancillary chunks ...... 19 4.2.1. bKGD Background color ...... 19 4.2.2. cHRM Primary chromaticities and white point ...... 20 4.2.3. gAMA Image gamma ...... 21 4.2.4. hIST Image histogram ...... 21 4.2.5. pHYs Physical pixel dimensions ...... 22 4.2.6. sBIT Significant bits ...... 22 4.2.7. tEXt Textual data ...... 24 4.2.8. tIME Image last-modification time ...... 25 4.2.9. tRNS Transparency ...... 26 4.2.10. zTXt Compressed textual data ...... 27 4.3. Summary of standard chunks ...... 28 4.4. Additional chunk types ...... 29 5. Deflate/Inflate Compression ...... 29 6. Filter Algorithms ...... 31 6.1. Filter types ...... 31 6.2. Filter type 0: None ...... 32 6.3. Filter type 1: Sub ...... 33 6.4. Filter type 2: Up ...... 33 6.5. Filter type 3: Average ...... 34 6.6. Filter type 4: Paeth...... 35 7. Chunk Ordering Rules ...... 36 7.1. Behavior of PNG editors ...... 37 7.2. Ordering of ancillary chunks ...... 38 7.3. Ordering of critical chunks ...... 38 8. Miscellaneous Topics ...... 39 8.1. File name extension ...... 39 8.2. Internet media type ...... 39 8.3. Macintosh file layout ...... 39 8.4. Multiple-image extension ...... 39 8.5. Security considerations ...... 40 9. Recommendations for Encoders ...... 41 9.1. Sample depth scaling ...... 41 9.2. Encoder gamma handling ...... 42 9.3. Encoder color handling ...... 45 9.4. Alpha channel creation ...... 47 Formatted and Indexed by: page 2 of 102 instructional media + magic, inc. RFC2083 Web Address: PNG: Portable Network Graphics http://sunsite.cnlab-switch.ch/ By: Boutell, et. al. 9.5. Suggested palettes ...... 48 9.6. Filter selection ...... 49 9.7. Text chunk processing ...... 49 9.8. Use of private chunks ...... 50 9.9. Private type and method codes ...... 51 10. Recommendations for Decoders ...... 51 10.1. Error checking ...... 52 10.2. Pixel dimensions ...... 52 10.3. Truecolor image handling ...... 52 10.4. Sample depth rescaling ...... 53 10.5. Decoder gamma handling ...... 54 10.6. Decoder color handling ...... 56 10.7. Background color ...... 57 10.8. Alpha channel processing ...... 58 10.9. Progressive display ...... 62 10.10. Suggested-palette and histogram usage ...... 63 10.11. Text chunk processing ...... 64 11. Glossary ...... 65 12. Appendix: Rationale ...... 69 12.1. Why a new file format? ...... 69 12.2. Why these features? ...... 70 12.3. Why not these features? ...... 70 12.4. Why not use format X? ...... 72 12.5. Byte order ...... 73 12.6. Interlacing ...... 73 12.7. Why gamma? ...... 73 12.8. Non-premultiplied alpha ...... 75 12.9. Filtering ...... 75 12.10. Text strings ...... 76 12.11. PNG file signature ...... 77 12.12. Chunk layout ...... 77 12.13. Chunk naming conventions ...... 78 12.14. Palette histograms ...... 80 13. Appendix: Gamma Tutorial ...... 81 14. Appendix: Color Tutorial ...... 89 15. Appendix: Sample CRC Code ...... 94 16. Appendix: Online Resources ...... 96 17. Appendix: Revision History ...... 96 18. References ...... 97 19. Credits ...... 100 Formatted and Indexed by: page 3 of 102 instructional media + magic, inc. RFC2083 Web Address: PNG: Portable Network Graphics http://sunsite.cnlab-switch.ch/ By: Boutell, et. al. 1. Introduction The PNG format provides a portable, legally unencumbered, well- compressed, well-specified standard for lossless bitmapped image files. Although the initial motivation for developing PNG was to replace GIF, the design provides some useful new features not available in GIF, with minimal cost to developers. GIF features retained in PNG include: * Indexed-color images of up to 256 colors. * Streamability: files can be read and written serially, thus allowing the file format to be used as a communications protocol for on-the-fly generation and display of images. * Progressive display: a suitably prepared image file can be displayed as it is received over a communications link, yielding a low-resolution image very quickly followed by gradual improvement of detail. * Transparency: portions of the image can be marked as transparent, creating the effect of a non-rectangular image. * Ancillary information: textual comments and other data can be stored within the image file. * Complete hardware and platform independence. * Effective, 100% lossless compression. Important new features of PNG, not available in GIF, include: * Truecolor images of up to 48 bits per pixel. * Grayscale images of up to 16 bits per pixel. * Full alpha channel (general transparency masks). * Image gamma information, which supports automatic display of images with correct brightness/contrast regardless of the machines used to originate and display the image. * Reliable, straightforward detection of file corruption. * Faster initial presentation in progressive display mode. PNG is designed to be: * Simple and portable: developers should be able to implement PNG easily. * Legally unencumbered: to the best knowledge of the PNG authors, no algorithms under legal challenge are used. (Some considerable effort has been spent to verify this.) * Well compressed: both indexed-color and truecolor images are compressed as effectively as in any other widely used lossless format, and in most cases more effectively. Formatted and Indexed by: page 4 of 102 instructional media + magic, inc. RFC2083 Web Address: PNG: Portable Network Graphics http://sunsite.cnlab-switch.ch/ By: Boutell, et. al. * Interchangeable: any standard-conforming PNG decoder must read all conforming PNG files. * Flexible: the format allows for future extensions and private add-ons, without compromising interchangeability of basic PNG. * Robust: the design supports full file integrity checking as well as simple, quick detection of common transmission errors. The main part of this specification gives the definition of the file format and recommendations for encoder and decoder behavior. An appendix gives the rationale for many design decisions. Although the rationale is not part of the formal specification, reading it can help implementors understand the design. Cross-references in the main text point to relevant parts of the rationale. Additional appendixes, also not part of the formal specification, provide tutorials on gamma and color theory as well as other supporting material. In this specification, the word "must" indicates a mandatory requirement, while "should" indicates recommended behavior. See Rationale: Why a new file format? (Section 12.1), Why these features? (Section 12.2), Why not these features? (Section 12.3), Why not use format X? (Section 12.4). Pronunciation PNG is pronounced "ping". 2. Data Representation This chapter discusses basic data representations used in PNG files, as well as the expected representation of the image data. 2.1. Integers and byte order All integers that require more than one byte must be in network byte order: the most significant byte comes first, then the less significant bytes in descending order of significance (MSB LSB for two-byte integers, B3 B2 B1 B0 for four-byte integers). The highest bit (value 128) of a byte is numbered bit 7; the lowest bit (value 1) is numbered bit 0. Values are unsigned unless otherwise noted. Values explicitly noted as signed are represented in two's complement notation. See Rationale: Byte order (Section 12.5). Formatted and Indexed by: page 5 of 102 instructional media + magic, inc. RFC2083 Web Address: PNG: Portable Network Graphics http://sunsite.cnlab-switch.ch/ By: Boutell, et. al. 2.2. Color values Colors can be represented by either grayscale or RGB (red, green, blue) sample data. Grayscale data represents luminance; RGB data represents calibrated color information (if the cHRM chunk is present) or uncalibrated device-dependent color (if cHRM is absent). All color values range from zero (representing black) to most intense at the maximum value for the sample depth. Note that the maximum value at a given sample depth is (2^sampledepth)-1, not 2^sampledepth. Sample values are not necessarily linear; the gAMA chunk specifies the gamma characteristic of the source device, and viewers are strongly encouraged to compensate properly. See Gamma correction (Section 2.7). Source data with a precision not directly supported in PNG (for example, 5 bit/sample truecolor) must be scaled up to the next higher supported bit depth. This scaling is reversible with no loss of data, and it reduces the number of cases that decoders have to cope with. See Recommendations for Encoders: Sample depth scaling (Section 9.1) and Recommendations for Decoders: Sample depth rescaling (Section 10.4). 2.3. Image layout Conceptually, a PNG image is a rectangular pixel array, with pixels appearing left-to-right within each scanline, and scanlines appearing top-to-bottom. (For progressive display purposes, the data may actually be transmitted in a different order; see Interlaced data order, Section 2.6.) The size of each pixel is determined by the bit depth, which is the number of bits per sample in the image data. Three types of pixel are supported: * An indexed-color pixel is represented by a single sample that is an index into a supplied palette. The image bit depth determines the maximum number of palette entries, but not the color precision within the palette. * A grayscale pixel is represented by a single sample that is a grayscale level, where zero is black and the largest value for the bit depth is white. * A truecolor pixel is represented by three samples: red (zero = black, max = red) appears first, then green (zero = black, max = green), then blue (zero = black, max = blue). The bit depth specifies the size of each sample, not the total pixel size. Formatted and Indexed by: page 6 of 102 instructional media + magic, inc. RFC2083 Web Address: PNG: Portable Network Graphics http://sunsite.cnlab-switch.ch/ By: Boutell, et. al. Optionally, grayscale and truecolor pixels can also include an alpha sample, as described in the next section. Pixels are always packed into scanlines with no wasted bits between pixels. Pixels smaller than a byte never cross byte boundaries; they are packed into bytes with the leftmost pixel in the high-order bits of a byte, the rightmost in the low-order bits. Permitted bit depths and pixel types are restricted so that in all cases the packing is simple and efficient. PNG permits multi-sample pixels only with 8- and 16-bit samples, so multiple samples of a single pixel are never packed into one byte. 16-bit samples are stored in network byte order (MSB first). Scanlines always begin on byte boundaries. When pixels have fewer than 8 bits and the scanline width is not evenly divisible by the number of pixels per byte, the low-order bits in the last byte of each scanline are wasted. The contents of these wasted bits are unspecified. An additional "filter type" byte is added to the beginning of every scanline (see Filtering, Section 2.5). The filter type byte is not considered part of the image data, but it is included in the datastream sent to the compression step. 2.4. Alpha channel An alpha channel, representing transparency information on a per- pixel basis, can be included in grayscale and truecolor PNG images. An alpha value of zero represents full transparency, and a value of (2^bitdepth)-1 represents a fully opaque pixel. Intermediate values indicate partially transparent pixels that can be combined with a background image to yield a composite image. (Thus, alpha is really the degree of opacity of the pixel. But most people refer to alpha as providing transparency information, not opacity information, and we continue that custom here.) Alpha channels can be included with images that have either 8 or 16 bits per sample, but not with images that have fewer than 8 bits per sample. Alpha samples are represented with the same bit depth used for the image samples. The alpha sample for each pixel is stored immediately following the grayscale or RGB samples of the pixel. Formatted and Indexed by: page 7 of 102 instructional media + magic, inc. RFC2083 Web Address: PNG: Portable Network Graphics http://sunsite.cnlab-switch.ch/ By: Boutell, et. al. The color values stored for a pixel are not affected by the alpha value assigned to the pixel. This rule is sometimes called "unassociated" or "non-premultiplied" alpha. (Another common technique is to store sample values premultiplied by the alpha fraction; in effect, such an image is already composited against a black background. PNG does not use premultiplied alpha.) Transparency control is also possible without the storage cost of a full alpha channel. In an indexed-color image, an alpha value can be defined for each palette entry. In grayscale and truecolor images, a single pixel value can be identified as being "transparent". These techniques are controlled by the tRNS ancillary chunk type. If no alpha channel nor tRNS chunk is present, all pixels in the image are to be treated as fully opaque. Viewers can support transparency control partially, or not at all. See Rationale: Non-premultiplied alpha (Section 12.8), Recommendations for Encoders: Alpha channel creation (Section 9.4), and Recommendations for Decoders: Alpha channel processing (Section 10.8). 2.5. Filtering PNG allows the image data to be filtered before it is compressed. Filtering can improve the compressibility of the data. The filter step itself does not reduce the size of the data. All PNG filters are strictly lossless. PNG defines several different filter algorithms, including "None" which indicates no filtering. The filter algorithm is specified for each scanline by a filter type byte that precedes the filtered scanline in the precompression datastream. An intelligent encoder can switch filters from one scanline to the next. The method for choosing which filter to employ is up to the encoder. See Filter Algorithms (Chapter 6) and Rationale: Filtering (Section 12.9). 2.6. Interlaced data order A PNG image can be stored in interlaced order to allow progressive display. The purpose of this feature is to allow images to "fade in" when they are being displayed on-the-fly. Interlacing slightly expands the file size on average, but it gives the user a meaningful display much more rapidly. Note that decoders are Formatted and Indexed by: page 8 of 102 instructional media + magic, inc. RFC2083 Web Address: PNG: Portable Network Graphics http://sunsite.cnlab-switch.ch/ By: Boutell, et. al. required to be able to read interlaced images, whether or not they actually perform progressive display. With interlace method 0, pixels are stored sequentially from left to right, and scanlines sequentially from top to bottom (no interlacing). Interlace method 1, known as Adam7 after its author, Adam M. Costello, consists of seven distinct passes over the image. Each pass transmits a subset of the pixels in the image. The pass in which each pixel is transmitted is defined by replicating the following 8-by-8 pattern over the entire image, starting at the upper left corner: 1 6 4 6 2 6 4 6 7 7 7 7 7 7 7 7 5 6 5 6 5 6 5 6 7 7 7 7 7 7 7 7 3 6 4 6 3 6 4 6 7 7 7 7 7 7 7 7 5 6 5 6 5 6 5 6 7 7 7 7 7 7 7 7 Within each pass, the selected pixels are transmitted left to right within a scanline, and selected scanlines sequentially from top to bottom. For example, pass 2 contains pixels 4, 12, 20, etc. of scanlines 0, 8, 16, etc. (numbering from 0,0 at the upper left corner). The last pass contains the entirety of scanlines 1, 3, 5, etc. The data within each pass is laid out as though it were a complete image of the appropriate dimensions. For example, if the complete image is 16 by 16 pixels, then pass 3 will contain two scanlines, each containing four pixels. When pixels have fewer than 8 bits, each such scanline is padded as needed to fill an integral number of bytes (see Image layout, Section 2.3). Filtering is done on this reduced image in the usual way, and a filter type byte is transmitted before each of its scanlines (see Filter Algorithms, Chapter 6). Notice that the transmission order is defined so that all the scanlines transmitted in a pass will have the same number of pixels; this is necessary for proper application of some of the filters. Caution: If the image contains fewer than five columns or fewer than five rows, some passes will be entirely empty. Encoders and decoders must handle this case correctly. In particular, filter type bytes are only associated with nonempty scanlines; no filter type bytes are present in an empty pass. Formatted and Indexed by: page 9 of 102 instructional media + magic, inc. RFC2083 Web Address: PNG: Portable Network Graphics http://sunsite.cnlab-switch.ch/ By: Boutell, et. al. See Rationale: Interlacing (Section 12.6) and Recommendations for Decoders: Progressive display (Section 10.9). 2.7. Gamma correction PNG images can specify, via the gAMA chunk, the gamma characteristic of the image with respect to the original scene. Display programs are strongly encouraged to use this information, plus information about the display device they are using and room lighting, to present the image to the viewer in a way that reproduces what the image's original author saw as closely as possible. See Gamma Tutorial (Chapter 13) if you aren't already familiar with gamma issues. Gamma correction is not applied to the alpha channel, if any. Alpha samples always represent a linear fraction of full opacity. For high-precision applications, the exact chromaticity of the RGB data in a PNG image can be specified via the cHRM chunk, allowing more accurate color matching than gamma correction alone will provide. See Color Tutorial (Chapter 14) if you aren't already familiar with color representation issues. See Rationale: Why gamma? (Section 12.7), Recommendations for Encoders: Encoder gamma handling (Section 9.2), and Recommendations for Decoders: Decoder gamma handling (Section 10.5). 2.8. Text strings A PNG file can store text associated with the image, such as an image description or copyright notice. Keywords are used to indicate what each text string represents. ISO 8859-1 (Latin-1) is the character set recommended for use in text strings [ISO-8859]. This character set is a superset of 7- bit ASCII. Character codes not defined in Latin-1 should not be used, because they have no platform-independent meaning. If a non-Latin-1 code does appear in a PNG text string, its interpretation will vary across platforms and decoders. Some systems might not even be able to display all the characters in Latin-1, but most modern systems can. Provision is also made for the storage of compressed text. See Rationale: Text strings (Section 12.10). Formatted and Indexed by: page 10 of 102 instructional media + magic, inc. RFC2083 Web Address: PNG: Portable Network Graphics http://sunsite.cnlab-switch.ch/ By: Boutell, et. al. 3. File Structure A PNG file consists of a PNG signature followed by a series of chunks. This chapter defines the signature and the basic properties of chunks. Individual chunk types are discussed in the next chapter. 3.1. PNG file signature The first eight bytes of a PNG file always contain the following (decimal) values: 137 80 78 71 13 10 26 10 This signature indicates that the remainder of the file contains a single PNG image, consisting of a series of chunks beginning with an IHDR chunk and ending with an IEND chunk. See Rationale: PNG file signature (Section 12.11). 3.2. Chunk layout Each chunk consists of four parts: Length A 4-byte unsigned integer giving the number of bytes in the chunk's data field. The length counts only the data field, not itself, the chunk type code, or the CRC. Zero is a valid length. Although encoders and decoders should treat the length as unsigned, its value must not exceed (2^31)-1 bytes. Chunk Type A 4-byte chunk type code. For convenience in description and in examining PNG files, type codes are restricted to consist of uppercase and lowercase ASCII letters (A-Z and a-z, or 65-90 and 97-122 decimal). However, encoders and decoders must treat the codes as fixed binary values, not character strings. For example, it would not be correct to represent the type code IDAT by the EBCDIC equivalents of those letters. Additional naming conventions for chunk types are discussed in the next section. Chunk Data The data bytes appropriate to the chunk type, if any. This field can be of zero length. Formatted and Indexed by: page 11 of 102 instructional media + magic, inc. RFC2083 Web Address: PNG: Portable Network Graphics http://sunsite.cnlab-switch.ch/ By: Boutell, et. al. CRC A 4-byte CRC (Cyclic Redundancy Check) calculated on the preceding bytes in the chunk, including the chunk type code and chunk data fields, but not including the length field. The CRC is always present, even for chunks containing no data. See CRC algorithm (Section 3.4). The chunk data length can be any number of bytes up to the maximum; therefore, implementors cannot assume that chunks are aligned on any boundaries larger than bytes. Chunks can appear in any order, subject to the restrictions placed on each chunk type. (One notable restriction is that IHDR must appear first and IEND must appear last; thus the IEND chunk serves as an end-of-file marker.) Multiple chunks of the same type can appear, but only if specifically permitted for that type. See Rationale: Chunk layout (Section 12.12). 3.3. Chunk naming conventions Chunk type codes are assigned so that a decoder can determine some properties of a chunk even when it does not recognize the type code. These rules are intended to allow safe, flexible extension of the PNG format, by allowing a decoder to decide what to do when it encounters an unknown chunk. The naming rules are not normally of interest when the decoder does recognize the chunk's type. Four bits of the type code, namely bit 5 (value 32) of each byte, are used to convey chunk properties. This choice means that a human can read off the assigned properties according to whether each letter of the type code is uppercase (bit 5 is 0) or lowercase (bit 5 is 1). However, decoders should test the properties of an unknown chunk by numerically testing the specified bits; testing whether a character is uppercase or lowercase is inefficient, and even incorrect if a locale-specific case definition is used. It is worth noting that the property bits are an inherent part of the chunk name, and hence are fixed for any chunk type. Thus, TEXT and Text would be unrelated chunk type codes, not the same chunk with different properties. Decoders must recognize type codes by a simple four-byte literal comparison; it is incorrect to perform case conversion on type codes. Formatted and Indexed by: page 12 of 102 instructional media + magic, inc. RFC2083 Web Address: PNG: Portable Network Graphics http://sunsite.cnlab-switch.ch/ By: Boutell, et. al. The semantics of the property bits are: Ancillary bit: bit 5 of first byte 0 (uppercase) = critical, 1 (lowercase) = ancillary. Chunks that are not strictly necessary in order to meaningfully display the contents of the file are known as "ancillary" chunks. A decoder encountering an unknown chunk in which the ancillary bit is 1 can safely ignore the chunk and proceed to display the image. The time chunk (tIME) is an example of an ancillary chunk. Chunks that are necessary for successful display of the file's contents are called "critical" chunks. A decoder encountering an unknown chunk in which the ancillary bit is 0 must indicate to the user that the image contains information it cannot safely interpret. The image header chunk (IHDR) is an example of a critical chunk. Private bit: bit 5 of second byte 0 (uppercase) = public, 1 (lowercase) = private. A public chunk is one that is part of the PNG specification or is registered in the list of PNG special-purpose public chunk types. Applications can also define private (unregistered) chunks for their own purposes. The names of private chunks must have a lowercase second letter, while public chunks will always be assigned names with uppercase second letters. Note that decoders do not need to test the private-chunk property bit, since it has no functional significance; it is simply an administrative convenience to ensure that public and private chunk names will not conflict. See Additional chunk types (Section 4.4) and Recommendations for Encoders: Use of private chunks (Section 9.8). Reserved bit: bit 5 of third byte Must be 0 (uppercase) in files conforming to this version of PNG. The significance of the case of the third letter of the chunk name is reserved for possible future expansion. At the present time all chunk names must have uppercase third letters. (Decoders should not complain about a lowercase third letter, however, as some future version of the PNG specification could define a meaning for this bit. It is sufficient to treat a chunk with a lowercase third letter in the same way as any other unknown chunk type.) Formatted and Indexed by: page 13 of 102 instructional media + magic, inc. RFC2083 Web Address: PNG: Portable Network Graphics http://sunsite.cnlab-switch.ch/ By: Boutell, et. al. Safe-to-copy bit: bit 5 of fourth byte 0 (uppercase) = unsafe to copy, 1 (lowercase) = safe to copy. This property bit is not of interest to pure decoders, but it is needed by PNG editors (programs that modify PNG files). This bit defines the proper handling of unrecognized chunks in a file that is being modified. If a chunk's safe-to-copy bit is 1, the chunk may be copied to a modified PNG file whether or not the software recognizes the chunk type, and regardless of the extent of the file modifications. If a chunk's safe-to-copy bit is 0, it indicates that the chunk depends on the image data. If the program has made any changes to critical chunks, including addition, modification, deletion, or reordering of critical chunks, then unrecognized unsafe chunks must not be copied to the output PNG file. (Of course, if the program does recognize the chunk, it can choose to output an appropriately modified version.) A PNG editor is always allowed to copy all unrecognized chunks if it has only added, deleted, modified, or reordered ancillary chunks. This implies that it is not permissible for ancillary chunks to depend on other ancillary chunks. PNG editors that do not recognize a critical chunk must report an error and refuse to process that PNG file at all. The safe/unsafe mechanism is intended for use with ancillary chunks. The safe-to-copy bit will always be 0 for critical chunks. Rules for PNG editors are discussed further in Chunk Ordering Rules (Chapter 7). For example, the hypothetical chunk type name "bLOb" has the property bits: bLOb <-- 32 bit chunk type code represented in text form |||| |||+- Safe-to-copy bit is 1 (lower case letter; bit 5 is 1) ||+-- Reserved bit is 0 (upper case letter; bit 5 is 0) |+--- Private bit is 0 (upper case letter; bit 5 is 0) +---- Ancillary bit is 1 (lower case letter; bit 5 is 1) Therefore, this name represents an ancillary, public, safe-to-copy chunk. Formatted and Indexed by: page 14 of 102 instructional media + magic, inc. RFC2083 Web Address: PNG: Portable Network Graphics http://sunsite.cnlab-switch.ch/ By: Boutell, et. al. See Rationale: Chunk naming conventions (Section 12.13). 3.4. CRC algorithm Chunk CRCs are calculated using standard CRC methods with pre and post conditioning, as defined by ISO 3309 [ISO-3309] or ITU-T V.42 [ITU-V42]. The CRC polynomial employed is x^32+x^26+x^23+x^22+x^16+x^12+x^11+x^10+x^8+x^7+x^5+x^4+x^2+x+1 The 32-bit CRC register is initialized to all 1's, and then the data from each byte is processed from the least significant bit (1) to the most significant bit (128). After all the data bytes are processed, the CRC register is inverted (its ones complement is taken). This value is transmitted (stored in the file) MSB first. For the purpose of separating into bytes and ordering, the least significant bit of the 32-bit CRC is defined to be the coefficient of the x^31 term. Practical calculation of the CRC always employs a precalculated table to greatly accelerate the computation. See Sample CRC Code (Chapter 15). 4. Chunk Specifications This chapter defines the standard types of PNG chunks. 4.1. Critical chunks All implementations must understand and successfully render the standard critical chunks. A valid PNG image must contain an IHDR chunk, one or more IDAT chunks, and an IEND chunk. 4.1.1. IHDR Image header The IHDR chunk must appear FIRST. It contains: Width: 4 bytes Height: 4 bytes Bit depth: 1 byte Color type: 1 byte Compression method: 1 byte Filter method: 1 byte Interlace method: 1 byte Formatted and Indexed by: page 15 of 102 instructional media + magic, inc. RFC2083 Web Address: PNG: Portable Network Graphics http://sunsite.cnlab-switch.ch/ By: Boutell, et. al. Width and height give the image dimensions in pixels. They are 4-byte integers. Zero is an invalid value. The maximum for each is (2^31)-1 in order to accommodate languages that have difficulty with unsigned 4-byte values. Bit depth is a single-byte integer giving the number of bits per sample or per palette index (not per pixel). Valid values are 1, 2, 4, 8, and 16, although not all values are allowed for all color types. Color type is a single-byte integer that describes the interpretation of the image data. Color type codes represent sums of the following values: 1 (palette used), 2 (color used), and 4 (alpha channel used). Valid values are 0, 2, 3, 4, and 6. Bit depth restrictions for each color type are imposed to simplify implementations and to prohibit combinations that do not compress well. Decoders must support all legal combinations of bit depth and color type. The allowed combinations are: Color Allowed Interpretation Type Bit Depths 0 1,2,4,8,16 Each pixel is a grayscale sample. 2 8,16 Each pixel is an R,G,B triple. 3 1,2,4,8 Each pixel is a palette index; a PLTE chunk must appear. 4 8,16 Each pixel is a grayscale sample, followed by an alpha sample. 6 8,16 Each pixel is an R,G,B triple, followed by an alpha sample. The sample depth is the same as the bit depth except in the case of color type 3, in which the sample depth is always 8 bits. Compression method is a single-byte integer that indicates the method used to compress the image data. At present, only compression method 0 (deflate/inflate compression with a 32K sliding window) is defined. All standard PNG images must be compressed with this scheme. The compression method field is provided for possible future expansion or proprietary variants. Decoders must check this byte and report an error if it holds Formatted and Indexed by: page 16 of 102 instructional media + magic, inc. RFC2083 Web Address: PNG: Portable Network Graphics http://sunsite.cnlab-switch.ch/ By: Boutell, et. al. an unrecognized code. See Deflate/Inflate Compression (Chapter 5) for details. Filter method is a single-byte integer that indicates the preprocessing method applied to the image data before compression. At present, only filter method 0 (adaptive filtering with five basic filter types) is defined. As with the compression method field, decoders must check this byte and report an error if it holds an unrecognized code. See Filter Algorithms (Chapter 6) for details. Interlace method is a single-byte integer that indicates the transmission order of the image data. Two values are currently defined: 0 (no interlace) or 1 (Adam7 interlace). See Interlaced data order (Section 2.6) for details. 4.1.2. PLTE Palette The PLTE chunk contains from 1 to 256 palette entries, each a three-byte series of the form: Red: 1 byte (0 = black, 255 = red) Green: 1 byte (0 = black, 255 = green) Blue: 1 byte (0 = black, 255 = blue) The number of entries is determined from the chunk length. A chunk length not divisible by 3 is an error. This chunk must appear for color type 3, and can appear for color types 2 and 6; it must not appear for color types 0 and 4. If this chunk does appear, it must precede the first IDAT chunk. There must not be more than one PLTE chunk. For color type 3 (indexed color), the PLTE chunk is required. The first entry in PLTE is referenced by pixel value 0, the second by pixel value 1, etc. The number of palette entries must not exceed the range that can be represented in the image bit depth (for example, 2^4 = 16 for a bit depth of 4). It is permissible to have fewer entries than the bit depth would allow. In that case, any out-of-range pixel value found in the image data is an error. For color types 2 and 6 (truecolor and truecolor with alpha), the PLTE chunk is optional. If present, it provides a suggested set of from 1 to 256 colors to which the truecolor image can be quantized if the viewer cannot display truecolor directly. If PLTE is not present, such a viewer will need to select colors on its own, but it is often preferable for this Formatted and Indexed by: page 17 of 102 instructional media + magic, inc. RFC2083 Web Address: PNG: Portable Network Graphics http://sunsite.cnlab-switch.ch/ By: Boutell, et. al. to be done once by the encoder. (See Recommendations for Encoders: Suggested palettes, Section 9.5.) Note that the palette uses 8 bits (1 byte) per sample regardless of the image bit depth specification. In particular, the palette is 8 bits deep even when it is a suggested quantization of a 16-bit truecolor image. There is no requirement that the palette entries all be used by the image, nor that they all be different. 4.1.3. IDAT Image data The IDAT chunk contains the actual image data. To create this data: * Begin with image scanlines represented as described in Image layout (Section 2.3); the layout and total size of this raw data are determined by the fields of IHDR. * Filter the image data according to the filtering method specified by the IHDR chunk. (Note that with filter method 0, the only one currently defined, this implies prepending a filter type byte to each scanline.) * Compress the filtered data using the compression method specified by the IHDR chunk. The IDAT chunk contains the output datastream of the compression algorithm. To read the image data, reverse this process. There can be multiple IDAT chunks; if so, they must appear consecutively with no other intervening chunks. The compressed datastream is then the concatenation of the contents of all the IDAT chunks. The encoder can divide the compressed datastream into IDAT chunks however it wishes. (Multiple IDAT chunks are allowed so that encoders can work in a fixed amount of memory; typically the chunk size will correspond to the encoder's buffer size.) It is important to emphasize that IDAT chunk boundaries have no semantic significance and can occur at any point in the compressed datastream. A PNG file in which each IDAT chunk contains only one data byte is legal, though remarkably wasteful of space. (For that matter, zero-length IDAT chunks are legal, though even more wasteful.) See Filter Algorithms (Chapter 6) and Deflate/Inflate Compression (Chapter 5) for details. Formatted and Indexed by: page 18 of 102 instructional media + magic, inc. RFC2083 Web Address: PNG: Portable Network Graphics http://sunsite.cnlab-switch.ch/ By: Boutell, et. al. 4.1.4. IEND Image trailer The IEND chunk must appear LAST. It marks the end of the PNG datastream. The chunk's data field is empty. 4.2. Ancillary chunks All ancillary chunks are optional, in the sense that encoders need not write them and decoders can ignore them. However, encoders are encouraged to write the standard ancillary chunks when the information is available, and decoders are encouraged to interpret these chunks when appropriate and feasible. The standard ancillary chunks are listed in alphabetical order. This is not necessarily the order in which they would appear in a file. 4.2.1. bKGD Background color The bKGD chunk specifies a default background color to present the image against. Note that viewers are not bound to honor this chunk; a viewer can choose to use a different background. For color type 3 (indexed color), the bKGD chunk contains: Palette index: 1 byte The value is the palette index of the color to be used as background. For color types 0 and 4 (grayscale, with or without alpha), bKGD contains: Gray: 2 bytes, range 0 .. (2^bitdepth)-1 (For consistency, 2 bytes are used regardless of the image bit depth.) The value is the gray level to be used as background. For color types 2 and 6 (truecolor, with or without alpha), bKGD contains: Red: 2 bytes, range 0 .. (2^bitdepth)-1 Green: 2 bytes, range 0 .. (2^bitdepth)-1 Blue: 2 bytes, range 0 .. (2^bitdepth)-1 (For consistency, 2 bytes per sample are used regardless of the image bit depth.) This is the RGB color to be used as background. Formatted and Indexed by: page 19 of 102 instructional media + magic, inc. RFC2083 Web Address: PNG: Portable Network Graphics http://sunsite.cnlab-switch.ch/ By: Boutell, et. al. When present, the bKGD chunk must precede the first IDAT chunk, and must follow the PLTE chunk, if any. See Recommendations for Decoders: Background color (Section 10.7). 4.2.2. cHRM Primary chromaticities and white point Applications that need device-independent specification of colors in a PNG file can use the cHRM chunk to specify the 1931 CIE x,y chromaticities of the red, green, and blue primaries used in the image, and the referenced white point. See Color Tutorial (Chapter 14) for more information. The cHRM chunk contains: White Point x: 4 bytes White Point y: 4 bytes Red x: 4 bytes Red y: 4 bytes Green x: 4 bytes Green y: 4 bytes Blue x: 4 bytes Blue y: 4 bytes Each value is encoded as a 4-byte unsigned integer, representing the x or y value times 100000. For example, a value of 0.3127 would be stored as the integer 31270. cHRM is allowed in all PNG files, although it is of little value for grayscale images. If the encoder does not know the chromaticity values, it should not write a cHRM chunk; the absence of a cHRM chunk indicates that the image's primary colors are device-dependent. If the cHRM chunk appears, it must precede the first IDAT chunk, and it must also precede the PLTE chunk if present. See Recommendations for Encoders: Encoder color handling (Section 9.3), and Recommendations for Decoders: Decoder color handling (Section 10.6). Formatted and Indexed by: page 20 of 102 instructional media + magic, inc. RFC2083 Web Address: PNG: Portable Network Graphics http://sunsite.cnlab-switch.ch/ By: Boutell, et. al. 4.2.3. gAMA Image gamma The gAMA chunk specifies the gamma of the camera (or simulated camera) that produced the image, and thus the gamma of the image with respect to the original scene. More precisely, the gAMA chunk encodes the file_gamma value, as defined in Gamma Tutorial (Chapter 13). The gAMA chunk contains: Image gamma: 4 bytes The value is encoded as a 4-byte unsigned integer, representing gamma times 100000. For example, a gamma of 0.45 would be stored as the integer 45000. If the encoder does not know the image's gamma value, it should not write a gAMA chunk; the absence of a gAMA chunk indicates that the gamma is unknown. If the gAMA chunk appears, it must precede the first IDAT chunk, and it must also precede the PLTE chunk if present. See Gamma correction (Section 2.7), Recommendations for Encoders: Encoder gamma handling (Section 9.2), and Recommendations for Decoders: Decoder gamma handling (Section 10.5). 4.2.4. hIST Image histogram The hIST chunk gives the approximate usage frequency of each color in the color palette. A histogram chunk can appear only when a palette chunk appears. If a viewer is unable to provide all the colors listed in the palette, the histogram may help it decide how to choose a subset of the colors for display. The hIST chunk contains a series of 2-byte (16 bit) unsigned integers. There must be exactly one entry for each entry in the PLTE chunk. Each entry is proportional to the fraction of pixels in the image that have that palette index; the exact scale factor is chosen by the encoder. Histogram entries are approximate, with the exception that a zero entry specifies that the corresponding palette entry is not used at all in the image. It is required that a histogram entry be nonzero if there are any pixels of that color. Formatted and Indexed by: page 21 of 102 instructional media + magic, inc. RFC2083 Web Address: PNG: Portable Network Graphics http://sunsite.cnlab-switch.ch/ By: Boutell, et. al. When the palette is a suggested quantization of a truecolor image, the histogram is necessarily approximate, since a decoder may map pixels to palette entries differently than the encoder did. In this situation, zero entries should not appear. The hIST chunk, if it appears, must follow the PLTE chunk, and must precede the first IDAT chunk. See Rationale: Palette histograms (Section 12.14), and Recommendations for Decoders: Suggested-palette and histogram usage (Section 10.10). 4.2.5. pHYs Physical pixel dimensions The pHYs chunk specifies the intended pixel size or aspect ratio for display of the image. It contains: Pixels per unit, X axis: 4 bytes (unsigned integer) Pixels per unit, Y axis: 4 bytes (unsigned integer) Unit specifier: 1 byte The following values are legal for the unit specifier: 0: unit is unknown 1: unit is the meter When the unit specifier is 0, the pHYs chunk defines pixel aspect ratio only; the actual size of the pixels remains unspecified. Conversion note: one inch is equal to exactly 0.0254 meters. If this ancillary chunk is not present, pixels are assumed to be square, and the physical size of each pixel is unknown. If present, this chunk must precede the first IDAT chunk. See Recommendations for Decoders: Pixel dimensions (Section 10.2). 4.2.6. sBIT Significant bits To simplify decoders, PNG specifies that only certain sample depths can be used, and further specifies that sample values should be scaled to the full range of possible values at the sample depth. However, the sBIT chunk is provided in order to store the original number of significant bits. This allows Formatted and Indexed by: page 22 of 102 instructional media + magic, inc. RFC2083 Web Address: PNG: Portable Network Graphics http://sunsite.cnlab-switch.ch/ By: Boutell, et. al. decoders to recover the original data losslessly even if the data had a sample depth not directly supported by PNG. We recommend that an encoder emit an sBIT chunk if it has converted the data from a lower sample depth. For color type 0 (grayscale), the sBIT chunk contains a single byte, indicating the number of bits that were significant in the source data. For color type 2 (truecolor), the sBIT chunk contains three bytes, indicating the number of bits that were significant in the source data for the red, green, and blue channels, respectively. For color type 3 (indexed color), the sBIT chunk contains three bytes, indicating the number of bits that were significant in the source data for the red, green, and blue components of the palette entries, respectively. For color type 4 (grayscale with alpha channel), the sBIT chunk contains two bytes, indicating the number of bits that were significant in the source grayscale data and the source alpha data, respectively. For color type 6 (truecolor with alpha channel), the sBIT chunk contains four bytes, indicating the number of bits that were significant in the source data for the red, green, blue and alpha channels, respectively. Each depth specified in sBIT must be greater than zero and less than or equal to the sample depth (which is 8 for indexed-color images, and the bit depth given in IHDR for other color types). A decoder need not pay attention to sBIT: the stored image is a valid PNG file of the sample depth indicated by IHDR. However, if the decoder wishes to recover the original data at its original precision, this can be done by right-shifting the stored samples (the stored palette entries, for an indexed- color image). The encoder must scale the data in such a way that the high-order bits match the original data. If the sBIT chunk appears, it must precede the first IDAT chunk, and it must also precede the PLTE chunk if present. See Recommendations for Encoders: Sample depth scaling (Section 9.1) and Recommendations for Decoders: Sample depth rescaling (Section 10.4). Formatted and Indexed by: page 23 of 102 instructional media + magic, inc. RFC2083 Web Address: PNG: Portable Network Graphics http://sunsite.cnlab-switch.ch/ By: Boutell, et. al. 4.2.7. tEXt Textual data Textual information that the encoder wishes to record with the image can be stored in tEXt chunks. Each tEXt chunk contains a keyword and a text string, in the format: Keyword: 1-79 bytes (character string) Null separator: 1 byte Text: n bytes (character string) The keyword and text string are separated by a zero byte (null character). Neither the keyword nor the text string can contain a null character. Note that the text string is not null-terminated (the length of the chunk is sufficient information to locate the ending). The keyword must be at least one character and less than 80 characters long. The text string can be of any length from zero bytes up to the maximum permissible chunk size less the length of the keyword and separator. Any number of tEXt chunks can appear, and more than one with the same keyword is permissible. The keyword indicates the type of information represented by the text string. The following keywords are predefined and should be used where appropriate: Title Short (one line) title or caption for image Author Name of image's creator Description Description of image (possibly long) Copyright Copyright notice Creation Time Time of original image creation Software Software used to create the image Disclaimer Legal disclaimer Warning Warning of nature of content Source Device used to create the image Comment Miscellaneous comment; conversion from GIF comment For the Creation Time keyword, the date format defined in section 5.2.14 of RFC 1123 is suggested, but not required [RFC-1123]. Decoders should allow for free-format text associated with this or any other keyword. Other keywords may be invented for other purposes. Keywords of general interest can be registered with the maintainers of the PNG specification. However, it is also permitted to use private unregistered keywords. (Private keywords should be Formatted and Indexed by: page 24 of 102 instructional media + magic, inc. RFC2083 Web Address: PNG: Portable Network Graphics http://sunsite.cnlab-switch.ch/ By: Boutell, et. al. reasonably self-explanatory, in order to minimize the chance that the same keyword will be used for incompatible purposes by different people.) Both keyword and text are interpreted according to the ISO 8859-1 (Latin-1) character set [ISO-8859]. The text string can contain any Latin-1 character. Newlines in the text string should be represented by a single linefeed character (decimal 10); use of other control characters in the text is discouraged. Keywords must contain only printable Latin-1 characters and spaces; that is, only character codes 32-126 and 161-255 decimal are allowed. To reduce the chances for human misreading of a keyword, leading and trailing spaces are forbidden, as are consecutive spaces. Note also that the non- breaking space (code 160) is not permitted in keywords, since it is visually indistinguishable from an ordinary space. Keywords must be spelled exactly as registered, so that decoders can use simple literal comparisons when looking for particular keywords. In particular, keywords are considered case-sensitive. See Recommendations for Encoders: Text chunk processing (Section 9.7) and Recommendations for Decoders: Text chunk processing (Section 10.11). 4.2.8. tIME Image last-modification time The tIME chunk gives the time of the last image modification (not the time of initial image creation). It contains: Year: 2 bytes (complete; for example, 1995, not 95) Month: 1 byte (1-12) Day: 1 byte (1-31) Hour: 1 byte (0-23) Minute: 1 byte (0-59) Second: 1 byte (0-60) (yes, 60, for leap seconds; not 61, a common error) Universal Time (UTC, also called GMT) should be specified rather than local time. Formatted and Indexed by: page 25 of 102 instructional media + magic, inc. RFC2083 Web Address: PNG: Portable Network Graphics http://sunsite.cnlab-switch.ch/ By: Boutell, et. al. The tIME chunk is intended for use as an automatically-applied time stamp that is updated whenever the image data is changed. It is recommended that tIME not be changed by PNG editors that do not change the image data. See also the Creation Time tEXt keyword, which can be used for a user-supplied time. 4.2.9. tRNS Transparency The tRNS chunk specifies that the image uses simple transparency: either alpha values associated with palette entries (for indexed-color images) or a single transparent color (for grayscale and truecolor images). Although simple transparency is not as elegant as the full alpha channel, it requires less storage space and is sufficient for many common cases. For color type 3 (indexed color), the tRNS chunk contains a series of one-byte alpha values, corresponding to entries in the PLTE chunk: Alpha for palette index 0: 1 byte Alpha for palette index 1: 1 byte ... etc ... Each entry indicates that pixels of the corresponding palette index must be treated as having the specified alpha value. Alpha values have the same interpretation as in an 8-bit full alpha channel: 0 is fully transparent, 255 is fully opaque, regardless of image bit depth. The tRNS chunk must not contain more alpha values than there are palette entries, but tRNS can contain fewer values than there are palette entries. In this case, the alpha value for all remaining palette entries is assumed to be 255. In the common case in which only palette index 0 need be made transparent, only a one-byte tRNS chunk is needed. For color type 0 (grayscale), the tRNS chunk contains a single gray level value, stored in the format: Gray: 2 bytes, range 0 .. (2^bitdepth)-1 (For consistency, 2 bytes are used regardless of the image bit depth.) Pixels of the specified gray level are to be treated as transparent (equivalent to alpha value 0); all other pixels are to be treated as fully opaque (alpha value (2^bitdepth)-1). Formatted and Indexed by: page 26 of 102 instructional media + magic, inc. RFC2083 Web Address: PNG: Portable Network Graphics http://sunsite.cnlab-switch.ch/ By: Boutell, et. al. For color type 2 (truecolor), the tRNS chunk contains a single RGB color value, stored in the format: Red: 2 bytes, range 0 .. (2^bitdepth)-1 Green: 2 bytes, range 0 .. (2^bitdepth)-1 Blue: 2 bytes, range 0 .. (2^bitdepth)-1 (For consistency, 2 bytes per sample are used regardless of the image bit depth.) Pixels of the specified color value are to be treated as transparent (equivalent to alpha value 0); all other pixels are to be treated as fully opaque (alpha value (2^bitdepth)-1). tRNS is prohibited for color types 4 and 6, since a full alpha channel is already present in those cases. Note: when dealing with 16-bit grayscale or truecolor data, it is important to compare both bytes of the sample values to determine whether a pixel is transparent. Although decoders may drop the low-order byte of the samples for display, this must not occur until after the data has been tested for transparency. For example, if the grayscale level 0x0001 is specified to be transparent, it would be incorrect to compare only the high-order byte and decide that 0x0002 is also transparent. When present, the tRNS chunk must precede the first IDAT chunk, and must follow the PLTE chunk, if any. 4.2.10. zTXt Compressed textual data The zTXt chunk contains textual data, just as tEXt does; however, zTXt takes advantage of compression. zTXt and tEXt chunks are semantically equivalent, but zTXt is recommended for storing large blocks of text. A zTXt chunk contains: Keyword: 1-79 bytes (character string) Null separator: 1 byte Compression method: 1 byte Compressed text: n bytes The keyword and null separator are exactly the same as in the tEXt chunk. Note that the keyword is not compressed. The compression method byte identifies the compression method used in this zTXt chunk. The only value presently defined for it is 0 (deflate/inflate compression). The compression method byte is Formatted and Indexed by: page 27 of 102 instructional media + magic, inc. RFC2083 Web Address: PNG: Portable Network Graphics http://sunsite.cnlab-switch.ch/ By: Boutell, et. al. followed by a compressed datastream that makes up the remainder of the chunk. For compression method 0, this datastream adheres to the zlib datastream format (see Deflate/Inflate Compression, Chapter 5). Decompression of this datastream yields Latin-1 text that is identical to the text that would be stored in an equivalent tEXt chunk. Any number of zTXt and tEXt chunks can appear in the same file. See the preceding definition of the tEXt chunk for the predefined keywords and the recommended format of the text. See Recommendations for Encoders: Text chunk processing (Section 9.7), and Recommendations for Decoders: Text chunk processing (Section 10.11). 4.3. Summary of standard chunks This table summarizes some properties of the standard chunk types. Critical chunks (must appear in this order, except PLTE is optional): Name Multiple Ordering constraints OK? IHDR No Must be first PLTE No Before IDAT IDAT Yes Multiple IDATs must be consecutive IEND No Must be last Ancillary chunks (need not appear in this order): Name Multiple Ordering constraints OK? cHRM No Before PLTE and IDAT gAMA No Before PLTE and IDAT sBIT No Before PLTE and IDAT bKGD No After PLTE; before IDAT hIST No After PLTE; before IDAT tRNS No After PLTE; before IDAT pHYs No Before IDAT tIME No None tEXt Yes None zTXt Yes None Formatted and Indexed by: page 28 of 102 instructional media + magic, inc. RFC2083 Web Address: PNG: Portable Network Graphics http://sunsite.cnlab-switch.ch/ By: Boutell, et. al. Standard keywords for tEXt and zTXt chunks: Title Short (one line) title or caption for image Author Name of image's creator Description Description of image (possibly long) Copyright Copyright notice Creation Time Time of original image creation Software Software used to create the image Disclaimer Legal disclaimer Warning Warning of nature of content Source Device used to create the image Comment Miscellaneous comment; conversion from GIF comment 4.4. Additional chunk types Additional public PNG chunk types are defined in the document "PNG Special-Purpose Public Chunks" [PNG-EXTENSIONS]. Chunks described there are expected to be less widely supported than those defined in this specification. However, application authors are encouraged to use those chunk types whenever appropriate for their applications. Additional chunk types can be proposed for inclusion in that list by contacting the PNG specification maintainers at [email protected] or at [email protected]. New public chunks will only be registered if they are of use to others and do not violate the design philosophy of PNG. Chunk registration is not automatic, although it is the intent of the authors that it be straightforward when a new chunk of potentially wide application is needed. Note that the creation of new critical chunk types is discouraged unless absolutely necessary. Applications can also use private chunk types to carry data that is not of interest to other applications. See Recommendations for Encoders: Use of private chunks (Section 9.8). Decoders must be prepared to encounter unrecognized public or private chunk type codes. Unrecognized chunk types must be handled as described in Chunk naming conventions (Section 3.3). 5. Deflate/Inflate Compression PNG compression method 0 (the only compression method presently defined for PNG) specifies deflate/inflate compression with a 32K sliding window. Deflate compression is an LZ77 derivative used in zip, gzip, pkzip and related programs. Extensive research has been done supporting its patent-free status. Portable C implementations are freely available. Formatted and Indexed by: page 29 of 102 instructional media + magic, inc. RFC2083 Web Address: PNG: Portable Network Graphics http://sunsite.cnlab-switch.ch/ By: Boutell, et. al. Deflate-compressed datastreams within PNG are stored in the "zlib" format, which has the structure: Compression method/flags code: 1 byte Additional flags/check bits: 1 byte Compressed data blocks: n bytes Check value: 4 bytes Further details on this format are given in the zlib specification [RFC-1950]. For PNG compression method 0, the zlib compression method/flags code must specify method code 8 ("deflate" compression) and an LZ77 window size of not more than 32K. Note that the zlib compression method number is not the same as the PNG compression method number. The additional flags must not specify a preset dictionary. The compressed data within the zlib datastream is stored as a series of blocks, each of which can represent raw (uncompressed) data, LZ77-compressed data encoded with fixed Huffman codes, or LZ77- compressed data encoded with custom Huffman codes. A marker bit in the final block identifies it as the last block, allowing the decoder to recognize the end of the compressed datastream. Further details on the compression algorithm and the encoding are given in the deflate specification [RFC-1951]. The check value stored at the end of the zlib datastream is calculated on the uncompressed data represented by the datastream. Note that the algorithm used is not the same as the CRC calculation used for PNG chunk check values. The zlib check value is useful mainly as a cross-check that the deflate and inflate algorithms are implemented correctly. Verifying the chunk CRCs provides adequate confidence that the PNG file has been transmitted undamaged. In a PNG file, the concatenation of the contents of all the IDAT chunks makes up a zlib datastream as specified above. This datastream decompresses to filtered image data as described elsewhere in this document. It is important to emphasize that the boundaries between IDAT chunks are arbitrary and can fall anywhere in the zlib datastream. There is not necessarily any correlation between IDAT chunk boundaries and deflate block boundaries or any other feature of the zlib data. For example, it is entirely possible for the terminating zlib check value to be split across IDAT chunks. Formatted and Indexed by: page 30 of 102 instructional media + magic, inc. RFC2083 Web Address: PNG: Portable Network Graphics http://sunsite.cnlab-switch.ch/ By: Boutell, et. al. In the same vein, there is no required correlation between the structure of the image data (i.e., scanline boundaries) and deflate block boundaries or IDAT chunk boundaries. The complete image data is represented by a single zlib datastream that is stored in some number of IDAT chunks; a decoder that assumes any more than this is incorrect. (Of course, some encoder implementations may emit files in which some of these structures are indeed related. But decoders cannot rely on this.) PNG also uses zlib datastreams in zTXt chunks. In a zTXt chunk, the remainder of the chunk following the compression method byte is a zlib datastream as specified above. This datastream decompresses to the user-readable text described by the chunk's keyword. Unlike the image data, such datastreams are not split across chunks; each zTXt chunk contains an independent zlib datastream. Additional documentation and portable C code for deflate and inflate are available from the Info-ZIP archives at 6. Filter Algorithms This chapter describes the filter algorithms that can be applied before compression. The purpose of these filters is to prepare the image data for optimum compression. 6.1. Filter types PNG filter method 0 defines five basic filter types: Type Name 0 None 1 Sub 2 Up 3 Average 4 Paeth (Note that filter method 0 in IHDR specifies exactly this set of five filter types. If the set of filter types is ever extended, a different filter method number will be assigned to the extended set, so that decoders need not decompress the data to discover that it contains unsupported filter types.) The encoder can choose which of these filter algorithms to apply on a scanline-by-scanline basis. In the image data sent to the compression step, each scanline is preceded by a filter type byte that specifies the filter algorithm used for that scanline. Formatted and Indexed by: page 31 of 102 instructional media + magic, inc. RFC2083 Web Address: PNG: Portable Network Graphics http://sunsite.cnlab-switch.ch/ By: Boutell, et. al. Filtering algorithms are applied to bytes, not to pixels, regardless of the bit depth or color type of the image. The filtering algorithms work on the byte sequence formed by a scanline that has been represented as described in Image layout (Section 2.3). If the image includes an alpha channel, the alpha data is filtered in the same way as the image data. When the image is interlaced, each pass of the interlace pattern is treated as an independent image for filtering purposes. The filters work on the byte sequences formed by the pixels actually transmitted during a pass, and the "previous scanline" is the one previously transmitted in the same pass, not the one adjacent in the complete image. Note that the subimage transmitted in any one pass is always rectangular, but is of smaller width and/or height than the complete image. Filtering is not applied when this subimage is empty. For all filters, the bytes "to the left of" the first pixel in a scanline must be treated as being zero. For filters that refer to the prior scanline, the entire prior scanline must be treated as being zeroes for the first scanline of an image (or of a pass of an interlaced image). To reverse the effect of a filter, the decoder must use the decoded values of the prior pixel on the same line, the pixel immediately above the current pixel on the prior line, and the pixel just to the left of the pixel above. This implies that at least one scanline's worth of image data will have to be stored by the decoder at all times. Even though some filter types do not refer to the prior scanline, the decoder will always need to store each scanline as it is decoded, since the next scanline might use a filter that refers to it. PNG imposes no restriction on which filter types can be applied to an image. However, the filters are not equally effective on all types of data. See Recommendations for Encoders: Filter selection (Section 9.6). See also Rationale: Filtering (Section 12.9). 6.2. Filter type 0: None With the None filter, the scanline is transmitted unmodified; it is only necessary to insert a filter type byte before the data. Formatted and Indexed by: page 32 of 102 instructional media + magic, inc. RFC2083 Web Address: PNG: Portable Network Graphics http://sunsite.cnlab-switch.ch/ By: Boutell, et. al. 6.3. Filter type 1: Sub The Sub filter transmits the difference between each byte and the value of the corresponding byte of the prior pixel. To compute the Sub filter, apply the following formula to each byte of the scanline: Sub(x) = Raw(x) - Raw(x-bpp) where x ranges from zero to the number of bytes representing the scanline minus one, Raw(x) refers to the raw data byte at that byte position in the scanline, and bpp is defined as the number of bytes per complete pixel, rounding up to one. For example, for color type 2 with a bit depth of 16, bpp is equal to 6 (three samples, two bytes per sample); for color type 0 with a bit depth of 2, bpp is equal to 1 (rounding up); for color type 4 with a bit depth of 16, bpp is equal to 4 (two-byte grayscale sample, plus two-byte alpha sample). Note this computation is done for each byte, regardless of bit depth. In a 16-bit image, each MSB is predicted from the preceding MSB and each LSB from the preceding LSB, because of the way that bpp is defined. Unsigned arithmetic modulo 256 is used, so that both the inputs and outputs fit into bytes. The sequence of Sub values is transmitted as the filtered scanline. For all x < 0, assume Raw(x) = 0. To reverse the effect of the Sub filter after decompression, output the following value: Sub(x) + Raw(x-bpp) (computed mod 256), where Raw refers to the bytes already decoded. 6.4. Filter type 2: Up The Up filter is just like the Sub filter except that the pixel immediately above the current pixel, rather than just to its left, is used as the predictor. To compute the Up filter, apply the following formula to each byte of the scanline: Up(x) = Raw(x) - Prior(x) Formatted and Indexed by: page 33 of 102 instructional media + magic, inc. RFC2083 Web Address: PNG: Portable Network Graphics http://sunsite.cnlab-switch.ch/ By: Boutell, et. al. where x ranges from zero to the number of bytes representing the scanline minus one, Raw(x) refers to the raw data byte at that byte position in the scanline, and Prior(x) refers to the unfiltered bytes of the prior scanline. Note this is done for each byte, regardless of bit depth. Unsigned arithmetic modulo 256 is used, so that both the inputs and outputs fit into bytes. The sequence of Up values is transmitted as the filtered scanline. On the first scanline of an image (or of a pass of an interlaced image), assume Prior(x) = 0 for all x. To reverse the effect of the Up filter after decompression, output the following value: Up(x) + Prior(x) (computed mod 256), where Prior refers to the decoded bytes of the prior scanline. 6.5. Filter type 3: Average The Average filter uses the average of the two neighboring pixels (left and above) to predict the value of a pixel. To compute the Average filter, apply the following formula to each byte of the scanline: Average(x) = Raw(x) - floor((Raw(x-bpp)+Prior(x))/2) where x ranges from zero to the number of bytes representing the scanline minus one, Raw(x) refers to the raw data byte at that byte position in the scanline, Prior(x) refers to the unfiltered bytes of the prior scanline, and bpp is defined as for the Sub filter. Note this is done for each byte, regardless of bit depth. The sequence of Average values is transmitted as the filtered scanline. The subtraction of the predicted value from the raw byte must be done modulo 256, so that both the inputs and outputs fit into bytes. However, the sum Raw(x-bpp)+Prior(x) must be formed without overflow (using at least nine-bit arithmetic). floor() indicates that the result of the division is rounded to the next lower integer if fractional; in other words, it is an integer division or right shift operation. Formatted and Indexed by: page 34 of 102 instructional media + magic, inc. RFC2083 Web Address: PNG: Portable Network Graphics http://sunsite.cnlab-switch.ch/ By: Boutell, et. al. For all x < 0, assume Raw(x) = 0. On the first scanline of an image (or of a pass of an interlaced image), assume Prior(x) = 0 for all x. To reverse the effect of the Average filter after decompression, output the following value: Average(x) + floor((Raw(x-bpp)+Prior(x))/2) where the result is computed mod 256, but the prediction is calculated in the same way as for encoding. Raw refers to the bytes already decoded, and Prior refers to the decoded bytes of the prior scanline. 6.6. Filter type 4: Paeth The Paeth filter computes a simple linear function of the three neighboring pixels (left, above, upper left), then chooses as predictor the neighboring pixel closest to the computed value. This technique is due to Alan W. Paeth [PAETH]. To compute the Paeth filter, apply the following formula to each byte of the scanline: Paeth(x) = Raw(x) - PaethPredictor(Raw(x-bpp), Prior(x), Prior(x-bpp)) where x ranges from zero to the number of bytes representing the scanline minus one, Raw(x) refers to the raw data byte at that byte position in the scanline, Prior(x) refers to the unfiltered bytes of the prior scanline, and bpp is defined as for the Sub filter. Note this is done for each byte, regardless of bit depth. Unsigned arithmetic modulo 256 is used, so that both the inputs and outputs fit into bytes. The sequence of Paeth values is transmitted as the filtered scanline. Formatted and Indexed by: page 35 of 102 instructional media + magic, inc. RFC2083 Web Address: PNG: Portable Network Graphics http://sunsite.cnlab-switch.ch/ By: Boutell, et. al. The PaethPredictor function is defined by the following pseudocode: function PaethPredictor (a, b, c) begin ; a = left, b = above, c = upper left p := a + b - c ; initial estimate pa := abs(p - a) ; distances to a, b, c pb := abs(p - b) pc := abs(p - c) ; return nearest of a,b,c, ; breaking ties in order a,b,c. if pa <= pb AND pa <= pc then return a else if pb <= pc then return b else return c end The calculations within the PaethPredictor function must be performed exactly, without overflow. Arithmetic modulo 256 is to be used only for the final step of subtracting the function result from the target byte value. Note that the order in which ties are broken is critical and must not be altered. The tie break order is: pixel to the left, pixel above, pixel to the upper left. (This order differs from that given in Paeth's article.) For all x < 0, assume Raw(x) = 0 and Prior(x) = 0. On the first scanline of an image (or of a pass of an interlaced image), assume Prior(x) = 0 for all x. To reverse the effect of the Paeth filter after decompression, output the following value: Paeth(x) + PaethPredictor(Raw(x-bpp), Prior(x), Prior(x-bpp)) (computed mod 256), where Raw and Prior refer to bytes already decoded. Exactly the same PaethPredictor function is used by both encoder and decoder. 7. Chunk Ordering Rules To allow new chunk types to be added to PNG, it is necessary to establish rules about the ordering requirements for all chunk types. Otherwise a PNG editing program cannot know what to do when it encounters an unknown chunk. Formatted and Indexed by: page 36 of 102 instructional media + magic, inc. RFC2083 Web Address: PNG: Portable Network Graphics http://sunsite.cnlab-switch.ch/ By: Boutell, et. al. We define a "PNG editor" as a program that modifies a PNG file and wishes to preserve as much as possible of the ancillary information in the file. Two examples of PNG editors are a program that adds or modifies text chunks, and a program that adds a suggested palette to a truecolor PNG file. Ordinary image editors are not PNG editors in this sense, because they usually discard all unrecognized information while reading in an image. (Note: we strongly encourage programs handling PNG files to preserve ancillary information whenever possible.) As an example of possible problems, consider a hypothetical new ancillary chunk type that is safe-to-copy and is required to appear after PLTE if PLTE is present. If our program to add a suggested PLTE does not recognize this new chunk, it may insert PLTE in the wrong place, namely after the new chunk. We could prevent such problems by requiring PNG editors to discard all unknown chunks, but that is a very unattractive solution. Instead, PNG requires ancillary chunks not to have ordering restrictions like this. To prevent this type of problem while allowing for future extension, we put some constraints on both the behavior of PNG editors and the allowed ordering requirements for chunks. 7.1. Behavior of PNG editors The rules for PNG editors are: * When copying an unknown unsafe-to-copy ancillary chunk, a PNG editor must not move the chunk relative to any critical chunk. It can relocate the chunk freely relative to other ancillary chunks that occur between the same pair of critical chunks. (This is well defined since the editor must not add, delete, modify, or reorder critical chunks if it is preserving unknown unsafe-to-copy chunks.) * When copying an unknown safe-to-copy ancillary chunk, a PNG editor must not move the chunk from before IDAT to after IDAT or vice versa. (This is well defined because IDAT is always present.) Any other reordering is permitted. * When copying a known ancillary chunk type, an editor need only honor the specific chunk ordering rules that exist for that chunk type. However, it can always choose to apply the above general rules instead. * PNG editors must give up on encountering an unknown critical chunk type, because there is no way to be certain that a valid file will result from modifying a file containing such a chunk. (Note that simply discarding the chunk is not good enough, because it might have unknown implications for the interpretation of other chunks.) Formatted and Indexed by: page 37 of 102 instructional media + magic, inc. RFC2083 Web Address: PNG: Portable Network Graphics http://sunsite.cnlab-switch.ch/ By: Boutell, et. al. These rules are expressed in terms of copying chunks from an input file to an output file, but they apply in the obvious way if a PNG file is modified in place. See also Chunk naming conventions (Section 3.3). 7.2. Ordering of ancillary chunks The ordering rules for an ancillary chunk type cannot be any stricter than this: * Unsafe-to-copy chunks can have ordering requirements relative to critical chunks. * Safe-to-copy chunks can have ordering requirements relative to IDAT. The actual ordering rules for any particular ancillary chunk type may be weaker. See for example the ordering rules for the standard ancillary chunk types (Summary of standard chunks, Section 4.3). Decoders must not assume more about the positioning of any ancillary chunk than is specified by the chunk ordering rules. In particular, it is never valid to assume that a specific ancillary chunk type occurs with any particular positioning relative to other ancillary chunks. (For example, it is unsafe to assume that your private ancillary chunk occurs immediately before IEND. Even if your application always writes it there, a PNG editor might have inserted some other ancillary chunk after it. But you can safely assume that your chunk will remain somewhere between IDAT and IEND.) 7.3. Ordering of critical chunks Critical chunks can have arbitrary ordering requirements, because PNG editors are required to give up if they encounter unknown critical chunks. For example, IHDR has the special ordering rule that it must always appear first. A PNG editor, or indeed any PNG-writing program, must know and follow the ordering rules for any critical chunk type that it can emit. Formatted and Indexed by: page 38 of 102 instructional media + magic, inc. RFC2083 Web Address: PNG: Portable Network Graphics http://sunsite.cnlab-switch.ch/ By: Boutell, et. al. 8. Miscellaneous Topics 8.1. File name extension On systems where file names customarily include an extension signifying file type, the extension ".png" is recommended for PNG files. Lower case ".png" is preferred if file names are case- sensitive. 8.2. Internet media type The Internet Assigned Numbers Authority (IANA) has registered "image/png" as the Internet Media Type for PNG [RFC-2045, RFC- 2048]. For robustness, decoders may choose to also support the interim media type "image/x-png" which was in use before registration was complete. 8.3. Macintosh file layout In the Apple Macintosh system, the following conventions are recommended: * The four-byte file type code for PNG files is "PNGf". (This code has been registered with Apple for PNG files.) The creator code will vary depending on the creating application. * The contents of the data fork must be a PNG file exactly as described in the rest of this specification. * The contents of the resource fork are unspecified. It may be empty or may contain application-dependent resources. * When transferring a Macintosh PNG file to a non-Macintosh system, only the data fork should be transferred. 8.4. Multiple-image extension PNG itself is strictly a single-image format. However, it may be necessary to store multiple images within one file; for example, this is needed to convert some GIF files. In the future, a multiple-image format based on PNG may be defined. Such a format will be considered a separate file format and will have a different signature. PNG-supporting applications may or may not choose to support the multiple-image format. See Rationale: Why not these features? (Section 12.3). Formatted and Indexed by: page 39 of 102 instructional media + magic, inc. RFC2083 Web Address: PNG: Portable Network Graphics http://sunsite.cnlab-switch.ch/ By: Boutell, et. al. 8.5. Security considerations A PNG file or datastream is composed of a collection of explicitly typed "chunks". Chunks whose contents are defined by the specification could actually contain anything, including malicious code. But there is no known risk that such malicious code could be executed on the recipient's computer as a result of decoding the PNG image. The possible security risks associated with future chunk types cannot be specified at this time. Security issues will be considered when evaluating chunks proposed for registration as public chunks. There is no additional security risk associated with unknown or unimplemented chunk types, because such chunks will be ignored, or at most be copied into another PNG file. The tEXt and zTXt chunks contain data that is meant to be displayed as plain text. It is possible that if the decoder displays such text without filtering out control characters, especially the ESC (escape) character, certain systems or terminals could behave in undesirable and insecure ways. We recommend that decoders filter out control characters to avoid this risk; see Recommendations for Decoders: Text chunk processing (Section 10.11). Because every chunk's length is available at its beginning, and because every chunk has a CRC trailer, there is a very robust defense against corrupted data and against fraudulent chunks that attempt to overflow the decoder's buffers. Also, the PNG signature bytes provide early detection of common file transmission errors. A decoder that fails to check CRCs could be subject to data corruption. The only likely consequence of such corruption is incorrectly displayed pixels within the image. Worse things might happen if the CRC of the IHDR chunk is not checked and the width or height fields are corrupted. See Recommendations for Decoders: Error checking (Section 10.1). A poorly written decoder might be subject to buffer overflow, because chunks can be extremely large, up to (2^31)-1 bytes long. But properly written decoders will handle large chunks without difficulty. Formatted and Indexed by: page 40 of 102 instructional media + magic, inc. RFC2083 Web Address: PNG: Portable Network Graphics http://sunsite.cnlab-switch.ch/ By: Boutell, et. al. 9. Recommendations for Encoders This chapter gives some recommendations for encoder behavior. The only absolute requirement on a PNG encoder is that it produce files that conform to the format specified in the preceding chapters. However, best results will usually be achieved by following these recommendations. 9.1. Sample depth scaling When encoding input samples that have a sample depth that cannot be directly represented in PNG, the encoder must scale the samples up to a sample depth that is allowed by PNG. The most accurate scaling method is the linear equation output = ROUND(input * MAXOUTSAMPLE / MAXINSAMPLE) where the input samples range from 0 to MAXINSAMPLE and the outputs range from 0 to MAXOUTSAMPLE (which is (2^sampledepth)-1). A close approximation to the linear scaling method can be achieved by "left bit replication", which is shifting the valid bits to begin in the most significant bit and repeating the most significant bits into the open bits. This method is often faster to compute than linear scaling. As an example, assume that 5-bit samples are being scaled up to 8 bits. If the source sample value is 27 (in the range from 0-31), then the original bits are: 4 3 2 1 0 ------1 1 0 1 1 Left bit replication gives a value of 222: 7 6 5 4 3 2 1 0 ------1 1 0 1 1 1 1 0 |======| |===| | Leftmost Bits Repeated to Fill Open Bits | Original Bits which matches the value computed by the linear equation. Left bit replication usually gives the same value as linear scaling, and is never off by more than one. Formatted and Indexed by: page 41 of 102 instructional media + magic, inc. RFC2083 Web Address: PNG: Portable Network Graphics http://sunsite.cnlab-switch.ch/ By: Boutell, et. al. A distinctly less accurate approximation is obtained by simply left-shifting the input value and filling the low order bits with zeroes. This scheme cannot reproduce white exactly, since it does not generate an all-ones maximum value; the net effect is to darken the image slightly. This method is not recommended in general, but it does have the effect of improving compression, particularly when dealing with greater-than-eight-bit sample depths. Since the relative error introduced by zero-fill scaling is small at high sample depths, some encoders may choose to use it. Zero-fill must not be used for alpha channel data, however, since many decoders will special-case alpha values of all zeroes and all ones. It is important to represent both those values exactly in the scaled data. When the encoder writes an sBIT chunk, it is required to do the scaling in such a way that the high-order bits of the stored samples match the original data. That is, if the sBIT chunk specifies a sample depth of S, the high-order S bits of the stored data must agree with the original S-bit data values. This allows decoders to recover the original data by shifting right. The added low-order bits are not constrained. Note that all the above scaling methods meet this restriction. When scaling up source data, it is recommended that the low-order bits be filled consistently for all samples; that is, the same source value should generate the same sample value at any pixel position. This improves compression by reducing the number of distinct sample values. However, this is not a requirement, and some encoders may choose not to follow it. For example, an encoder might instead dither the low-order bits, improving displayed image quality at the price of increasing file size. In some applications the original source data may have a range that is not a power of 2. The linear scaling equation still works for this case, although the shifting methods do not. It is recommended that an sBIT chunk not be written for such images, since sBIT suggests that the original data range was exactly 0..2^S-1. 9.2. Encoder gamma handling See Gamma Tutorial (Chapter 13) if you aren't already familiar with gamma issues. Proper handling of gamma encoding and the gAMA chunk in an encoder depends on the prior history of the sample values and on whether these values have already been quantized to integers. Formatted and Indexed by: page 42 of 102 instructional media + magic, inc. RFC2083 Web Address: PNG: Portable Network Graphics http://sunsite.cnlab-switch.ch/ By: Boutell, et. al. If the encoder has access to sample intensity values in floating- point or high-precision integer form (perhaps from a computer image renderer), then it is recommended that the encoder perform its own gamma encoding before quantizing the data to integer values for storage in the file. Applying gamma encoding at this stage results in images with fewer banding artifacts at a given sample depth, or allows smaller samples while retaining the same visual quality. A linear intensity level, expressed as a floating-point value in the range 0 to 1, can be converted to a gamma-encoded sample value by sample = ROUND((intensity ^ encoder_gamma) * MAXSAMPLE) The file_gamma value to be written in the PNG gAMA chunk is the same as encoder_gamma in this equation, since we are assuming the initial intensity value is linear (in effect, camera_gamma is 1.0). If the image is being written to a file only, the encoder_gamma value can be selected somewhat arbitrarily. Values of 0.45 or 0.5 are generally good choices because they are common in video systems, and so most PNG decoders should do a good job displaying such images. Some image renderers may simultaneously write the image to a PNG file and display it on-screen. The displayed pixels should be gamma corrected for the display system and viewing conditions in use, so that the user sees a proper representation of the intended scene. An appropriate gamma correction value is screen_gc = viewing_gamma / display_gamma If the renderer wants to write the same gamma-corrected sample values to the PNG file, avoiding a separate gamma-encoding step for file output, then this screen_gc value should be written in the gAMA chunk. This will allow a PNG decoder to reproduce what the file's originator saw on screen during rendering (provided the decoder properly supports arbitrary values in a gAMA chunk). However, it is equally reasonable for a renderer to apply gamma correction for screen display using a gamma appropriate to the viewing conditions, and to separately gamma-encode the sample values for file storage using a standard value of gamma such as 0.5. In fact, this is preferable, since some PNG decoders may not accurately display images with unusual gAMA values. Formatted and Indexed by: page 43 of 102 instructional media + magic, inc. RFC2083 Web Address: PNG: Portable Network Graphics http://sunsite.cnlab-switch.ch/ By: Boutell, et. al. Computer graphics renderers often do not perform gamma encoding, instead making sample values directly proportional to scene light intensity. If the PNG encoder receives sample values that have already been quantized into linear-light integer values, there is no point in doing gamma encoding on them; that would just result in further loss of information. The encoder should just write the sample values to the PNG file. This "linear" sample encoding is equivalent to gamma encoding with a gamma of 1.0, so graphics programs that produce linear samples should always emit a gAMA chunk specifying a gamma of 1.0. When the sample values come directly from a piece of hardware, the correct gAMA value is determined by the gamma characteristic of the hardware. In the case of video digitizers ("frame grabbers"), gAMA should be 0.45 or 0.5 for NTSC (possibly less for PAL or SECAM) since video camera transfer functions are standardized. Image scanners are less predictable. Their output samples may be linear (gamma 1.0) since CCD sensors themselves are linear, or the scanner hardware may have already applied gamma correction designed to compensate for dot gain in subsequent printing (gamma of about 0.57), or the scanner may have corrected the samples for display on a CRT (gamma of 0.4-0.5). You will need to refer to the scanner's manual, or even scan a calibrated gray wedge, to determine what a particular scanner does. File format converters generally should not attempt to convert supplied images to a different gamma. Store the data in the PNG file without conversion, and record the source gamma if it is known. Gamma alteration at file conversion time causes re- quantization of the set of intensity levels that are represented, introducing further roundoff error with little benefit. It's almost always better to just copy the sample values intact from the input to the output file. In some cases, the supplied image may be in an image format (e.g., TIFF) that can describe the gamma characteristic of the image. In such cases, a file format converter is strongly encouraged to write a PNG gAMA chunk that corresponds to the known gamma of the source image. Note that some file formats specify the gamma of the display system, not the camera. If the input file's gamma value is greater than 1.0, it is almost certainly a display system gamma, and you should use its reciprocal for the PNG gAMA. Formatted and Indexed by: page 44 of 102 instructional media + magic, inc. RFC2083 Web Address: PNG: Portable Network Graphics http://sunsite.cnlab-switch.ch/ By: Boutell, et. al. If the encoder or file format converter does not know how an image was originally created, but does know that the image has been displayed satisfactorily on a display with gamma display_gamma under lighting conditions where a particular viewing_gamma is appropriate, then the image can be marked as having the file_gamma: file_gamma = viewing_gamma / display_gamma This will allow viewers of the PNG file to see the same image that the person running the file format converter saw. Although this may not be precisely the correct value of the image gamma, it's better to write a gAMA chunk with an approximately right value than to omit the chunk and force PNG decoders to guess at an appropriate gamma. On the other hand, if the image file is being converted as part of a "bulk" conversion, with no one looking at each image, then it is better to omit the gAMA chunk entirely. If the image gamma has to be guessed at, leave it to the decoder to do the guessing. Gamma does not apply to alpha samples; alpha is always represented linearly. See also Recommendations for Decoders: Decoder gamma handling (Section 10.5). 9.3. Encoder color handling See Color Tutorial (Chapter 14) if you aren't already familiar with color issues. If it is possible for the encoder to determine the chromaticities of the source display primaries, or to make a strong guess based on the origin of the image or the hardware running it, then the encoder is strongly encouraged to output the cHRM chunk. If it does so, the gAMA chunk should also be written; decoders can do little with cHRM if gAMA is missing. Formatted and Indexed by: page 45 of 102 instructional media + magic, inc. RFC2083 Web Address: PNG: Portable Network Graphics http://sunsite.cnlab-switch.ch/ By: Boutell, et. al. Video created with recent video equipment probably uses the CCIR 709 primaries and D65 white point [ITU-BT709], which are: R G B White x 0.640 0.300 0.150 0.3127 y 0.330 0.600 0.060 0.3290 An older but still very popular video standard is SMPTE-C [SMPTE- 170M]: R G B White x 0.630 0.310 0.155 0.3127 y 0.340 0.595 0.070 0.3290 The original NTSC color primaries have not been used in decades. Although you may still find the NTSC numbers listed in standards documents, you won't find any images that actually use them. Scanners that produce PNG files as output should insert the filter chromaticities into a cHRM chunk and the camera_gamma into a gAMA chunk. In the case of hand-drawn or digitally edited images, you have to determine what monitor they were viewed on when being produced. Many image editing programs allow you to specify what type of monitor you are using. This is often because they are working in some device-independent space internally. Such programs have enough information to write valid cHRM and gAMA chunks, and should do so automatically. If the encoder is compiled as a portion of a computer image renderer that performs full-spectral rendering, the monitor values that were used to convert from the internal device-independent color space to RGB should be written into the cHRM chunk. Any colors that are outside the gamut of the chosen RGB device should be clipped or otherwise constrained to be within the gamut; PNG does not store out of gamut colors. If the computer image renderer performs calculations directly in device-dependent RGB space, a cHRM chunk should not be written unless the scene description and rendering parameters have been adjusted to look good on a particular monitor. In that case, the data for that monitor (if known) should be used to construct a cHRM chunk. Formatted and Indexed by: page 46 of 102 instructional media + magic, inc. RFC2083 Web Address: PNG: Portable Network Graphics http://sunsite.cnlab-switch.ch/ By: Boutell, et. al. There are often cases where an image's exact origins are unknown, particularly if it began life in some other format. A few image formats store calibration information, which can be used to fill in the cHRM chunk. For example, all PhotoCD images use the CCIR 709 primaries and D65 whitepoint, so these values can be written into the cHRM chunk when converting a PhotoCD file. PhotoCD also uses the SMPTE-170M transfer function, which is closely approximated by a gAMA of 0.5. (PhotoCD can store colors outside the RGB gamut, so the image data will require gamut mapping before writing to PNG format.) TIFF 6.0 files can optionally store calibration information, which if present should be used to construct the cHRM chunk. GIF and most other formats do not store any calibration information. It is not recommended that file format converters attempt to convert supplied images to a different RGB color space. Store the data in the PNG file without conversion, and record the source primary chromaticities if they are known. Color space transformation at file conversion time is a bad idea because of gamut mismatches and rounding errors. As with gamma conversions, it's better to store the data losslessly and incur at most one conversion when the image is finally displayed. See also Recommendations for Decoders: Decoder color handling (Section 10.6). 9.4. Alpha channel creation The alpha channel can be regarded either as a mask that temporarily hides transparent parts of the image, or as a means for constructing a non-rectangular image. In the first case, the color values of fully transparent pixels should be preserved for future use. In the second case, the transparent pixels carry no useful data and are simply there to fill out the rectangular image area required by PNG. In this case, fully transparent pixels should all be assigned the same color value for best compression. Image authors should keep in mind the possibility that a decoder will ignore transparency control. Hence, the colors assigned to transparent pixels should be reasonable background colors whenever feasible. For applications that do not require a full alpha channel, or cannot afford the price in compression efficiency, the tRNS transparency chunk is also available. Formatted and Indexed by: page 47 of 102 instructional media + magic, inc. RFC2083 Web Address: PNG: Portable Network Graphics http://sunsite.cnlab-switch.ch/ By: Boutell, et. al. If the image has a known background color, this color should be written in the bKGD chunk. Even decoders that ignore transparency may use the bKGD color to fill unused screen area. If the original image has premultiplied (also called "associated") alpha data, convert it to PNG's non-premultiplied format by dividing each sample value by the corresponding alpha value, then multiplying by the maximum value for the image bit depth, and rounding to the nearest integer. In valid premultiplied data, the sample values never exceed their corresponding alpha values, so the result of the division should always be in the range 0 to 1. If the alpha value is zero, output black (zeroes). 9.5. Suggested palettes A PLTE chunk can appear in truecolor PNG files. In such files, the chunk is not an essential part of the image data, but simply represents a suggested palette that viewers may use to present the image on indexed-color display hardware. A suggested palette is of no interest to viewers running on truecolor hardware. If an encoder chooses to provide a suggested palette, it is recommended that a hIST chunk also be written to indicate the relative importance of the palette entries. The histogram values are most easily computed as "nearest neighbor" counts, that is, the approximate usage of each palette entry if no dithering is applied. (These counts will often be available for free as a consequence of developing the suggested palette.) For images of color type 2 (truecolor without alpha channel), it is recommended that the palette and histogram be computed with reference to the RGB data only, ignoring any transparent-color specification. If the file uses transparency (has a tRNS chunk), viewers can easily adapt the resulting palette for use with their intended background color. They need only replace the palette entry closest to the tRNS color with their background color (which may or may not match the file's bKGD color, if any). For images of color type 6 (truecolor with alpha channel), it is recommended that a bKGD chunk appear and that the palette and histogram be computed with reference to the image as it would appear after compositing against the specified background color. This definition is necessary to ensure that useful palette entries are generated for pixels having fractional alpha values. The resulting palette will probably only be useful to viewers that present the image against the same background color. It is recommended that PNG editors delete or recompute the palette if they alter or remove the bKGD chunk in an image of color type 6. Formatted and Indexed by: page 48 of 102 instructional media + magic, inc. RFC2083 Web Address: PNG: Portable Network Graphics http://sunsite.cnlab-switch.ch/ By: Boutell, et. al. If PLTE appears without bKGD in an image of color type 6, the circumstances under which the palette was computed are unspecified. 9.6. Filter selection For images of color type 3 (indexed color), filter type 0 (None) is usually the most effective. Note that color images with 256 or fewer colors should almost always be stored in indexed color format; truecolor format is likely to be much larger. Filter type 0 is also recommended for images of bit depths less than 8. For low-bit-depth grayscale images, it may be a net win to expand the image to 8-bit representation and apply filtering, but this is rare. For truecolor and grayscale images, any of the five filters may prove the most effective. If an encoder uses a fixed filter, the Paeth filter is most likely to be the best. For best compression of truecolor and grayscale images, we recommend an adaptive filtering approach in which a filter is chosen for each scanline. The following simple heuristic has performed well in early tests: compute the output scanline using all five filters, and select the filter that gives the smallest sum of absolute values of outputs. (Consider the output bytes as signed differences for this test.) This method usually outperforms any single fixed filter choice. However, it is likely that much better heuristics will be found as more experience is gained with PNG. Filtering according to these recommendations is effective on interlaced as well as noninterlaced images. 9.7. Text chunk processing A nonempty keyword must be provided for each text chunk. The generic keyword "Comment" can be used if no better description of the text is available. If a user-supplied keyword is used, be sure to check that it meets the restrictions on keywords. PNG text strings are expected to use the Latin-1 character set. Encoders should avoid storing characters that are not defined in Latin-1, and should provide character code remapping if the local system's character set is not Latin-1. Encoders should discourage the creation of single lines of text longer than 79 characters, in order to facilitate easy reading. Formatted and Indexed by: page 49 of 102 instructional media + magic, inc. RFC2083 Web Address: PNG: Portable Network Graphics http://sunsite.cnlab-switch.ch/ By: Boutell, et. al. It is recommended that text items less than 1K (1024 bytes) in size should be output using uncompressed tEXt chunks. In particular, it is recommended that the basic title and author keywords should always be output using uncompressed tEXt chunks. Lengthy disclaimers, on the other hand, are ideal candidates for zTXt. Placing large tEXt and zTXt chunks after the image data (after IDAT) can speed up image display in some situations, since the decoder won't have to read over the text to get to the image data. But it is recommended that small text chunks, such as the image title, appear before IDAT. 9.8. Use of private chunks Applications can use PNG private chunks to carry information that need not be understood by other applications. Such chunks must be given names with lowercase second letters, to ensure that they can never conflict with any future public chunk definition. Note, however, that there is no guarantee that some other application will not use the same private chunk name. If you use a private chunk type, it is prudent to store additional identifying information at the beginning of the chunk data. Use an ancillary chunk type (lowercase first letter), not a critical chunk type, for all private chunks that store information that is not absolutely essential to view the image. Creation of private critical chunks is discouraged because they render PNG files unportable. Such chunks should not be used in publicly available software or files. If private critical chunks are essential for your application, it is recommended that one appear near the start of the file, so that a standard decoder need not read very far before discovering that it cannot handle the file. If you want others outside your organization to understand a chunk type that you invent, contact the maintainers of the PNG specification to submit a proposed chunk name and definition for addition to the list of special-purpose public chunks (see Additional chunk types, Section 4.4). Note that a proposed public chunk name (with uppercase second letter) must not be used in publicly available software or files until registration has been approved. If an ancillary chunk contains textual information that might be of interest to a human user, you should not create a special chunk type for it. Instead use a tEXt chunk and define a suitable keyword. That way, the information will be available to users not using your software. Formatted and Indexed by: page 50 of 102 instructional media + magic, inc. RFC2083 Web Address: PNG: Portable Network Graphics http://sunsite.cnlab-switch.ch/ By: Boutell, et. al. Keywords in tEXt chunks should be reasonably self-explanatory, since the idea is to let other users figure out what the chunk contains. If of general usefulness, new keywords can be registered with the maintainers of the PNG specification. But it is permissible to use keywords without registering them first. 9.9. Private type and method codes This specification defines the meaning of only some of the possible values of some fields. For example, only compression method 0 and filter types 0 through 4 are defined. Numbers greater than 127 must be used when inventing experimental or private definitions of values for any of these fields. Numbers below 128 are reserved for possible future public extensions of this specification. Note that use of private type codes may render a file unreadable by standard decoders. Such codes are strongly discouraged except for experimental purposes, and should not appear in publicly available software or files. 10. Recommendations for Decoders This chapter gives some recommendations for decoder behavior. The only absolute requirement on a PNG decoder is that it successfully read any file conforming to the format specified in the preceding chapters. However, best results will usually be achieved by following these recommendations. 10.1. Error checking To ensure early detection of common file-transfer problems, decoders should verify that all eight bytes of the PNG file signature are correct. (See Rationale: PNG file signature, Section 12.11.) A decoder can have additional confidence in the file's integrity if the next eight bytes are an IHDR chunk header with the correct chunk length. Unknown chunk types must be handled as described in Chunk naming conventions (Section 3.3). An unknown chunk type is not to be treated as an error unless it is a critical chunk. It is strongly recommended that decoders should verify the CRC on each chunk. In some situations it is desirable to check chunk headers (length and type code) before reading the chunk data and CRC. The chunk type can be checked for plausibility by seeing whether all four bytes are ASCII letters (codes 65-90 and 97-122); note that this need only be done for unrecognized type codes. If the total file Formatted and Indexed by: page 51 of 102 instructional media + magic, inc. RFC2083 Web Address: PNG: Portable Network Graphics http://sunsite.cnlab-switch.ch/ By: Boutell, et. al. size is known (from file system information, HTTP protocol, etc), the chunk length can be checked for plausibility as well. If CRCs are not checked, dropped/added data bytes or an erroneous chunk length can cause the decoder to get out of step and misinterpret subsequent data as a chunk header. Verifying that the chunk type contains letters is an inexpensive way of providing early error detection in this situation. For known-length chunks such as IHDR, decoders should treat an unexpected chunk length as an error. Future extensions to this specification will not add new fields to existing chunks; instead, new chunk types will be added to carry new information. Unexpected values in fields of known chunks (for example, an unexpected compression method in the IHDR chunk) must be checked for and treated as errors. However, it is recommended that unexpected field values be treated as fatal errors only in critical chunks. An unexpected value in an ancillary chunk can be handled by ignoring the whole chunk as though it were an unknown chunk type. (This recommendation assumes that the chunk's CRC has been verified. In decoders that do not check CRCs, it is safer to treat any unexpected value as indicating a corrupted file.) 10.2. Pixel dimensions Non-square pixels can be represented (see the pHYs chunk), but viewers are not required to account for them; a viewer can present any PNG file as though its pixels are square. Conversely, viewers running on display hardware with non-square pixels are strongly encouraged to rescale images for proper display. 10.3. Truecolor image handling To achieve PNG's goal of universal interchangeability, decoders are required to accept all types of PNG image: indexed-color, truecolor, and grayscale. Viewers running on indexed-color display hardware need to be able to reduce truecolor images to indexed format for viewing. This process is usually called "color quantization". Formatted and Indexed by: page 52 of 102 instructional media + magic, inc. RFC2083 Web Address: PNG: Portable Network Graphics http://sunsite.cnlab-switch.ch/ By: Boutell, et. al. A simple, fast way of doing this is to reduce the image to a fixed palette. Palettes with uniform color spacing ("color cubes") are usually used to minimize the per-pixel computation. For photograph-like images, dithering is recommended to avoid ugly contours in what should be smooth gradients; however, dithering introduces graininess that can be objectionable. The quality of rendering can be improved substantially by using a palette chosen specifically for the image, since a color cube usually has numerous entries that are unused in any particular image. This approach requires more work, first in choosing the palette, and second in mapping individual pixels to the closest available color. PNG allows the encoder to supply a suggested palette in a PLTE chunk, but not all encoders will do so, and the suggested palette may be unsuitable in any case (it may have too many or too few colors). High-quality viewers will therefore need to have a palette selection routine at hand. A large lookup table is usually the most feasible way of mapping individual pixels to palette entries with adequate speed. Numerous implementations of color quantization are available. The PNG reference implementation, libpng, includes code for the purpose. 10.4. Sample depth rescaling Decoders may wish to scale PNG data to a lesser sample depth (data precision) for display. For example, 16-bit data will need to be reduced to 8-bit depth for use on most present-day display hardware. Reduction of 8-bit data to 5-bit depth is also common. The most accurate scaling is achieved by the linear equation output = ROUND(input * MAXOUTSAMPLE / MAXINSAMPLE) where MAXINSAMPLE = (2^sampledepth)-1 MAXOUTSAMPLE = (2^desired_sampledepth)-1 A slightly less accurate conversion is achieved by simply shifting right by sampledepth-desired_sampledepth places. For example, to reduce 16-bit samples to 8-bit, one need only discard the low- order byte. In many situations the shift method is sufficiently accurate for display purposes, and it is certainly much faster. (But if gamma correction is being done, sample rescaling can be merged into the gamma correction lookup table, as is illustrated in Decoder gamma handling, Section 10.5.) Formatted and Indexed by: page 53 of 102 instructional media + magic, inc. RFC2083 Web Address: PNG: Portable Network Graphics http://sunsite.cnlab-switch.ch/ By: Boutell, et. al. When an sBIT chunk is present, the original pre-PNG data can be recovered by shifting right to the sample depth specified by sBIT. Note that linear scaling will not necessarily reproduce the original data, because the encoder is not required to have used linear scaling to scale the data up. However, the encoder is required to have used a method that preserves the high-order bits, so shifting always works. This is the only case in which shifting might be said to be more accurate than linear scaling. When comparing pixel values to tRNS chunk values to detect transparent pixels, it is necessary to do the comparison exactly. Therefore, transparent pixel detection must be done before reducing sample precision. 10.5. Decoder gamma handling See Gamma Tutorial (Chapter 13) if you aren't already familiar with gamma issues. To produce correct tone reproduction, a good image display program should take into account the gammas of the image file and the display device, as well as the viewing_gamma appropriate to the lighting conditions near the display. This can be done by calculating gbright = insample / MAXINSAMPLE bright = gbright ^ (1.0 / file_gamma) vbright = bright ^ viewing_gamma gcvideo = vbright ^ (1.0 / display_gamma) fbval = ROUND(gcvideo * MAXFBVAL) where MAXINSAMPLE is the maximum sample value in the file (255 for 8-bit, 65535 for 16-bit, etc), MAXFBVAL is the maximum value of a frame buffer sample (255 for 8-bit, 31 for 5-bit, etc), insample is the value of the sample in the PNG file, and fbval is the value to write into the frame buffer. The first line converts from integer samples into a normalized 0 to 1 floating point value, the second undoes the gamma encoding of the image file to produce a linear intensity value, the third adjusts for the viewing conditions, the fourth corrects for the display system's gamma value, and the fifth converts to an integer frame buffer sample. In practice, the second through fourth lines can be merged into gcvideo = gbright^(viewing_gamma / (file_gamma*display_gamma)) so as to perform only one power calculation. For color images, the entire calculation is performed separately for R, G, and B values. Formatted and Indexed by: page 54 of 102 instructional media + magic, inc. RFC2083 Web Address: PNG: Portable Network Graphics http://sunsite.cnlab-switch.ch/ By: Boutell, et. al. It is not necessary to perform transcendental math for every pixel. Instead, compute a lookup table that gives the correct output value for every possible sample value. This requires only 256 calculations per image (for 8-bit accuracy), not one or three calculations per pixel. For an indexed-color image, a one-time correction of the palette is sufficient, unless the image uses transparency and is being displayed against a nonuniform background. In some cases even the cost of computing a gamma lookup table may be a concern. In these cases, viewers are encouraged to have precomputed gamma correction tables for file_gamma values of 1.0 and 0.5 with some reasonable choice of viewing_gamma and display_gamma, and to use the table closest to the gamma indicated in the file. This will produce acceptable results for the majority of real files. When the incoming image has unknown gamma (no gAMA chunk), choose a likely default file_gamma value, but allow the user to select a new one if the result proves too dark or too light. In practice, it is often difficult to determine what value of display_gamma should be used. In systems with no built-in gamma correction, the display_gamma is determined entirely by the CRT. Assuming a CRT_gamma of 2.5 is recommended, unless you have detailed calibration measurements of this particular CRT available. However, many modern frame buffers have lookup tables that are used to perform gamma correction, and on these systems the display_gamma value should be the gamma of the lookup table and CRT combined. You may not be able to find out what the lookup table contains from within an image viewer application, so you may have to ask the user what the system's gamma value is. Unfortunately, different manufacturers use different ways of specifying what should go into the lookup table, so interpretation of the system gamma value is system-dependent. Gamma Tutorial (Chapter 13) gives some examples. The response of real displays is actually more complex than can be described by a single number (display_gamma). If actual measurements of the monitor's light output as a function of voltage input are available, the fourth and fifth lines of the computation above can be replaced by a lookup in these measurements, to find the actual frame buffer value that most nearly gives the desired brightness. Formatted and Indexed by: page 55 of 102 instructional media + magic, inc. RFC2083 Web Address: PNG: Portable Network Graphics http://sunsite.cnlab-switch.ch/ By: Boutell, et. al. The value of viewing_gamma depends on lighting conditions; see Gamma Tutorial (Chapter 13) for more detail. Ideally, a viewer would allow the user to specify viewing_gamma, either directly numerically, or via selecting from "bright surround", "dim surround", and "dark surround" conditions. Viewers that don't want to do this should just assume a value for viewing_gamma of 1.0, since most computer displays live in brightly-lit rooms. When viewing images that are digitized from video, or that are destined to become video frames, the user might want to set the viewing_gamma to about 1.25 regardless of the actual level of room lighting. This value of viewing_gamma is "built into" NTSC video practice, and displaying an image with that viewing_gamma allows the user to see what a TV set would show under the current room lighting conditions. (This is not the same thing as trying to obtain the most accurate rendition of the content of the scene, which would require adjusting viewing_gamma to correspond to the room lighting level.) This is another reason viewers might want to allow users to adjust viewing_gamma directly. 10.6. Decoder color handling See Color Tutorial (Chapter 14) if you aren't already familiar with color issues. In many cases, decoders will treat image data in PNG files as device-dependent RGB data and display it without modification (except for appropriate gamma correction). This provides the fastest display of PNG images. But unless the viewer uses exactly the same display hardware as the original image author used, the colors will not be exactly the same as the original author saw, particularly for darker or near-neutral colors. The cHRM chunk provides information that allows closer color matching than that provided by gamma correction alone. Decoders can use the cHRM data to transform the image data from RGB to XYZ and thence into a perceptually linear color space such as CIE LAB. They can then partition the colors to generate an optimal palette, because the geometric distance between two colors in CIE LAB is strongly related to how different those colors appear (unlike, for example, RGB or XYZ spaces). The resulting palette of colors, once transformed back into RGB color space, could be used for display or written into a PLTE chunk. Decoders that are part of image processing applications might also transform image data into CIE LAB space for analysis. Formatted and Indexed by: page 56 of 102 instructional media + magic, inc. RFC2083 Web Address: PNG: Portable Network Graphics http://sunsite.cnlab-switch.ch/ By: Boutell, et. al. In applications where color fidelity is critical, such as product design, scientific visualization, medicine, architecture, or advertising, decoders can transform the image data from source_RGB to the display_RGB space of the monitor used to view the image. This involves calculating the matrix to go from source_RGB to XYZ and the matrix to go from XYZ to display_RGB, then combining them to produce the overall transformation. The decoder is responsible for implementing gamut mapping. Decoders running on platforms that have a Color Management System (CMS) can pass the image data, gAMA and cHRM values to the CMS for display or further processing. Decoders that provide color printing facilities can use the facilities in Level 2 PostScript to specify image data in calibrated RGB space or in a device-independent color space such as XYZ. This will provide better color fidelity than a simple RGB to CMYK conversion. The PostScript Language Reference manual gives examples of this process [POSTSCRIPT]. Such decoders are responsible for implementing gamut mapping between source_RGB (specified in the cHRM chunk) and the target printer. The PostScript interpreter is then responsible for producing the required colors. Decoders can use the cHRM data to calculate an accurate grayscale representation of a color image. Conversion from RGB to gray is simply a case of calculating the Y (luminance) component of XYZ, which is a weighted sum of the R G and B values. The weights depend on the monitor type, i.e., the values in the cHRM chunk. Decoders may wish to do this for PNG files with no cHRM chunk. In that case, a reasonable default would be the CCIR 709 primaries [ITU-BT709]. Do not use the original NTSC primaries, unless you really do have an image color-balanced for such a monitor. Few monitors ever used the NTSC primaries, so such images are probably nonexistent these days. 10.7. Background color The background color given by bKGD will typically be used to fill unused screen space around the image, as well as any transparent pixels within the image. (Thus, bKGD is valid and useful even when the image does not use transparency.) If no bKGD chunk is present, the viewer will need to make its own decision about a suitable background color. Formatted and Indexed by: page 57 of 102 instructional media + magic, inc. RFC2083 Web Address: PNG: Portable Network Graphics http://sunsite.cnlab-switch.ch/ By: Boutell, et. al. Viewers that have a specific background against which to present the image (such as Web browsers) should ignore the bKGD chunk, in effect overriding bKGD with their preferred background color or background image. The background color given by bKGD is not to be considered transparent, even if it happens to match the color given by tRNS (or, in the case of an indexed-color image, refers to a palette index that is marked as transparent by tRNS). Otherwise one would have to imagine something "behind the background" to composite against. The background color is either used as background or ignored; it is not an intermediate layer between the PNG image and some other background. Indeed, it will be common that bKGD and tRNS specify the same color, since then a decoder that does not implement transparency processing will give the intended display, at least when no partially-transparent pixels are present. 10.8. Alpha channel processing In the most general case, the alpha channel can be used to composite a foreground image against a background image; the PNG file defines the foreground image and the transparency mask, but not the background image. Decoders are not required to support this most general case. It is expected that most will be able to support compositing against a single background color, however. The equation for computing a composited sample value is output = alpha * foreground + (1-alpha) * background where alpha and the input and output sample values are expressed as fractions in the range 0 to 1. This computation should be performed with linear (non-gamma-encoded) sample values. For color images, the computation is done separately for R, G, and B samples. The following code illustrates the general case of compositing a foreground image over a background image. It assumes that you have the original pixel data available for the background image, and that output is to a frame buffer for display. Other variants are possible; see the comments below the code. The code allows the sample depths and gamma values of foreground image, background image, and frame buffer/CRT all to be different. Don't assume they are the same without checking. Formatted and Indexed by: page 58 of 102 instructional media + magic, inc. RFC2083 Web Address: PNG: Portable Network Graphics http://sunsite.cnlab-switch.ch/ By: Boutell, et. al. This code is standard C, with line numbers added for reference in the comments below. 01 int foreground[4]; /* image pixel: R, G, B, A */ 02 int background[3]; /* background pixel: R, G, B */ 03 int fbpix[3]; /* frame buffer pixel */ 04 int fg_maxsample; /* foreground max sample */ 05 int bg_maxsample; /* background max sample */ 06 int fb_maxsample; /* frame buffer max sample */ 07 int ialpha; 08 float alpha, compalpha; 09 float gamfg, linfg, gambg, linbg, comppix, gcvideo; /* Get max sample values in data and frame buffer */ 10 fg_maxsample = (1 << fg_sample_depth) - 1; 11 bg_maxsample = (1 << bg_sample_depth) - 1; 12 fb_maxsample = (1 << frame_buffer_sample_depth) - 1; /* * Get integer version of alpha. * Check for opaque and transparent special cases; * no compositing needed if so. * * We show the whole gamma decode/correct process in * floating point, but it would more likely be done * with lookup tables. */ 13 ialpha = foreground[3]; 14 if (ialpha == 0) { /* * Foreground image is transparent here. * If the background image is already in the frame * buffer, there is nothing to do. */ 15 ; 16 } else if (ialpha == fg_maxsample) { /* * Copy foreground pixel to frame buffer. */ 17 for (i = 0; i < 3; i++) { 18 gamfg = (float) foreground[i] / fg_maxsample; 19 linfg = pow(gamfg, 1.0/fg_gamma); 20 comppix = linfg; 21 gcvideo = pow(comppix,viewing_gamma/display_gamma); 22 fbpix[i] = (int) (gcvideo * fb_maxsample + 0.5); 23 } Formatted and Indexed by: page 59 of 102 instructional media + magic, inc. RFC2083 Web Address: PNG: Portable Network Graphics http://sunsite.cnlab-switch.ch/ By: Boutell, et. al. 24 } else { /* * Compositing is necessary. * Get floating-point alpha and its complement. * Note: alpha is always linear; gamma does not * affect it. */ 25 alpha = (float) ialpha / fg_maxsample; 26 compalpha = 1.0 - alpha; 27 for (i = 0; i < 3; i++) { /* * Convert foreground and background to floating * point, then linearize (undo gamma encoding). */ 28 gamfg = (float) foreground[i] / fg_maxsample; 29 linfg = pow(gamfg, 1.0/fg_gamma); 30 gambg = (float) background[i] / bg_maxsample; 31 linbg = pow(gambg, 1.0/bg_gamma); /* * Composite. */ 32 comppix = linfg * alpha + linbg * compalpha; /* * Gamma correct for display. * Convert to integer frame buffer pixel. */ 33 gcvideo = pow(comppix,viewing_gamma/display_gamma); 34 fbpix[i] = (int) (gcvideo * fb_maxsample + 0.5); 35 } 36 } Variations: * If output is to another PNG image file instead of a frame buffer, lines 21, 22, 33, and 34 should be changed to be something like /* * Gamma encode for storage in output file. * Convert to integer sample value. */ gamout = pow(comppix, outfile_gamma); outpix[i] = (int) (gamout * out_maxsample + 0.5); Also, it becomes necessary to process background pixels when alpha is zero, rather than just skipping pixels. Thus, line 15 will need to be replaced by copies of lines 17-23, but processing background instead of foreground pixel values. Formatted and Indexed by: page 60 of 102 instructional media + magic, inc. RFC2083 Web Address: PNG: Portable Network Graphics http://sunsite.cnlab-switch.ch/ By: Boutell, et. al. * If the sample depths of the output file, foreground file, and background file are all the same, and the three gamma values also match, then the no-compositing code in lines 14-23 reduces to nothing more than copying pixel values from the input file to the output file if alpha is one, or copying pixel values from background to output file if alpha is zero. Since alpha is typically either zero or one for the vast majority of pixels in an image, this is a great savings. No gamma computations are needed for most pixels. * When the sample depths and gamma values all match, it may appear attractive to skip the gamma decoding and encoding (lines 28-31, 33-34) and just perform line 32 using gamma- encoded sample values. Although this doesn't hurt image quality too badly, the time savings are small if alpha values of zero and one are special-cased as recommended here. * If the original pixel values of the background image are no longer available, only processed frame buffer pixels left by display of the background image, then lines 30 and 31 need to extract intensity from the frame buffer pixel values using code like /* * Decode frame buffer value back into linear space. */ gcvideo = (float) fbpix[i] / fb_maxsample; linbg = pow(gcvideo, display_gamma / viewing_gamma); However, some roundoff error can result, so it is better to have the original background pixels available if at all possible. * Note that lines 18-22 are performing exactly the same gamma computation that is done when no alpha channel is present. So, if you handle the no-alpha case with a lookup table, you can use the same lookup table here. Lines 28-31 and 33-34 can also be done with (different) lookup tables. * Of course, everything here can be done in integer arithmetic. Just be careful to maintain sufficient precision all the way through. Note: in floating point, no overflow or underflow checks are needed, because the input sample values are guaranteed to be between 0 and 1, and compositing always yields a result that is in between the input values (inclusive). With integer arithmetic, some roundoff-error analysis might be needed to guarantee no overflow or underflow. Formatted and Indexed by: page 61 of 102 instructional media + magic, inc. RFC2083 Web Address: PNG: Portable Network Graphics http://sunsite.cnlab-switch.ch/ By: Boutell, et. al. When displaying a PNG image with full alpha channel, it is important to be able to composite the image against some background, even if it's only black. Ignoring the alpha channel will cause PNG images that have been converted from an associated-alpha representation to look wrong. (Of course, if the alpha channel is a separate transparency mask, then ignoring alpha is a useful option: it allows the hidden parts of the image to be recovered.) Even if the decoder author does not wish to implement true compositing logic, it is simple to deal with images that contain only zero and one alpha values. (This is implicitly true for grayscale and truecolor PNG files that use a tRNS chunk; for indexed-color PNG files, it is easy to check whether tRNS contains any values other than 0 and 255.) In this simple case, transparent pixels are replaced by the background color, while others are unchanged. If a decoder contains only this much transparency capability, it should deal with a full alpha channel by treating all nonzero alpha values as fully opaque; that is, do not replace partially transparent pixels by the background. This approach will not yield very good results for images converted from associated-alpha formats, but it's better than doing nothing. 10.9. Progressive display When receiving images over slow transmission links, decoders can improve perceived performance by displaying interlaced images progressively. This means that as each pass is received, an approximation to the complete image is displayed based on the data received so far. One simple yet pleasing effect can be obtained by expanding each received pixel to fill a rectangle covering the yet-to-be-transmitted pixel positions below and to the right of the received pixel. This process can be described by the following pseudocode: Starting_Row [1..7] = { 0, 0, 4, 0, 2, 0, 1 } Starting_Col [1..7] = { 0, 4, 0, 2, 0, 1, 0 } Row_Increment [1..7] = { 8, 8, 8, 4, 4, 2, 2 } Col_Increment [1..7] = { 8, 8, 4, 4, 2, 2, 1 } Block_Height [1..7] = { 8, 8, 4, 4, 2, 2, 1 } Block_Width [1..7] = { 8, 4, 4, 2, 2, 1, 1 } pass := 1 while pass <= 7 begin row := Starting_Row[pass] while row < height Formatted and Indexed by: page 62 of 102 instructional media + magic, inc. RFC2083 Web Address: PNG: Portable Network Graphics http://sunsite.cnlab-switch.ch/ By: Boutell, et. al. begin col := Starting_Col[pass] while col < width begin visit (row, col, min (Block_Height[pass], height - row), min (Block_Width[pass], width - col)) col := col + Col_Increment[pass] end row := row + Row_Increment[pass] end pass := pass + 1 end Here, the function "visit(row,column,height,width)" obtains the next transmitted pixel and paints a rectangle of the specified height and width, whose upper-left corner is at the specified row and column, using the color indicated by the pixel. Note that row and column are measured from 0,0 at the upper left corner. If the decoder is merging the received image with a background image, it may be more convenient just to paint the received pixel positions; that is, the "visit()" function sets only the pixel at the specified row and column, not the whole rectangle. This produces a "fade-in" effect as the new image gradually replaces the old. An advantage of this approach is that proper alpha or transparency processing can be done as each pixel is replaced. Painting a rectangle as described above will overwrite background-image pixels that may be needed later, if the pixels eventually received for those positions turn out to be wholly or partially transparent. Of course, this is only a problem if the background image is not stored anywhere offscreen. 10.10. Suggested-palette and histogram usage In truecolor PNG files, the encoder may have provided a suggested PLTE chunk for use by viewers running on indexed-color hardware. If the image has a tRNS chunk, the viewer will need to adapt the suggested palette for use with its desired background color. To do this, replace the palette entry closest to the tRNS color with the desired background color; or just add a palette entry for the background color, if the viewer can handle more colors than there are PLTE entries. Formatted and Indexed by: page 63 of 102 instructional media + magic, inc. RFC2083 Web Address: PNG: Portable Network Graphics http://sunsite.cnlab-switch.ch/ By: Boutell, et. al. For images of color type 6 (truecolor with alpha channel), any suggested palette should have been designed for display of the image against a uniform background of the color specified by bKGD. Viewers should probably ignore the palette if they intend to use a different background, or if the bKGD chunk is missing. Viewers can use a suggested palette for display against a different background than it was intended for, but the results may not be very good. If the viewer presents a transparent truecolor image against a background that is more complex than a single color, it is unlikely that the suggested palette will be optimal for the composite image. In this case it is best to perform a truecolor compositing step on the truecolor PNG image and background image, then color-quantize the resulting image. The histogram chunk is useful when the viewer cannot provide as many colors as are used in the image's palette. If the viewer is only short a few colors, it is usually adequate to drop the least-used colors from the palette. To reduce the number of colors substantially, it's best to choose entirely new representative colors, rather than trying to use a subset of the existing palette. This amounts to performing a new color quantization step; however, the existing palette and histogram can be used as the input data, thus avoiding a scan of the image data. If no palette or histogram chunk is provided, a decoder can develop its own, at the cost of an extra pass over the image data. Alternatively, a default palette (probably a color cube) can be used. See also Recommendations for Encoders: Suggested palettes (Section 9.5). 10.11. Text chunk processing If practical, decoders should have a way to display to the user all tEXt and zTXt chunks found in the file. Even if the decoder does not recognize a particular text keyword, the user might be able to understand it. PNG text is not supposed to contain any characters outside the ISO 8859-1 "Latin-1" character set (that is, no codes 0-31 or 127- 159), except for the newline character (decimal 10). But decoders might encounter such characters anyway. Some of these characters can be safely displayed (e.g., TAB, FF, and CR, decimal 9, 12, and 13, respectively), but others, especially the ESC character (decimal 27), could pose a security hazard because unexpected Formatted and Indexed by: page 64 of 102 instructional media + magic, inc. RFC2083 Web Address: PNG: Portable Network Graphics http://sunsite.cnlab-switch.ch/ By: Boutell, et. al. actions may be taken by display hardware or software. To prevent such hazards, decoders should not attempt to directly display any non-Latin-1 characters (except for newline and perhaps TAB, FF, CR) encountered in a tEXt or zTXt chunk. Instead, ignore them or display them in a visible notation such as "\nnn". See Security considerations (Section 8.5). Even though encoders are supposed to represent newlines as LF, it is recommended that decoders not rely on this; it's best to recognize all the common newline combinations (CR, LF, and CR-LF) and display each as a single newline. TAB can be expanded to the proper number of spaces needed to arrive at a column multiple of 8. Decoders running on systems with non-Latin-1 character set encoding should provide character code remapping so that Latin-1 characters are displayed correctly. Some systems may not provide all the characters defined in Latin-1. Mapping unavailable characters to a visible notation such as "\nnn" is a good fallback. In particular, character codes 127-255 should be displayed only if they are printable characters on the decoding system. Some systems may interpret such codes as control characters; for security, decoders running on such systems should not display such characters literally. Decoders should be prepared to display text chunks that contain any number of printing characters between newline characters, even though encoders are encouraged to avoid creating lines in excess of 79 characters. 11. Glossary a^b Exponentiation; a raised to the power b. C programmers should be careful not to misread this notation as exclusive-or. Note that in gamma-related calculations, zero raised to any power is valid and must give a zero result. Alpha A value representing a pixel's degree of transparency. The more transparent a pixel, the less it hides the background against which the image is presented. In PNG, alpha is really the degree of opacity: zero alpha represents a completely transparent pixel, maximum alpha represents a completely opaque pixel. But most people refer to alpha as providing transparency information, not opacity information, and we continue that custom here. Formatted and Indexed by: page 65 of 102 instructional media + magic, inc. RFC2083 Web Address: PNG: Portable Network Graphics http://sunsite.cnlab-switch.ch/ By: Boutell, et. al. Ancillary chunk A chunk that provides additional information. A decoder can still produce a meaningful image, though not necessarily the best possible image, without processing the chunk. Bit depth The number of bits per palette index (in indexed-color PNGs) or per sample (in other color types). This is the same value that appears in IHDR. Byte Eight bits; also called an octet. Channel The set of all samples of the same kind within an image; for example, all the blue samples in a truecolor image. (The term "component" is also used, but not in this specification.) A sample is the intersection of a channel and a pixel. Chromaticity A pair of values x,y that precisely specify the hue, though not the absolute brightness, of a perceived color. Chunk A section of a PNG file. Each chunk has a type indicated by its chunk type name. Most types of chunks also include some data. The format and meaning of the data within the chunk are determined by the type name. Composite As a verb, to form an image by merging a foreground image and a background image, using transparency information to determine where the background should be visible. The foreground image is said to be "composited against" the background. CRC Cyclic Redundancy Check. A CRC is a type of check value designed to catch most transmission errors. A decoder calculates the CRC for the received data and compares it to the CRC that the encoder calculated, which is appended to the data. A mismatch indicates that the data was corrupted in transit. Critical chunk A chunk that must be understood and processed by the decoder in order to produce a meaningful image from a PNG file. CRT Cathode Ray Tube: a common type of computer display hardware. Formatted and Indexed by: page 66 of 102 instructional media + magic, inc. RFC2083 Web Address: PNG: Portable Network Graphics http://sunsite.cnlab-switch.ch/ By: Boutell, et. al. Datastream A sequence of bytes. This term is used rather than "file" to describe a byte sequence that is only a portion of a file. We also use it to emphasize that a PNG image might be generated and consumed "on the fly", never appearing in a stored file at all. Deflate The name of the compression algorithm used in standard PNG files, as well as in zip, gzip, pkzip, and other compression programs. Deflate is a member of the LZ77 family of compression methods. Filter A transformation applied to image data in hopes of improving its compressibility. PNG uses only lossless (reversible) filter algorithms. Frame buffer The final digital storage area for the image shown by a computer display. Software causes an image to appear onscreen by loading it into the frame buffer. Gamma The brightness of mid-level tones in an image. More precisely, a parameter that describes the shape of the transfer function for one or more stages in an imaging pipeline. The transfer function is given by the expression output = input ^ gamma where both input and output are scaled to the range 0 to 1. Grayscale An image representation in which each pixel is represented by a single sample value representing overall luminance (on a scale from black to white). PNG also permits an alpha sample to be stored for each pixel of a grayscale image. Indexed color An image representation in which each pixel is represented by a single sample that is an index into a palette or lookup table. The selected palette entry defines the actual color of the pixel. Lossless compression Any method of data compression that guarantees the original data can be reconstructed exactly, bit-for-bit. Formatted and Indexed by: page 67 of 102 instructional media + magic, inc. RFC2083 Web Address: PNG: Portable Network Graphics http://sunsite.cnlab-switch.ch/ By: Boutell, et. al. Lossy compression Any method of data compression that reconstructs the original data approximately, rather than exactly. LSB Least Significant Byte of a multi-byte value. Luminance Perceived brightness, or grayscale level, of a color. Luminance and chromaticity together fully define a perceived color. LUT Look Up Table. In general, a table used to transform data. In frame buffer hardware, a LUT can be used to map indexed-color pixels into a selected set of truecolor values, or to perform gamma correction. In software, a LUT can be used as a fast way of implementing any one-variable mathematical function. MSB Most Significant Byte of a multi-byte value. Palette The set of colors available in an indexed-color image. In PNG, a palette is an array of colors defined by red, green, and blue samples. (Alpha values can also be defined for palette entries, via the tRNS chunk.) Pixel The information stored for a single grid point in the image. The complete image is a rectangular array of pixels. PNG editor A program that modifies a PNG file and preserves ancillary information, including chunks that it does not recognize. Such a program must obey the rules given in Chunk Ordering Rules (Chapter 7). Sample A single number in the image data; for example, the red value of a pixel. A pixel is composed of one or more samples. When discussing physical data layout (in particular, in Image layout, Section 2.3), we use "sample" to mean a number stored in the image array. It would be more precise but much less readable to say "sample or palette index" in that context. Elsewhere in the specification, "sample" means a color value or alpha value. In the indexed-color case, these are palette entries not palette indexes. Formatted and Indexed by: page 68 of 102 instructional media + magic, inc. RFC2083 Web Address: PNG: Portable Network Graphics http://sunsite.cnlab-switch.ch/ By: Boutell, et. al. Sample depth The precision, in bits, of color values and alpha values. In indexed-color PNGs the sample depth is always 8 by definition of the PLTE chunk. In other color types it is the same as the bit depth. Scanline One horizontal row of pixels within an image. Truecolor An image representation in which pixel colors are defined by storing three samples for each pixel, representing red, green, and blue intensities respectively. PNG also permits an alpha sample to be stored for each pixel of a truecolor image. White point The chromaticity of a computer display's nominal white value. zlib A particular format for data that has been compressed using deflate-style compression. Also the name of a library implementing this method. PNG implementations need not use the zlib library, but they must conform to its format for compressed data. 12. Appendix: Rationale (This appendix is not part of the formal PNG specification.) This appendix gives the reasoning behind some of the design decisions in PNG. Many of these decisions were the subject of considerable debate. The authors freely admit that another group might have made different decisions; however, we believe that our choices are defensible and consistent. 12.1. Why a new file format? Does the world really need yet another graphics format? We believe so. GIF is no longer freely usable, but no other commonly used format can directly replace it, as is discussed in more detail below. We might have used an adaptation of an existing format, for example GIF with an unpatented compression scheme. But this would require new code anyway; it would not be all that much easier to implement than a whole new file format. (PNG is designed to be simple to implement, with the exception of the compression engine, which would be needed in any case.) We feel that this is an excellent opportunity to design a new format that fixes some of the known limitations of GIF. Formatted and Indexed by: page 69 of 102 instructional media + magic, inc. RFC2083 Web Address: PNG: Portable Network Graphics http://sunsite.cnlab-switch.ch/ By: Boutell, et. al. 12.2. Why these features? The features chosen for PNG are intended to address the needs of applications that previously used the special strengths of GIF. In particular, GIF is well adapted for online communications because of its streamability and progressive display capability. PNG shares those attributes. We have also addressed some of the widely known shortcomings of GIF. In particular, PNG supports truecolor images. We know of no widely used image format that losslessly compresses truecolor images as effectively as PNG does. We hope that PNG will make use of truecolor images more practical and widespread. Some form of transparency control is desirable for applications in which images are displayed against a background or together with other images. GIF provided a simple transparent-color specification for this purpose. PNG supports a full alpha channel as well as transparent-color specifications. This allows both highly flexible transparency and compression efficiency. Robustness against transmission errors has been an important consideration. For example, images transferred across Internet are often mistakenly processed as text, leading to file corruption. PNG is designed so that such errors can be detected quickly and reliably. PNG has been expressly designed not to be completely dependent on a single compression technique. Although deflate/inflate compression is mentioned in this document, PNG would still exist without it. 12.3. Why not these features? Some features have been deliberately omitted from PNG. These choices were made to simplify implementation of PNG, promote portability and interchangeability, and make the format as simple and foolproof as possible for users. In particular: Formatted and Indexed by: page 70 of 102 instructional media + magic, inc. RFC2083 Web Address: PNG: Portable Network Graphics http://sunsite.cnlab-switch.ch/ By: Boutell, et. al. * There is no uncompressed variant of PNG. It is possible to store uncompressed data by using only uncompressed deflate blocks (a feature normally used to guarantee that deflate does not make incompressible data much larger). However, PNG software must support full deflate/inflate; any software that does not is not compliant with the PNG standard. The two most important features of PNG---portability and compression---are absolute requirements for online applications, and users demand them. Failure to support full deflate/inflate compromises both of these objectives. * There is no lossy compression in PNG. Existing formats such as JFIF already handle lossy compression well. Furthermore, available lossy compression methods (e.g., JPEG) are far from foolproof --- a poor choice of quality level can ruin an image. To avoid user confusion and unintentional loss of information, we feel it is best to keep lossy and lossless formats strictly separate. Also, lossy compression is complex to implement. Adding JPEG support to a PNG decoder might increase its size by an order of magnitude. This would certainly cause some decoders to omit support for the feature, which would destroy our goal of interchangeability. * There is no support for CMYK or other unusual color spaces. Again, this is in the name of promoting portability. CMYK, in particular, is far too device-dependent to be useful as a portable image representation. * There is no standard chunk for thumbnail views of images. In discussions with software vendors who use thumbnails in their products, it has become clear that most would not use a "standard" thumbnail chunk. For one thing, every vendor has a different idea of what the dimensions and characteristics of a thumbnail ought to be. Also, some vendors keep thumbnails in separate files to accommodate varied image formats; they are not going to stop doing that simply because of a thumbnail chunk in one new format. Proprietary chunks containing vendor-specific thumbnails appear to be more practical than a common thumbnail format. It is worth noting that private extensions to PNG could easily add these features. We will not, however, include them as part of the basic PNG standard. Formatted and Indexed by: page 71 of 102 instructional media + magic, inc. RFC2083 Web Address: PNG: Portable Network Graphics http://sunsite.cnlab-switch.ch/ By: Boutell, et. al. PNG also does not support multiple images in one file. This restriction is a reflection of the reality that many applications do not need and will not support multiple images per file. In any case, single images are a fundamentally different sort of object from sequences of images. Rather than make false promises of interchangeability, we have drawn a clear distinction between single-image and multi-image formats. PNG is a single-image format. (But see Multiple-image extension, Section 8.4.) 12.4. Why not use format X? Numerous existing formats were considered before deciding to develop PNG. None could meet the requirements we felt were important for PNG. GIF is no longer suitable as a universal standard because of legal entanglements. Although just replacing GIF's compression method would avoid that problem, GIF does not support truecolor images, alpha channels, or gamma correction. The spec has more subtle problems too. Only a small subset of the GIF89 spec is actually portable across a variety of implementations, but there is no codification of the most portable part of the spec. TIFF is far too complex to meet our goals of simplicity and interchangeability. Defining a TIFF subset would meet that objection, but would frustrate users making the reasonable assumption that a file saved as TIFF from their existing software would load into a program supporting our flavor of TIFF. Furthermore, TIFF is not designed for stream processing, has no provision for progressive display, and does not currently provide any good, legally unencumbered, lossless compression method. IFF has also been suggested, but is not suitable in detail: available image representations are too machine-specific or not adequately compressed. The overall chunk structure of IFF is a useful concept that PNG has liberally borrowed from, but we did not attempt to be bit-for-bit compatible with IFF chunk structure. Again this is due to detailed issues, notably the fact that IFF FORMs are not designed to be serially writable. Lossless JPEG is not suitable because it does not provide for the storage of indexed-color images. Furthermore, its lossless truecolor compression is often inferior to that of PNG. Formatted and Indexed by: page 72 of 102 instructional media + magic, inc. RFC2083 Web Address: PNG: Portable Network Graphics http://sunsite.cnlab-switch.ch/ By: Boutell, et. al. 12.5. Byte order It has been asked why PNG uses network byte order. We have selected one byte ordering and used it consistently. Which order in particular is of little relevance, but network byte order has the advantage that routines to convert to and from it are already available on any platform that supports TCP/IP networking, including all PC platforms. The functions are trivial and will be included in the reference implementation. 12.6. Interlacing PNG's two-dimensional interlacing scheme is more complex to implement than GIF's line-wise interlacing. It also costs a little more in file size. However, it yields an initial image eight times faster than GIF (the first pass transmits only 1/64th of the pixels, compared to 1/8th for GIF). Although this initial image is coarse, it is useful in many situations. For example, if the image is a World Wide Web imagemap that the user has seen before, PNG's first pass is often enough to determine where to click. The PNG scheme also looks better than GIF's, because horizontal and vertical resolution never differ by more than a factor of two; this avoids the odd "stretched" look seen when interlaced GIFs are filled in by replicating scanlines. Preliminary results show that small text in an interlaced PNG image is typically readable about twice as fast as in an equivalent GIF, i.e., after PNG's fifth pass or 25% of the image data, instead of after GIF's third pass or 50%. This is again due to PNG's more balanced increase in resolution. 12.7. Why gamma? It might seem natural to standardize on storing sample values that are linearly proportional to light intensity (that is, have gamma of 1.0). But in fact, it is common for images to have a gamma of less than 1. There are three good reasons for this: * For reasons detailed in Gamma Tutorial (Chapter 13), all video cameras apply a "gamma correction" function to the intensity information. This causes the video signal to have a gamma of about 0.5 relative to the light intensity in the original scene. Thus, images obtained by frame-grabbing video already have a gamma of about 0.5. * The human eye has a nonlinear response to intensity, so linear encoding of samples either wastes sample codes in bright areas of the image, or provides too few sample codes to avoid banding artifacts in dark areas of the image, or both. At least 12 bits per sample are needed to avoid Formatted and Indexed by: page 73 of 102 instructional media + magic, inc. RFC2083 Web Address: PNG: Portable Network Graphics http://sunsite.cnlab-switch.ch/ By: Boutell, et. al. visible artifacts in linear encoding with a 100:1 image intensity range. An image gamma in the range 0.3 to 0.5 allocates sample values in a way that roughly corresponds to the eye's response, so that 8 bits/sample are enough to avoid artifacts caused by insufficient sample precision in almost all images. This makes "gamma encoding" a much better way of storing digital images than the simpler linear encoding. * Many images are created on PCs or workstations with no gamma correction hardware and no software willing to provide gamma correction either. In these cases, the images have had their lighting and color chosen to look best on this platform --- they can be thought of as having "manual" gamma correction built in. To see what the image author intended, it is necessary to treat such images as having a file_gamma value in the range 0.4-0.6, depending on the room lighting level that the author was working in. In practice, image gamma values around 1.0 and around 0.5 are both widely found. Older image standards such as GIF often do not account for this fact. The JFIF standard specifies that images in that format should use linear samples, but many JFIF images found on the Internet actually have a gamma somewhere near 0.4 or 0.5. The variety of images found and the variety of systems that people display them on have led to widespread problems with images appearing "too dark" or "too light". PNG expects viewers to compensate for image gamma at the time that the image is displayed. Another possible approach is to expect encoders to convert all images to a uniform gamma at encoding time. While that method would speed viewers slightly, it has fundamental flaws: * Gamma correction is inherently lossy due to quantization and roundoff error. Requiring conversion at encoding time thus causes irreversible loss. Since PNG is intended to be a lossless storage format, this is undesirable; we should store unmodified source data. * The encoder might not know the source gamma value. If the decoder does gamma correction at viewing time, it can adjust the gamma (change the displayed brightness) in response to feedback from a human user. The encoder has no such recourse. * Whatever "standard" gamma we settled on would be wrong for some displays. Hence viewers would still need gamma correction capability. Formatted and Indexed by: page 74 of 102 instructional media + magic, inc. RFC2083 Web Address: PNG: Portable Network Graphics http://sunsite.cnlab-switch.ch/ By: Boutell, et. al. Since there will always be images with no gamma or an incorrect recorded gamma, good viewers will need to incorporate gamma adjustment code anyway. Gamma correction at viewing time is thus the right way to go. See Gamma Tutorial (Chapter 13) for more information. 12.8. Non-premultiplied alpha PNG uses "unassociated" or "non-premultiplied" alpha so that images with separate transparency masks can be stored losslessly. Another common technique, "premultiplied alpha", stores pixel values premultiplied by the alpha fraction; in effect, the image is already composited against a black background. Any image data hidden by the transparency mask is irretrievably lost by that method, since multiplying by a zero alpha value always produces zero. Some image rendering techniques generate images with premultiplied alpha (the alpha value actually represents how much of the pixel is covered by the image). This representation can be converted to PNG by dividing the sample values by alpha, except where alpha is zero. The result will look good if displayed by a viewer that handles alpha properly, but will not look very good if the viewer ignores the alpha channel. Although each form of alpha storage has its advantages, we did not want to require all PNG viewers to handle both forms. We standardized on non-premultiplied alpha as being the lossless and more general case. 12.9. Filtering PNG includes filtering capability because filtering can significantly reduce the compressed size of truecolor and grayscale images. Filtering is also sometimes of value on indexed-color images, although this is less common. The filter algorithms are defined to operate on bytes, rather than pixels; this gains simplicity and speed with very little cost in compression performance. Tests have shown that filtering is usually ineffective for images with fewer than 8 bits per sample, so providing pixelwise filtering for such images would be pointless. For 16 bit/sample data, bytewise filtering is nearly as effective as pixelwise filtering, because MSBs are predicted from adjacent MSBs, and LSBs are predicted from adjacent LSBs. Formatted and Indexed by: page 75 of 102 instructional media + magic, inc. RFC2083 Web Address: PNG: Portable Network Graphics http://sunsite.cnlab-switch.ch/ By: Boutell, et. al. The encoder is allowed to change filters for each new scanline. This creates no additional complexity for decoders, since a decoder is required to contain defiltering logic for every filter type anyway. The only cost is an extra byte per scanline in the pre-compression datastream. Our tests showed that when the same filter is selected for all scanlines, this extra byte compresses away to almost nothing, so there is little storage cost compared to a fixed filter specified for the whole image. And the potential benefits of adaptive filtering are too great to ignore. Even with the simplistic filter-choice heuristics so far discovered, adaptive filtering usually outperforms fixed filters. In particular, an adaptive filter can change behavior for successive passes of an interlaced image; a fixed filter cannot. 12.10. Text strings Most graphics file formats include the ability to store some textual information along with the image. But many applications need more than that: they want to be able to store several identifiable pieces of text. For example, a database using PNG files to store medical X-rays would likely want to include patient's name, doctor's name, etc. A simple way to do this in PNG would be to invent new private chunks holding text. The disadvantage of such an approach is that other applications would have no idea what was in those chunks, and would simply ignore them. Instead, we recommend that textual information be stored in standard tEXt chunks with suitable keywords. Use of tEXt tells any PNG viewer that the chunk contains text that might be of interest to a human user. Thus, a person looking at the file with another viewer will still be able to see the text, and even understand what it is if the keywords are reasonably self- explanatory. (To this end, we recommend spelled-out keywords, not abbreviations that will be hard for a person to understand. Saving a few bytes on a keyword is false economy.) The ISO 8859-1 (Latin-1) character set was chosen as a compromise between functionality and portability. Some platforms cannot display anything more than 7-bit ASCII characters, while others can handle characters beyond the Latin-1 set. We felt that Latin-1 represents a widely useful and reasonably portable character set. Latin-1 is a direct subset of character sets commonly used on popular platforms such as Microsoft Windows and X Windows. It can also be handled on Macintosh systems with a simple remapping of characters. There is presently no provision for text employing character sets other than Latin-1. We recognize that the need for other character sets will increase. However, PNG already requires that Formatted and Indexed by: page 76 of 102 instructional media + magic, inc. RFC2083 Web Address: PNG: Portable Network Graphics http://sunsite.cnlab-switch.ch/ By: Boutell, et. al. programmers implement a number of new and unfamiliar features, and text representation is not PNG's primary purpose. Since PNG provides for the creation and public registration of new ancillary chunks of general interest, we expect that text chunks for other character sets, such as Unicode, eventually will be registered and increase gradually in popularity. 12.11. PNG file signature The first eight bytes of a PNG file always contain the following values: (decimal) 137 80 78 71 13 10 26 10 (hexadecimal) 89 50 4e 47 0d 0a 1a 0a (ASCII C notation) \211 P N G \r \n \032 \n This signature both identifies the file as a PNG file and provides for immediate detection of common file-transfer problems. The first two bytes distinguish PNG files on systems that expect the first two bytes to identify the file type uniquely. The first byte is chosen as a non-ASCII value to reduce the probability that a text file may be misrecognized as a PNG file; also, it catches bad file transfers that clear bit 7. Bytes two through four name the format. The CR-LF sequence catches bad file transfers that alter newline sequences. The control-Z character stops file display under MS-DOS. The final line feed checks for the inverse of the CR-LF translation problem. A decoder may further verify that the next eight bytes contain an IHDR chunk header with the correct chunk length; this will catch bad transfers that drop or alter null (zero) bytes. Note that there is no version number in the signature, nor indeed anywhere in the file. This is intentional: the chunk mechanism provides a better, more flexible way to handle format extensions, as explained in Chunk naming conventions (Section 12.13). 12.12. Chunk layout The chunk design allows decoders to skip unrecognized or uninteresting chunks: it is simply necessary to skip the appropriate number of bytes, as determined from the length field. Limiting chunk length to (2^31)-1 bytes avoids possible problems for implementations that cannot conveniently handle 4-byte unsigned values. In practice, chunks will usually be much shorter than that anyway. Formatted and Indexed by: page 77 of 102 instructional media + magic, inc. RFC2083 Web Address: PNG: Portable Network Graphics http://sunsite.cnlab-switch.ch/ By: Boutell, et. al. A separate CRC is provided for each chunk in order to detect badly-transferred images as quickly as possible. In particular, critical data such as the image dimensions can be validated before being used. The chunk length is excluded from the CRC so that the CRC can be calculated as the data is generated; this avoids a second pass over the data in cases where the chunk length is not known in advance. Excluding the length from the CRC does not create any extra risk of failing to discover file corruption, since if the length is wrong, the CRC check will fail: the CRC will be computed on the wrong set of bytes and then be tested against the wrong value from the file. 12.13. Chunk naming conventions The chunk naming conventions allow safe, flexible extension of the PNG format. This mechanism is much better than a format version number, because it works on a feature-by-feature basis rather than being an overall indicator. Decoders can process newer files if and only if the files use no unknown critical features (as indicated by finding unknown critical chunks). Unknown ancillary chunks can be safely ignored. We decided against having an overall format version number because experience has shown that format version numbers hurt portability as much as they help. Version numbers tend to be set unnecessarily high, leading to older decoders rejecting files that they could have processed (this was a serious problem for several years after the GIF89 spec came out, for example). Furthermore, private extensions can be made either critical or ancillary, and standard decoders should react appropriately; overall version numbers are no help for private extensions. A hypothetical chunk for vector graphics would be a critical chunk, since if ignored, important parts of the intended image would be missing. A chunk carrying the Mandelbrot set coordinates for a fractal image would be ancillary, since other applications could display the image without understanding what the image represents. In general, a chunk type should be made critical only if it is impossible to display a reasonable representation of the intended image without interpreting that chunk. Formatted and Indexed by: page 78 of 102 instructional media + magic, inc. RFC2083 Web Address: PNG: Portable Network Graphics http://sunsite.cnlab-switch.ch/ By: Boutell, et. al. The public/private property bit ensures that any newly defined public chunk type name cannot conflict with proprietary chunks that could be in use somewhere. However, this does not protect users of private chunk names from the possibility that someone else may use the same chunk name for a different purpose. It is a good idea to put additional identifying information at the start of the data for any private chunk type. When a PNG file is modified, certain ancillary chunks may need to be changed to reflect changes in other chunks. For example, a histogram chunk needs to be changed if the image data changes. If the file editor does not recognize histogram chunks, copying them blindly to a new output file is incorrect; such chunks should be dropped. The safe/unsafe property bit allows ancillary chunks to be marked appropriately. Not all possible modification scenarios are covered by the safe/unsafe semantics. In particular, chunks that are dependent on the total file contents are not supported. (An example of such a chunk is an index of IDAT chunk locations within the file: adding a comment chunk would inadvertently break the index.) Definition of such chunks is discouraged. If absolutely necessary for a particular application, such chunks can be made critical chunks, with consequent loss of portability to other applications. In general, ancillary chunks can depend on critical chunks but not on other ancillary chunks. It is expected that mutually dependent information should be put into a single chunk. In some situations it may be unavoidable to make one ancillary chunk dependent on another. Although the chunk property bits are insufficient to represent this case, a simple solution is available: in the dependent chunk, record the CRC of the chunk depended on. It can then be determined whether that chunk has been changed by some other program. The same technique can be useful for other purposes. For example, if a program relies on the palette being in a particular order, it can store a private chunk containing the CRC of the PLTE chunk. If this value matches when the file is again read in, then it provides high confidence that the palette has not been tampered with. Note that it is not necessary to mark the private chunk unsafe-to-copy when this technique is used; thus, such a private chunk can survive other editing of the file. Formatted and Indexed by: page 79 of 102 instructional media + magic, inc. RFC2083 Web Address: PNG: Portable Network Graphics http://sunsite.cnlab-switch.ch/ By: Boutell, et. al. 12.14. Palette histograms A viewer may not be able to provide as many colors as are listed in the image's palette. (For example, some colors could be reserved by a window system.) To produce the best results in this situation, it is helpful to have information about the frequency with which each palette index actually appears, in order to choose the best palette for dithering or to drop the least-used colors. Since images are often created once and viewed many times, it makes sense to calculate this information in the encoder, although it is not mandatory for the encoder to provide it. Other image formats have usually addressed this problem by specifying that the palette entries should appear in order of frequency of use. That is an inferior solution, because it doesn't give the viewer nearly as much information: the viewer can't determine how much damage will be done by dropping the last few colors. Nor does a sorted palette give enough information to choose a target palette for dithering, in the case that the viewer needs to reduce the number of colors substantially. A palette histogram provides the information needed to choose such a target palette without making a pass over the image data. Formatted and Indexed by: page 80 of 102 instructional media + magic, inc. RFC2083 Web Address: PNG: Portable Network Graphics http://sunsite.cnlab-switch.ch/ By: Boutell, et. al. 13. Appendix: Gamma Tutorial (This appendix is not part of the formal PNG specification.) It would be convenient for graphics programmers if all of the components of an imaging system were linear. The voltage coming from an electronic camera would be directly proportional to the intensity (power) of light in the scene, the light emitted by a CRT would be directly proportional to its input voltage, and so on. However, real-world devices do not behave in this way. All CRT displays, almost all photographic film, and many electronic cameras have nonlinear signal-to-light-intensity or intensity-to-signal characteristics. Fortunately, all of these nonlinear devices have a transfer function that is approximated fairly well by a single type of mathematical function: a power function. This power function has the general equation output = input ^ gamma where ^ denotes exponentiation, and "gamma" (often printed using the Greek letter gamma, thus the name) is simply the exponent of the power function. By convention, "input" and "output" are both scaled to the range 0..1, with 0 representing black and 1 representing maximum white (or red, etc). Normalized in this way, the power function is completely described by a single number, the exponent "gamma". So, given a particular device, we can measure its output as a function of its input, fit a power function to this measured transfer function, extract the exponent, and call it gamma. We often say "this device has a gamma of 2.5" as a shorthand for "this device has a power-law response with an exponent of 2.5". We can also talk about the gamma of a mathematical transform, or of a lookup table in a frame buffer, so long as the input and output of the thing are related by the power-law expression above. How do gammas combine? Real imaging systems will have several components, and more than one of these can be nonlinear. If all of the components have transfer characteristics that are power functions, then the transfer function of the entire system is also a power function. The exponent (gamma) of the whole system's transfer function is just the product of all of the individual exponents (gammas) of the separate stages in the system. Formatted and Indexed by: page 81 of 102 instructional media + magic, inc. RFC2083 Web Address: PNG: Portable Network Graphics http://sunsite.cnlab-switch.ch/ By: Boutell, et. al. Also, stages that are linear pose no problem, since a power function with an exponent of 1.0 is really a linear function. So a linear transfer function is just a special case of a power function, with a gamma of 1.0. Thus, as long as our imaging system contains only stages with linear and power-law transfer functions, we can meaningfully talk about the gamma of the entire system. This is indeed the case with most real imaging systems. What should overall gamma be? If the overall gamma of an imaging system is 1.0, its output is linearly proportional to its input. This means that the ratio between the intensities of any two areas in the reproduced image will be the same as it was in the original scene. It might seem that this should always be the goal of an imaging system: to accurately reproduce the tones of the original scene. Alas, that is not the case. When the reproduced image is to be viewed in "bright surround" conditions, where other white objects nearby in the room have about the same brightness as white in the image, then an overall gamma of 1.0 does indeed give real-looking reproduction of a natural scene. Photographic prints viewed under room light and computer displays in bright room light are typical "bright surround" viewing conditions. However, sometimes images are intended to be viewed in "dark surround" conditions, where the room is substantially black except for the image. This is typical of the way movies and slides (transparencies) are viewed by projection. Under these circumstances, an accurate reproduction of the original scene results in an image that human viewers judge as "flat" and lacking in contrast. It turns out that the projected image needs to have a gamma of about 1.5 relative to the original scene for viewers to judge it "natural". Thus, slide film is designed to have a gamma of about 1.5, not 1.0. There is also an intermediate condition called "dim surround", where the rest of the room is still visible to the viewer, but is noticeably darker than the reproduced image itself. This is typical of television viewing, at least in the evening, as well as subdued-light computer work areas. In dim surround conditions, the reproduced image needs to have a gamma of about 1.25 relative to the original scene in order to look natural. Formatted and Indexed by: page 82 of 102 instructional media + magic, inc. RFC2083 Web Address: PNG: Portable Network Graphics http://sunsite.cnlab-switch.ch/ By: Boutell, et. al. The requirement for boosted contrast (gamma) in dark surround conditions is due to the way the human visual system works, and applies equally well to computer monitors. Thus, a PNG viewer trying to achieve the maximum realism for the images it displays really needs to know what the room lighting conditions are, and adjust the gamma of the displayed image accordingly. If asking the user about room lighting conditions is inappropriate or too difficult, just assume that the overall gamma (viewing_gamma as defined below) should be 1.0 or 1.25. That's all that most systems that implement gamma correction do. What is a CRT's gamma? All CRT displays have a power-law transfer characteristic with a gamma of about 2.5. This is due to the physical processes involved in controlling the electron beam in the electron gun, and has nothing to do with the phosphor. An exception to this rule is fancy "calibrated" CRTs that have internal electronics to alter their transfer function. If you have one of these, you probably should believe what the manufacturer tells you its gamma is. But in all other cases, assuming 2.5 is likely to be pretty accurate. There are various images around that purport to measure gamma, usually by comparing the intensity of an area containing alternating white and black with a series of areas of continuous gray of different intensity. These are usually not reliable. Test images that use a "checkerboard" pattern of black and white are the worst, because a single white pixel will be reproduced considerably darker than a large area of white. An image that uses alternating black and white horizontal lines (such as the "gamma.png" test image at ftp://ftp.uu.net/graphics/png/images/suite/gamma.png) is much better, but even it may be inaccurate at high "picture" settings on some CRTs. If you have a good photometer, you can measure the actual light output of a CRT as a function of input voltage and fit a power function to the measurements. However, note that this procedure is very sensitive to the CRT's black level adjustment, somewhat sensitive to its picture adjustment, and also affected by ambient light. Furthermore, CRTs spread some light from bright areas of an image into nearby darker areas; a single bright spot against a black background may be seen to have a "halo". Your measuring technique will need to minimize the effects of this. Formatted and Indexed by: page 83 of 102 instructional media + magic, inc. RFC2083 Web Address: PNG: Portable Network Graphics http://sunsite.cnlab-switch.ch/ By: Boutell, et. al. Because of the difficulty of measuring gamma, using either test images or measuring equipment, you're usually better off just assuming gamma is 2.5 rather than trying to measure it. What is gamma correction? A CRT has a gamma of 2.5, and we can't change that. To get an overall gamma of 1.0 (or somewhere near that) for an imaging system, we need to have at least one other component of the "image pipeline" that is nonlinear. If, in fact, there is only one nonlinear stage in addition to the CRT, then it's traditional to say that the CRT has a certain gamma, and that the other nonlinear stage provides "gamma correction" to compensate for the CRT. However, exactly where the "correction" is done depends on circumstance. In all broadcast video systems, gamma correction is done in the camera. This choice was made in the days when television electronics were all analog, and a good gamma-correction circuit was expensive to build. The original NTSC video standard required cameras to have a transfer function with a gamma of 1/2.2, or about 0.45. Recently, a more complex two-part transfer function has been adopted [SMPTE-170M], but its behavior can be well approximated by a power function with a gamma of 0.5. When the resulting image is displayed on a CRT with a gamma of 2.5, the image on screen ends up with a gamma of about 1.25 relative to the original scene, which is appropriate for "dim surround" viewing. These days, video signals are often digitized and stored in computer frame buffers. This works fine, but remember that gamma correction is "built into" the video signal, and so the digitized video has a gamma of about 0.5 relative to the original scene. Computer rendering programs often produce linear samples. To display these correctly, intensity on the CRT needs to be directly proportional to the sample values in the frame buffer. This can be done with a special hardware lookup table between the frame buffer and the CRT hardware. The lookup table (often called LUT) is loaded with a mapping that implements a power function with a gamma of 0.4, thus providing "gamma correction" for the CRT gamma. Thus, gamma correction sometimes happens before the frame buffer, sometimes after. As long as images created in a particular environment are always displayed in that environment, everything is fine. But when people try to exchange images, differences in gamma correction conventions often result in images that seem far too bright and washed out, or far too dark and contrasty. Formatted and Indexed by: page 84 of 102 instructional media + magic, inc. RFC2083 Web Address: PNG: Portable Network Graphics http://sunsite.cnlab-switch.ch/ By: Boutell, et. al. Gamma-encoded samples are good So, is it better to do gamma correction before or after the frame buffer? In an ideal world, sample values would be stored in floating point, there would be lots of precision, and it wouldn't really matter much. But in reality, we're always trying to store images in as few bits as we can. If we decide to use samples that are linearly proportional to intensity, and do the gamma correction in the frame buffer LUT, it turns out that we need to use at least 12 bits for each of red, green, and blue to have enough precision in intensity. With any less than that, we will sometimes see "contour bands" or "Mach bands" in the darker areas of the image, where two adjacent sample values are still far enough apart in intensity for the difference to be visible. However, through an interesting coincidence, the human eye's subjective perception of brightness is related to the physical stimulation of light intensity in a manner that is very much like the power function used for gamma correction. If we apply gamma correction to measured (or calculated) light intensity before quantizing to an integer for storage in a frame buffer, we can get away with using many fewer bits to store the image. In fact, 8 bits per color is almost always sufficient to avoid contouring artifacts. This is because, since gamma correction is so closely related to human perception, we are assigning our 256 available sample codes to intensity values in a manner that approximates how visible those intensity changes are to the eye. Compared to a linear-sample image, we allocate fewer sample values to brighter parts of the tonal range and more sample values to the darker portions of the tonal range. Thus, for the same apparent image quality, images using gamma- encoded sample values need only about two-thirds as many bits of storage as images using linear samples. Formatted and Indexed by: page 85 of 102 instructional media + magic, inc. RFC2083 Web Address: PNG: Portable Network Graphics http://sunsite.cnlab-switch.ch/ By: Boutell, et. al. General gamma handling When more than two nonlinear transfer functions are involved in the image pipeline, the term "gamma correction" becomes too vague. If we consider a pipeline that involves capturing (or calculating) an image, storing it in an image file, reading the file, and displaying the image on some sort of display screen, there are at least 5 places in the pipeline that could have nonlinear transfer functions. Let's give each a specific name for their characteristic gamma: camera_gamma the characteristic of the image sensor encoding_gamma the gamma of any transformation performed by the software writing the image file decoding_gamma the gamma of any transformation performed by the software reading the image file LUT_gamma the gamma of the frame buffer LUT, if present CRT_gamma the gamma of the CRT, generally 2.5 In addition, let's add a few other names: file_gamma the gamma of the image in the file, relative to the original scene. This is file_gamma = camera_gamma * encoding_gamma display_gamma the gamma of the "display system" downstream of the frame buffer. This is display_gamma = LUT_gamma * CRT_gamma viewing_gamma the overall gamma that we want to obtain to produce pleasing images --- generally 1.0 to 1.5. Formatted and Indexed by: page 86 of 102 instructional media + magic, inc. RFC2083 Web Address: PNG: Portable Network Graphics http://sunsite.cnlab-switch.ch/ By: Boutell, et. al. The file_gamma value, as defined above, is what goes in the gAMA chunk in a PNG file. If file_gamma is not 1.0, we know that gamma correction has been done on the sample values in the file, and we could call them "gamma corrected" samples. However, since there can be so many different values of gamma in the image display chain, and some of them are not known at the time the image is written, the samples are not really being "corrected" for a specific display condition. We are really using a power function in the process of encoding an intensity range into a small integer field, and so it is more correct to say "gamma encoded" samples instead of "gamma corrected" samples. When displaying an image file, the image decoding program is responsible for making the overall gamma of the system equal to the desired viewing_gamma, by selecting the decoding_gamma appropriately. When displaying a PNG file, the gAMA chunk provides the file_gamma value. The display_gamma may be known for this machine, or it might be obtained from the system software, or the user might have to be asked what it is. The correct viewing_gamma depends on lighting conditions, and that will generally have to come from the user. Ultimately, you should have file_gamma * decoding_gamma * display_gamma = viewing_gamma Some specific examples In digital video systems, camera_gamma is about 0.5 by declaration of the various video standards documents. CRT_gamma is 2.5 as usual, while encoding_gamma, decoding_gamma, and LUT_gamma are all 1.0. As a result, viewing_gamma ends up being about 1.25. On frame buffers that have hardware gamma correction tables, and that are calibrated to display linear samples correctly, display_gamma is 1.0. Many workstations and X terminals and PC displays lack gamma correction lookup tables. Here, LUT_gamma is always 1.0, so display_gamma is 2.5. Formatted and Indexed by: page 87 of 102 instructional media + magic, inc. RFC2083 Web Address: PNG: Portable Network Graphics http://sunsite.cnlab-switch.ch/ By: Boutell, et. al. On the Macintosh, there is a LUT. By default, it is loaded with a table whose gamma is about 0.72, giving a display_gamma (LUT and CRT combined) of about 1.8. Some Macs have a "Gamma" control panel that allows gamma to be changed to 1.0, 1.2, 1.4, 1.8, or 2.2. These settings load alternate LUTs that are designed to give a display_gamma that is equal to the label on the selected button. Thus, the "Gamma" control panel setting can be used directly as display_gamma in decoder calculations. On recent SGI systems, there is a hardware gamma-correction table whose contents are controlled by the (privileged) "gamma" program. The gamma of the table is actually the reciprocal of the number that "gamma" prints, and it does not include the CRT gamma. To obtain the display_gamma, you need to find the SGI system gamma (either by looking in a file, or asking the user) and then calculating display_gamma = 2.5 / SGI_system_gamma You will find SGI systems with the system gamma set to 1.0 and 2.2 (or higher), but the default when machines are shipped is 1.7. A note about video gamma The original NTSC video standards specified a simple power-law camera transfer function with a gamma of 1/2.2 or 0.45. This is not possible to implement exactly in analog hardware because the function has infinite slope at x=0, so all cameras deviated to some degree from this ideal. More recently, a new camera transfer function that is physically realizable has been accepted as a standard [SMPTE-170M]. It is Vout = 4.5 * Vin if Vin < 0.018 Vout = 1.099 * (Vin^0.45) - 0.099 if Vin >= 0.018 where Vin and Vout are measured on a scale of 0 to 1. Although the exponent remains 0.45, the multiplication and subtraction change the shape of the transfer function, so it is no longer a pure power function. If you want to perform extremely precise calculations on video signals, you should use the expression above (or its inverse, as required). However, PNG does not provide a way to specify that an image uses this exact transfer function; the gAMA chunk always assumes a pure power-law function. If we plot the two-part transfer function above along with the family of pure power functions, we find that a power function with a gamma of about 0.5 to 0.52 (not 0.45) most closely approximates the transfer function. Thus, when writing a Formatted and Indexed by: page 88 of 102 instructional media + magic, inc. RFC2083 Web Address: PNG: Portable Network Graphics http://sunsite.cnlab-switch.ch/ By: Boutell, et. al. PNG file with data obtained from digitizing the output of a modern video camera, the gAMA chunk should contain 0.5 or 0.52, not 0.45. The remaining difference between the true transfer function and the power function is insignificant for almost all purposes. (In fact, the alignment errors in most cameras are likely to be larger than the difference between these functions.) The designers of PNG deemed the simplicity and flexibility of a power-law definition of gAMA to be more important than being able to describe the SMPTE-170M transfer curve exactly. The PAL and SECAM video standards specify a power-law camera transfer function with a gamma of 1/2.8 or 0.36 --- not the 1/2.2 of NTSC. However, this is too low in practice, so real cameras are likely to have their gamma set close to NTSC practice. Just guessing 0.45 or 0.5 is likely to give you viewable results, but if you want precise values you'll probably have to measure the particular camera. Further reading If you have access to the World Wide Web, read Charles Poynton's excellent "Gamma FAQ" [GAMMA-FAQ] for more information about gamma. 14. Appendix: Color Tutorial (This appendix is not part of the formal PNG specification.) About chromaticity The cHRM chunk is used, together with the gAMA chunk, to convey precise color information so that a PNG image can be displayed or printed with better color fidelity than is possible without this information. The preceding chapters state how this information is encoded in a PNG image. This tutorial briefly outlines the underlying color theory for those who might not be familiar with it. Note that displaying an image with incorrect gamma will produce much larger color errors than failing to use the chromaticity data. First be sure the monitor set-up and gamma correction are right, then worry about chromaticity. The problem The color of an object depends not only on the precise spectrum of light emitted or reflected from it, but also on the observer --- their species, what else they can see at the same time, even what Formatted and Indexed by: page 89 of 102 instructional media + magic, inc. RFC2083 Web Address: PNG: Portable Network Graphics http://sunsite.cnlab-switch.ch/ By: Boutell, et. al. they have recently looked at! Furthermore, two very different spectra can produce exactly the same color sensation. Color is not an objective property of real-world objects; it is a subjective, biological sensation. However, by making some simplifying assumptions (such as: we are talking about human vision) it is possible to produce a mathematical model of color and thereby obtain good color accuracy. Device-dependent color Display the same RGB data on three different monitors, side by side, and you will get a noticeably different color balance on each display. This is because each monitor emits a slightly different shade and intensity of red, green, and blue light. RGB is an example of a device-dependent color model --- the color you get depends on the device. This also means that a particular color --- represented as say RGB 87, 146, 116 on one monitor --- might have to be specified as RGB 98, 123, 104 on another to produce the same color. Device-independent color A full physical description of a color would require specifying the exact spectral power distribution of the light source. Fortunately, the human eye and brain are not so sensitive as to require exact reproduction of a spectrum. Mathematical, device- independent color models exist that describe fairly well how a particular color will be seen by humans. The most important device-independent color model, to which all others can be related, was developed by the International Lighting Committee (CIE, in French) and is called XYZ. In XYZ, X is the sum of a weighted power distribution over the whole visible spectrum. So are Y and Z, each with different weights. Thus any arbitrary spectral power distribution is condensed down to just three floating point numbers. The weights were derived from color matching experiments done on human subjects in the 1920s. CIE XYZ has been an International Standard since 1931, and it has a number of useful properties: * two colors with the same XYZ values will look the same to humans * two colors with different XYZ values will not look the same * the Y value represents all the brightness information (luminance) * the XYZ color of any object can be objectively measured Formatted and Indexed by: page 90 of 102 instructional media + magic, inc. RFC2083 Web Address: PNG: Portable Network Graphics http://sunsite.cnlab-switch.ch/ By: Boutell, et. al. Color models based on XYZ have been used for many years by people who need accurate control of color --- lighting engineers for film and TV, paint and dyestuffs manufacturers, and so on. They are thus proven in industrial use. Accurate, device-independent color started to spread from high-end, specialized areas into the mainstream during the late 1980s and early 1990s, and PNG takes notice of that trend. Calibrated, device-dependent color Traditionally, image file formats have used uncalibrated, device- dependent color. If the precise details of the original display device are known, it becomes possible to convert the device- dependent colors of a particular image to device-independent ones. Making simplifying assumptions, such as working with CRTs (which are much easier than printers), all we need to know are the XYZ values of each primary color and the CRT_gamma. So why does PNG not store images in XYZ instead of RGB? Well, two reasons. First, storing images in XYZ would require more bits of precision, which would make the files bigger. Second, all programs would have to convert the image data before viewing it. Whether calibrated or not, all variants of RGB are close enough that undemanding viewers can get by with simply displaying the data without color correction. By storing calibrated RGB, PNG retains compatibility with existing programs that expect RGB data, yet provides enough information for conversion to XYZ in applications that need precise colors. Thus, we get the best of both worlds. What are chromaticity and luminance? Chromaticity is an objective measurement of the color of an object, leaving aside the brightness information. Chromaticity uses two parameters x and y, which are readily calculated from XYZ: x = X / (X + Y + Z) y = Y / (X + Y + Z) XYZ colors having the same chromaticity values will appear to have the same hue but can vary in absolute brightness. Notice that x,y are dimensionless ratios, so they have the same values no matter what units we've used for X,Y,Z. Formatted and Indexed by: page 91 of 102 instructional media + magic, inc. RFC2083 Web Address: PNG: Portable Network Graphics http://sunsite.cnlab-switch.ch/ By: Boutell, et. al. The Y value of an XYZ color is directly proportional to its absolute brightness and is called the luminance of the color. We can describe a color either by XYZ coordinates or by chromaticity x,y plus luminance Y. The XYZ form has the advantage that it is linearly related to (linear, gamma=1.0) RGB color spaces. How are computer monitor colors described? The "white point" of a monitor is the chromaticity x,y of the monitor's nominal white, that is, the color produced when R=G=B=maximum. It's customary to specify monitor colors by giving the chromaticities of the individual phosphors R, G, and B, plus the white point. The white point allows one to infer the relative brightnesses of the three phosphors, which isn't determined by their chromaticities alone. Note that the absolute brightness of the monitor is not specified. For computer graphics work, we generally don't care very much about absolute brightness levels. Instead of dealing with absolute XYZ values (in which X,Y,Z are expressed in physical units of radiated power, such as candelas per square meter), it is convenient to work in "relative XYZ" units, where the monitor's nominal white is taken to have a luminance (Y) of 1.0. Given this assumption, it's simple to compute XYZ coordinates for the monitor's white, red, green, and blue from their chromaticity values. Why does cHRM use x,y rather than XYZ? Simply because that is how manufacturers print the information in their spec sheets! Usually, the first thing a program will do is convert the cHRM chromaticities into relative XYZ space. What can I do with it? If a PNG file has the gAMA and cHRM chunks, the source_RGB values can be converted to XYZ. This lets you: * do accurate grayscale conversion (just use the Y component) * convert to RGB for your own monitor (to see the original colors) * print the image in Level 2 PostScript with better color fidelity than a simple RGB to CMYK conversion could provide * calculate an optimal color palette * pass the image data to a color management system * etc. Formatted and Indexed by: page 92 of 102 instructional media + magic, inc. RFC2083 Web Address: PNG: Portable Network Graphics http://sunsite.cnlab-switch.ch/ By: Boutell, et. al. How do I convert from source_RGB to XYZ? Make a few simplifying assumptions first, like the monitor really is jet black with no input and the guns don't interfere with one another. Then, given that you know the CIE XYZ values for each of red, green, and blue for a particular monitor, you put them into a matrix m: Xr Xg Xb m = Yr Yg Yb Zr Zg Zb Here we assume we are working with linear RGB floating point data in the range 0..1. If the gamma is not 1.0, make it so on the floating point data. Then convert source_RGB to XYZ by matrix multiplication: X R Y = m G Z B In other words, X = Xr*R + Xg*G + Xb*B, and similarly for Y and Z. You can go the other way too: R X G = im Y B Z where im is the inverse of the matrix m. What is a gamut? The gamut of a device is the subset of visible colors which that device can display. (It has nothing to do with gamma.) The gamut of an RGB device can be visualized as a polyhedron in XYZ space; the vertices correspond to the device's black, blue, red, green, magenta, cyan, yellow and white. Different devices have different gamuts, in other words one device will be able to display certain colors (usually highly saturated ones) that another device cannot. The gamut of a particular RGB device can be determined from its R, G, and B chromaticities and white point (the same values given in the cHRM chunk). The gamut of a color printer is more complex and can only be determined by measurement. However, printer gamuts are typically smaller than monitor gamuts, meaning that there can be many colors in a displayable image that cannot physically be printed. Formatted and Indexed by: page 93 of 102 instructional media + magic, inc. RFC2083 Web Address: PNG: Portable Network Graphics http://sunsite.cnlab-switch.ch/ By: Boutell, et. al. Converting image data from one device to another generally results in gamut mismatches --- colors that cannot be represented exactly on the destination device. The process of making the colors fit, which can range from a simple clip to elaborate nonlinear scaling transformations, is termed gamut mapping. The aim is to produce a reasonable visual representation of the original image. Further reading References [COLOR-1] through [COLOR-5] provide more detail about color theory. 15. Appendix: Sample CRC Code The following sample code represents a practical implementation of the CRC (Cyclic Redundancy Check) employed in PNG chunks. (See also ISO 3309 [ISO-3309] or ITU-T V.42 [ITU-V42] for a formal specification.) The sample code is in the ANSI C programming language. Non C users may find it easier to read with these hints: & Bitwise AND operator. ^ Bitwise exclusive-OR operator. (Caution: elsewhere in this document, ^ represents exponentiation.) >> Bitwise right shift operator. When applied to an unsigned quantity, as here, right shift inserts zeroes at the left. ! Logical NOT operator. ++ "n++" increments the variable n. 0xNNN 0x introduces a hexadecimal (base 16) constant. Suffix L indicates a long value (at least 32 bits). /* Table of CRCs of all 8-bit messages. */ unsigned long crc_table[256]; /* Flag: has the table been computed? Initially false. */ int crc_table_computed = 0; Formatted and Indexed by: page 94 of 102 instructional media + magic, inc. RFC2083 Web Address: PNG: Portable Network Graphics http://sunsite.cnlab-switch.ch/ By: Boutell, et. al. /* Make the table for a fast CRC. */ void make_crc_table(void) { unsigned long c; int n, k; for (n = 0; n < 256; n++) { c = (unsigned long) n; for (k = 0; k < 8; k++) { if (c & 1) c = 0xedb88320L ^ (c >> 1); else c = c >> 1; } crc_table[n] = c; } crc_table_computed = 1; } /* Update a running CRC with the bytes buf[0..len-1]--the CRC should be initialized to all 1's, and the transmitted value is the 1's complement of the final running CRC (see the crc() routine below)). */ unsigned long update_crc(unsigned long crc, unsigned char *buf, int len) { unsigned long c = crc; int n; if (!crc_table_computed) make_crc_table(); for (n = 0; n < len; n++) { c = crc_table[(c ^ buf[n]) & 0xff] ^ (c >> 8); } return c; } /* Return the CRC of the bytes buf[0..len-1]. */ unsigned long crc(unsigned char *buf, int len) { return update_crc(0xffffffffL, buf, len) ^ 0xffffffffL; } Formatted and Indexed by: page 95 of 102 instructional media + magic, inc. RFC2083 Web Address: PNG: Portable Network Graphics http://sunsite.cnlab-switch.ch/ By: Boutell, et. al. 16. Appendix: Online Resources (This appendix is not part of the formal PNG specification.) This appendix gives the locations of some Internet resources for PNG software developers. By the nature of the Internet, the list is incomplete and subject to change. Archive sites The latest released versions of this document and related information can always be found at the PNG FTP archive site, ftp://ftp.uu.net/graphics/png/. The PNG specification is available in several formats, including HTML, plain text, and PostScript. Reference implementation and test images A reference implementation in portable C is available from the PNG FTP archive site, ftp://ftp.uu.net/graphics/png/src/. The reference implementation is freely usable in all applications, including commercial applications. Test images are available from ftp://ftp.uu.net/graphics/png/images/. Electronic mail The maintainers of the PNG specification can be contacted by e- mail at [email protected] or at [email protected]. PNG home page There is a World Wide Web home page for PNG at http://quest.jpl.nasa.gov/PNG/. This page is a central location for current information about PNG and PNG-related tools. 17. Appendix: Revision History (This appendix is not part of the formal PNG specification.) The PNG format has been frozen since the Ninth Draft of March 7, 1995, and all future changes are intended to be backwards compatible. The revisions since the Ninth Draft are simply clarifications, improvements in presentation, and additions of supporting material. On 1 October 1996, the PNG specification was approved as a W3C (World Formatted and Indexed by: page 96 of 102 instructional media + magic, inc. RFC2083 Web Address: PNG: Portable Network Graphics http://sunsite.cnlab-switch.ch/ By: Boutell, et. al. Wide Web Consortium) Recommendation. Changes since the Tenth Draft of 5 May, 1995 * Clarified meaning of a suggested-palette PLTE chunk in a truecolor image that uses transparency * Clarified exact semantics of sBIT and allowed sample depth scaling procedures * Clarified status of spaces in tEXt chunk keywords * Distinguished private and public extension values in type and method fields * Added a "Creation Time" tEXt keyword * Macintosh representation of PNG specified * Added discussion of security issues * Added more extensive discussion of gamma and chromaticity handling, including tutorial appendixes * Clarified terminology, notably sample depth vs. bit depth * Added a glossary * Editing and reformatting 18. References [COLOR-1] Hall, Roy, Illumination and Color in Computer Generated Imagery. Springer-Verlag, New York, 1989. ISBN 0-387-96774-5. [COLOR-2] Kasson, J., and W. Plouffe, "An Analysis of Selected Computer Interchange Color Spaces", ACM Transactions on Graphics, vol 11 no 4 (1992), pp 373-405. [COLOR-3] Lilley, C., F. Lin, W.T. Hewitt, and T.L.J. Howard, Colour in Computer Graphics. CVCP, Sheffield, 1993. ISBN 1-85889-022-5. Also available from [COLOR-4] Stone, M.C., W.B. Cowan, and J.C. Beatty, "Color gamut mapping and the printing of digital images", ACM Transactions on Graphics, vol 7 no 3 (1988), pp 249-292. [COLOR-5] Travis, David, Effective Color Displays --- Theory and Practice. Academic Press, London, 1991. ISBN 0-12-697690-2. [GAMMA-FAQ] Poynton, C., "Gamma FAQ". Formatted and Indexed by: page 97 of 102 instructional media + magic, inc. RFC2083 Web Address: PNG: Portable Network Graphics http://sunsite.cnlab-switch.ch/ By: Boutell, et. al. [ISO-3309] International Organization for Standardization, "Information Processing Systems --- Data Communication High-Level Data Link Control Procedure --- Frame Structure", IS 3309, October 1984, 3rd Edition. [ISO-8859] International Organization for Standardization, "Information Processing --- 8-bit Single-Byte Coded Graphic Character Sets --- Part 1: Latin Alphabet No. 1", IS 8859-1, 1987. Also see sample files at ftp://ftp.uu.net/graphics/png/documents/iso_8859-1.* [ITU-BT709] International Telecommunications Union, "Basic Parameter Values for the HDTV Standard for the Studio and for International Programme Exchange", ITU-R Recommendation BT.709 (formerly CCIR Rec. 709), 1990. [ITU-V42] International Telecommunications Union, "Error-correcting Procedures for DCEs Using Asynchronous-to-Synchronous Conversion", ITU-T Recommendation V.42, 1994, Rev. 1. [PAETH] Paeth, A.W., "Image File Compression Made Easy", in Graphics Gems II, James Arvo, editor. Academic Press, San Diego, 1991. ISBN 0-12-064480-0. [POSTSCRIPT] Adobe Systems Incorporated, PostScript Language Reference Manual, 2nd edition. Addison-Wesley, Reading, 1990. ISBN 0-201-18127-4. [PNG-EXTENSIONS] PNG Group, "PNG Special-Purpose Public Chunks". Available in several formats from ftp://ftp.uu.net/graphics/png/documents/pngextensions.* [RFC-1123] Braden, R., Editor, "Requirements for Internet Hosts --- Application and Support", STD 3, RFC 1123, USC/Information Sciences Institute, October 1989. Formatted and Indexed by: page 98 of 102 instructional media + magic, inc. RFC2083 Web Address: PNG: Portable Network Graphics http://sunsite.cnlab-switch.ch/ By: Boutell, et. al. [RFC-2045] Freed, N., and N. Borenstein, "Multipurpose Internet Mail Extensions (MIME) Part One: Format of Internet Message Bodies", RFC 2045, Innosoft, First Virtual, November 1996. [RFC-2048] Freed, N., Klensin, J., and J. Postel, "Multipurpose Internet Mail Extensions (MIME) Part Four: Registration Procedures", RFC 2048, Innosoft, MCI, USC/Information Sciences Institute, November 1996. [RFC-1950] Deutsch, P. and J-L. Gailly, "ZLIB Compressed Data Format Specification version 3.3", RFC 1950, Aladdin Enterprises, May 1996. [RFC-1951] Deutsch, P., "DEFLATE Compressed Data Format Specification version 1.3", RFC 1951, Aladdin Enterprises, May 1996. [SMPTE-170M] Society of Motion Picture and Television Engineers, "Television --- Composite Analog Video Signal --- NTSC for Studio Applications", SMPTE-170M, 1994. Formatted and Indexed by: page 99 of 102 instructional media + magic, inc. RFC2083 Web Address: PNG: Portable Network Graphics http://sunsite.cnlab-switch.ch/ By: Boutell, et. al. 19. Credits Editor Thomas Boutell, [email protected] Contributing Editor Tom Lane, [email protected] Authors Authors' names are presented in alphabetical order. * Mark Adler, [email protected] * Thomas Boutell, [email protected] * Christian Brunschen, [email protected] * Adam M. Costello, [email protected] * Lee Daniel Crocker, [email protected] * Andreas Dilger, [email protected] * Oliver Fromme, [email protected] * Jean-loup Gailly, [email protected] * Chris Herborth, [email protected] * Alex Jakulin, [email protected] * Neal Kettler, [email protected] * Tom Lane, [email protected] * Alexander Lehmann, [email protected] * Chris Lilley, [email protected] * Dave Martindale, [email protected] * Owen Mortensen, [email protected] * Keith S. Pickens, [email protected] * Robert P. Poole, [email protected] * Glenn Randers-Pehrson, [email protected] or [email protected] * Greg Roelofs, [email protected] * Willem van Schaik, [email protected] * Guy Schalnat * Paul Schmidt, [email protected] * Tim Wegner, [email protected] * Jeremy Wohl, [email protected] The authors wish to acknowledge the contributions of the Portable Network Graphics mailing list, the readers of comp.graphics, and the members of the World Wide Web Consortium (W3C). Formatted and Indexed by: page 100 of 102 instructional media + magic, inc. RFC2083 Web Address: PNG: Portable Network Graphics http://sunsite.cnlab-switch.ch/ By: Boutell, et. al. The Adam7 interlacing scheme is not patented and it is not the intention of the originator of that scheme to patent it. The scheme may be freely used by all PNG implementations. The name "Adam7" may be freely used to describe interlace method 1 of the PNG specification. Trademarks GIF is a service mark of CompuServe Incorporated. IBM PC is a trademark of International Business Machines Corporation. Macintosh is a trademark of Apple Computer, Inc. Microsoft and MS-DOS are trademarks of Microsoft Corporation. PhotoCD is a trademark of Eastman Kodak Company. PostScript and TIFF are trademarks of Adobe Systems Incorporated. SGI is a trademark of Silicon Graphics, Inc. X Window System is a trademark of the Massachusetts Institute of Technology. COPYRIGHT NOTICE Copyright (c) 1996 by: Massachusetts Institute of Technology (MIT) This W3C specification is being provided by the copyright holders under the following license. By obtaining, using and/or copying this specification, you agree that you have read, understood, and will comply with the following terms and conditions: Permission to use, copy, and distribute this specification for any purpose and without fee or royalty is hereby granted, provided that the full text of this NOTICE appears on ALL copies of the specification or portions thereof, including modifications, that you make. THIS SPECIFICATION IS PROVIDED "AS IS," AND COPYRIGHT HOLDERS MAKE NO REPRESENTATIONS OR WARRANTIES, EXPRESS OR IMPLIED. BY WAY OF EXAMPLE, BUT NOT LIMITATION, COPYRIGHT HOLDERS MAKE NO REPRESENTATIONS OR WARRANTIES OF MERCHANTABILITY OR FITNESS FOR ANY PARTICULAR PURPOSE OR THAT THE USE OF THE SPECIFICATION WILL NOT INFRINGE ANY THIRD PARTY PATENTS, COPYRIGHTS, TRADEMARKS OR OTHER RIGHTS. COPYRIGHT HOLDERS WILL BEAR NO LIABILITY FOR ANY USE OF THIS SPECIFICATION. Formatted and Indexed by: page 101 of 102 instructional media + magic, inc. RFC2083 Web Address: PNG: Portable Network Graphics http://sunsite.cnlab-switch.ch/ By: Boutell, et. al. The name and trademarks of copyright holders may NOT be used in advertising or publicity pertaining to the specification without specific, written prior permission. Title to copyright in this specification and any associated documentation will at all times remain with copyright holders. Security Considerations Security issues are discussed in Security considerations (Section 8.5). Author's Address Thomas Boutell PO Box 20837 Seattle, WA 98102 Phone: (206) 329-4969 EMail: [email protected] Formatted and Indexed by: page 102 of 102 instructional media + magic, inc. RFC2141 Web Address: URN Syntax http://sunsite.cnlab-switch.ch/ By: Moats Network Working Group R. Moats Request for Comments: 2141 AT&T Category: Standards Track May 1997 URN Syntax Status of This Memo This document specifies an Internet standards track protocol for the Internet community, and requests discussion and suggestions for improvements. Please refer to the current edition of the "Internet Official Protocol Standards" (STD 1) for the standardization state and status of this protocol. Distribution of this memo is unlimited. Abstract Uniform Resource Names (URNs) are intended to serve as persistent, location-independent, resource identifiers. This document sets forward the canonical syntax for URNs. A discussion of both existing legacy and new namespaces and requirements for URN presentation and transmission are presented. Finally, there is a discussion of URN equivalence and how to determine it. 1. Introduction Uniform Resource Names (URNs) are intended to serve as persistent, location-independent, resource identifiers and are designed to make it easy to map other namespaces (which share the properties of URNs) into URN-space. Therefore, the URN syntax provides a means to encode character data in a form that can be sent in existing protocols, transcribed on most keyboards, etc. 2. Syntax All URNs have the following syntax (phrases enclosed in quotes are REQUIRED): where Formatted and Indexed by: page 1 of 8 instructional media + magic, inc. RFC2141 Web Address: URN Syntax http://sunsite.cnlab-switch.ch/ By: Moats RFC 1630 [2] and RFC 1737 [3] each presents additional considerations for URN encoding, which have implications as far as limiting syntax. On the other hand, the requirement to support existing legacy naming systems has the effect of broadening syntax. Thus, we discuss the acceptable syntax for both the Namespace Identifier and the Namespace Specific String separately. 2.1 Namespace Identifier Syntax The following is the syntax for the Namespace Identifier. To (a) be consistent with all potential resolution schemes and (b) not put any undue constraints on any potential resolution scheme, the syntax for the Namespace Identifier is: This is slightly more restrictive that what is stated in [4] (which allows the characters "." and "+"). Further, the Namespace Identifier is case insensitive, so that "ISBN" and "isbn" refer to the same namespace. To avoid confusion with the "urn:" identifier, the NID "urn" is reserved and MUST NOT be used. Formatted and Indexed by: page 2 of 8 instructional media + magic, inc. RFC2141 Web Address: URN Syntax http://sunsite.cnlab-switch.ch/ By: Moats 2.2 Namespace Specific String Syntax As required by RFC 1737, there is a single canonical representation of the NSS portion of an URN. The format of this single canonical form follows: Depending on the rules governing a namespace, valid identifiers in a namespace might contain characters that are not members of the URN character set above ( 2.3 Reserved characters The remaining character set left to be discussed above is the reserved character set, which contains various characters reserved from normal use. The reserved character set follows, with a discussion on the specifics of why each character is reserved. The reserved character set is: 2.3.1 The "%" character The "%" character is reserved in the URN syntax for introducing the escape sequence for an octet. Literal use of the "%" character in a namespace must be encoded using "%25" in URNs for that namespace. The presence of an "%" character in an URN MUST be followed by two characters from the Formatted and Indexed by: page 3 of 8 instructional media + magic, inc. RFC2141 Web Address: URN Syntax http://sunsite.cnlab-switch.ch/ By: Moats Namespaces MAY designate one or more characters from the URN character set as having special meaning for that namespace. If the namespace also uses that character in a literal sense as well, the character used in a literal sense MUST be encoded with "%" followed by the hexadecimal representation of that octet. Further, a character MUST NOT be "%"-encoded if the character is not a reserved character. Therefore, the process of registering a namespace identifier shall include publication of a definition of which characters have a special meaning to that namespace. 2.3.2 The other reserved characters RFC 1630 [2] reserves the characters "/", "?", and "#" for particular purposes. The URN-WG has not yet debated the applicability and precise semantics of those purposes as applied to URNs. Therefore, these characters are RESERVED for future developments. Namespace developers SHOULD NOT use these characters in unencoded form, but rather use the appropriate %-encoding for each character. 2.4 Excluded characters The following list is included only for the sake of completeness. Any octets/characters on this list are explicitly NOT part of the URN character set, and if used in an URN, MUST be %encoded: In addition, octet 0 (0 hex) should NEVER be used, in either unencoded or %-encoded form. An URN ends when an octet/character from the excluded character set ( 3. Support of existing legacy naming systems and new naming systems Any namespace (existing or newly-devised) that is proposed as an URN-namespace and fulfills the criteria of URN-namespaces MUST be expressed in this syntax. If names in these namespaces contain characters other than those defined for the URN character set, they MUST be translated into canonical form as discussed in section 2.2. Formatted and Indexed by: page 4 of 8 instructional media + magic, inc. RFC2141 Web Address: URN Syntax http://sunsite.cnlab-switch.ch/ By: Moats 4. URN presentation and transport The URN syntax defines the canonical format for URNs and all URN transport and interchanges MUST take place in this format. Further, all URN-aware applications MUST offer the option of displaying URNs in this canonical form to allow for direct transcription (for example by cut and paste techniques). Such applications MAY support display of URNs in a more human-friendly form and may use a character set that includes characters that aren't permitted in URN syntax as defined in this RFC (that is, they may replace %-notation by characters in some extended character set in display to humans). 5. Lexical Equivalence in URNs For various purposes such as caching, it's often desirable to determine if two URNs are the same without resolving them. The general purpose means of doing so is by testing for "lexical equivalence" as defined below. Two URNs are lexically equivalent if they are octet-by-octet equal after the following preprocessing: 1. normalize the case of the leading "urn:" token 2. normalize the case of the NID 3. normalizing the case of any %-escaping Note that %-escaping MUST NOT be removed. Some namespaces may define additional lexical equivalences, such as case-insensitivity of the NSS (or parts thereof). Additional lexical equivalences MUST be documented as part of namespace registration, MUST always have the effect of eliminating some of the false negatives obtained by the procedure above, and MUST NEVER say that two URNs are not equivalent if the procedure above says they are equivalent. 6. Examples of lexical equivalence The following URN comparisons highlight the lexical equivalence definitions: 1- URN:foo:a123,456 2- urn:foo:a123,456 3- urn:FOO:a123,456 4- urn:foo:A123,456 5- urn:foo:a123%2C456 6- URN:FOO:a123%2c456 Formatted and Indexed by: page 5 of 8 instructional media + magic, inc. RFC2141 Web Address: URN Syntax http://sunsite.cnlab-switch.ch/ By: Moats URNs 1, 2, and 3 are all lexically equivalent. URN 4 is not lexically equivalent any of the other URNs of the above set. URNs 5 and 6 are only lexically equivalent to each other. 7. Functional Equivalence in URNs Functional equivalence is determined by practice within a given namespace and managed by resolvers for that namespeace. Thus, it is beyond the scope of this document. Namespace registration must include guidance on how to determine functional equivalence for that namespace, i.e. when two URNs are the identical within a namespace. 8. Security considerations This document specifies the syntax for URNs. While some namespaces resolvers may assign special meaning to certain of the characters of the Namespace Specific String, any security consideration resulting from such assignment are outside the scope of this document. It is strongly recommended that the process of registering a namespace identifier include any such considerations. 9. Acknowledgments Thanks to various members of the URN working group for comments on earlier drafts of this document. This document is partially supported by the National Science Foundation, Cooperative Agreement NCR-9218179. 10. References Request For Comments (RFC) and Internet Draft documents are available from [1] Sollins, K. R., "Requirements and a Framework for URN Resolution Systems," Work in Progress. [2] Berners-Lee, T., "Universal Resource Identifiers in WWW," RFC 1630, June 1994. [3] Sollins, K. and L. Masinter, "Functional Requirements for Uniform Resource Names," RFC 1737. December 1994. Formatted and Indexed by: page 6 of 8 instructional media + magic, inc. RFC2141 Web Address: URN Syntax http://sunsite.cnlab-switch.ch/ By: Moats [4] Berners-Lee, T., R. Fielding, L. Masinter, "Uniform Resource Locators (URL)," Work in Progress. [5] Appendix A.2 of The Unicode Consortium, "The Unicode Standard, Version 2.0", Addison-Wesley Developers Press, 1996. ISBN 0-201-48345-9. 11. Editor's address Ryan Moats AT&T 15621 Drexel Circle Omaha, NE 68135-2358 USA Phone: +1 402 894-9456 EMail: [email protected] Formatted and Indexed by: page 7 of 8 instructional media + magic, inc. RFC2141 Web Address: URN Syntax http://sunsite.cnlab-switch.ch/ By: Moats Appendix A. Handling of URNs by URL resolvers/browsers. The URN syntax has been defined so that URNs can be used in places where URLs are expected. A resolver that conforms to the current URL syntax specification [3] will extract a scheme value of "urn:" rather than a scheme value of "urn: An URN MUST be considered an opaque URL by URL resolvers and passed (with the "urn:" tag) to an URN resolver for resolution. The URN resolver can either be an external resolver that the URL resolver knows of, or it can be functionality built-in to the URL resolver. To avoid confusion of users, an URL browser SHOULD display the complete URN (including the "urn:" tag) to ensure that there is no confusion between URN namespace identifiers and URL scheme identifiers. _ Formatted and Indexed by: page 8 of 8 instructional media + magic, inc. RFC2192 Web Address: IMAP URL Scheme http://sunsite.cnlab-switch.ch/ By: Newman Network Working Group C. Newman Request for Comments: 2192 Innosoft Category: Standards Track September 1997 IMAP URL Scheme Status of this memo This document specifies an Internet standards track protocol for the Internet community, and requests discussion and suggestions for improvements. Please refer to the current edition of the "Internet Official Protocol Standards" (STD 1) for the standardization state and status of this protocol. Distribution of this memo is unlimited. Abstract IMAP [IMAP4] is a rich protocol for accessing remote message stores. It provides an ideal mechanism for accessing public mailing list archives as well as private and shared message stores. This document defines a URL scheme for referencing objects on an IMAP server. 1. Conventions used in this document The key words "MUST", "MUST NOT", "SHOULD", "SHOULD NOT", and "MAY" in this document are to be interpreted as defined in "Key words for use in RFCs to Indicate Requirement Levels" [KEYWORDS]. 2. IMAP scheme The IMAP URL scheme is used to designate IMAP servers, mailboxes, messages, MIME bodies [MIME], and search programs on Internet hosts accessible using the IMAP protocol. The IMAP URL follows the common Internet scheme syntax as defined in RFC 1738 [BASIC-URL] except that clear text passwords are not permitted. If : Formatted and Indexed by: page 1 of 16 instructional media + magic, inc. RFC2192 Web Address: IMAP URL Scheme http://sunsite.cnlab-switch.ch/ By: Newman An IMAP URL takes one of the following forms: imap:// The first form is used to refer to an IMAP server, the second form refers to a list of mailboxes, the third form refers to the contents of a mailbox or a set of messages resulting from a search, and the final form refers to a specific message or message part. Note that the syntax here is informal. The authoritative formal syntax for IMAP URLs is defined in section 11. 3. IMAP User Name and Authentication Mechanism A user name and/or authentication mechanism may be supplied. They are used in the "LOGIN" or "AUTHENTICATE" commands after making the connection to the IMAP server. If no user name or authentication mechanism is supplied, the user name "anonymous" is used with the "LOGIN" command and the password is supplied as the Internet e-mail address of the end user accessing the resource. If the URL doesn't supply a user name, the program interpreting the IMAP URL SHOULD request one from the user if necessary. An authentication mechanism can be expressed by adding ";AUTH= The string ";AUTH=*" indicates that the client SHOULD select an appropriate authentication mechanism. It MAY use any mechanism listed in the CAPABILITY command or use an out of band security service resulting in a PREAUTH connection. If no user name is specified and no appropriate authentication mechanisms are available, the client SHOULD fall back to anonymous login as described above. This allows a URL which grants read-write access to authorized users, and read-only anonymous access to other users. If a user name is included with no authentication mechanism, then ";AUTH=*" is assumed. Formatted and Indexed by: page 2 of 16 instructional media + magic, inc. RFC2192 Web Address: IMAP URL Scheme http://sunsite.cnlab-switch.ch/ By: Newman Since URLs can easily come from untrusted sources, care must be taken when resolving a URL which requires or requests any sort of authentication. If authentication credentials are supplied to the wrong server, it may compromise the security of the user's account. The program resolving the URL should make sure it meets at least one of the following criteria in this case: (1) The URL comes from a trusted source, such as a referral server which the client has validated and trusts according to site policy. Note that user entry of the URL may or may not count as a trusted source, depending on the experience level of the user and site policy. (2) Explicit local site policy permits the client to connect to the server in the URL. For example, if the client knows the site domain name, site policy may dictate that any hostname ending in that domain is trusted. (3) The user confirms that connecting to that domain name with the specified credentials and/or mechanism is permitted. (4) A mechanism is used which validates the server before passing potentially compromising client credentials. (5) An authentication mechanism is used which will not reveal information to the server which could be used to compromise future connections. URLs which do not include a user name must be treated with extra care, since they are more likely to compromise the user's primary account. A URL containing ";AUTH=*" must also be treated with extra care since it might fall back on a weaker security mechanism. Finally, clients are discouraged from using a plain text password as a fallback with ";AUTH=*" unless the connection has strong encryption (e.g. a key length of greater than 56 bits). A program interpreting IMAP URLs MAY cache open connections to an IMAP server for later re-use. If a URL contains a user name, only connections authenticated as that user may be re-used. If a URL does not contain a user name or authentication mechanism, then only an anonymous connection may be re-used. If a URL contains an authentication mechanism without a user name, then any non- anonymous connection may be re-used. Note that if unsafe or reserved characters such as " " or ";" are present in the user name or authentication mechanism, they MUST be encoded as described in RFC 1738 [BASIC-URL]. Formatted and Indexed by: page 3 of 16 instructional media + magic, inc. RFC2192 Web Address: IMAP URL Scheme http://sunsite.cnlab-switch.ch/ By: Newman 4. IMAP server An IMAP URL referring to an IMAP server has the following form: imap:// A program interpreting this URL would issue the standard set of commands it uses to present a view of the contents of an IMAP server. This is likely to be semanticly equivalent to one of the following URLs: imap:// The program interpreting this URL SHOULD use the LSUB form if it supports mailbox subscriptions. 5. Lists of mailboxes An IMAP URL referring to a list of mailboxes has the following form: imap:// The The Note that if unsafe or reserved characters such as " " or "%" are present in 6. Lists of messages An IMAP URL referring to a list of messages has the following form: imap:// Formatted and Indexed by: page 4 of 16 instructional media + magic, inc. RFC2192 Web Address: IMAP URL Scheme http://sunsite.cnlab-switch.ch/ By: Newman The The [uidvalidity] field is optional. If it is present, it MUST be the argument to the IMAP4 UIDVALIDITY status response at the time the URL was created. This SHOULD be used by the program interpreting the IMAP URL to determine if the URL is stale. The [? 7. A specific message or message part An IMAP URL referring to a specific message or message part has the following form: imap:// The If [uidvalidity] is present in this form, it SHOULD be used by the program interpreting the URL to determine if the URL is stale. The The [isection] field is optional. If not present, the URL refers to the entire Internet message as returned by the IMAP command "UID FETCH Formatted and Indexed by: page 5 of 16 instructional media + magic, inc. RFC2192 Web Address: IMAP URL Scheme http://sunsite.cnlab-switch.ch/ By: Newman 8. Relative IMAP URLs Relative IMAP URLs are permitted and are resolved according to the rules defined in RFC 1808 [REL-URL] with one exception. In IMAP URLs, parameters are treated as part of the normal path with respect to relative URL resolution. This is believed to be the behavior of the installed base and is likely to be documented in a future revision of the relative URL specification. The following observations are also important: The URLs always use "/" as the hierarchy delimiter for the purpose of resolving paths in relative URLs. IMAP4 permits the use of any hierarchy delimiter in mailbox names. For this reason, relative mailbox paths will only work if the mailbox uses "/" as the hierarchy delimiter. Relative URLs may be used on mailboxes which use other delimiters, but in that case, the entire mailbox name MUST be specified in the relative URL or inherited as a whole from the base URL. The base URL for a list of mailboxes or messages which was referred to by an IMAP URL is always the referring IMAP URL itself. The base URL for a message or message part which was referred to by an IMAP URL may be more complicated to determine. The program interpreting the relative URL will have to check the headers of the MIME entity and any enclosing MIME entities in order to locate the "Content-Base" and "Content-Location" headers. These headers are used to determine the base URL as defined in [HTTP]. For example, if the referring IMAP URL contains a "/;SECTION=1.2" parameter, then the MIME headers for section 1.2, for section 1, and for the enclosing message itself SHOULD be checked in that order for "Content-Base" or "Content-Location" headers. 9. Multinational Considerations IMAP4 [IMAP4] section 5.1.3 includes a convention for encoding non-US-ASCII characters in IMAP mailbox names. Because this convention is private to IMAP, it is necessary to convert IMAP's encoding to one that can be more easily interpreted by a URL display program. For this reason, IMAP's modified UTF-7 encoding for mailboxes MUST be converted to UTF-8 [UTF8]. Since 8-bit characters are not permitted in URLs, the UTF-8 characters are Formatted and Indexed by: page 6 of 16 instructional media + magic, inc. RFC2192 Web Address: IMAP URL Scheme http://sunsite.cnlab-switch.ch/ By: Newman encoded as required by the URL specification [BASIC-URL]. Sample code is included in Appendix A to demonstrate this conversion. 10. Examples The following examples demonstrate how an IMAP4 client program might translate various IMAP4 URLs into a series of IMAP4 commands. Commands sent from the client to the server are prefixed with "C:", and responses sent from the server to the client are prefixed with "S:". The URL: Results in the following client commands: The URL: Results in the following client commands: The URL: Results in the following client commands: Formatted and Indexed by: page 7 of 16 instructional media + magic, inc. RFC2192 Web Address: IMAP URL Scheme http://sunsite.cnlab-switch.ch/ By: Newman The URL: Results in the following client commands: If the following relative URL is located in that body part: <;section=1.4> This could result in the following client commands: C: A004 UID FETCH 20 (BODY.PEEK[1.2.MIME] BODY.PEEK[1.MIME] BODY.PEEK[HEADER.FIELDS (Content-Base Content-Location)]) The URL: Could result in the following: Formatted and Indexed by: page 8 of 16 instructional media + magic, inc. RFC2192 Web Address: IMAP URL Scheme http://sunsite.cnlab-switch.ch/ By: Newman NOTE: In this final example, the client has implementation dependent choices. The authentication mechanism could be anything, including PREAUTH. And the final FETCH command could fetch more or less information about the messages, depending on what it wishes to display to the user. 11. Security Considerations Security considerations discussed in the IMAP specification [IMAP4] and the URL specification [BASIC-URL] are relevant. Security considerations related to authenticated URLs are discussed in section 3 of this document. Many email clients store the plain text password for later use after logging into an IMAP server. Such clients MUST NOT use a stored password in response to an IMAP URL without explicit permission from the user to supply that password to the specified host name. 12. ABNF for IMAP URL scheme This uses ABNF as defined in RFC 822 [IMAIL]. Terminals from the BNF for IMAP [IMAP4] and URLs [BASIC-URL] are also used. Strings are not case sensitive and free insertion of linear-white-space is not permitted. achar = uchar / "&" / "=" / "~" ; see [BASIC-URL] for "uchar" definition bchar = achar / ":" / "@" / "/" enc_auth_type = 1*achar ; encoded version of [IMAP-AUTH] "auth_type" enc_list_mailbox = 1*bchar ; encoded version of [IMAP4] "list_mailbox" enc_mailbox = 1*bchar ; encoded version of [IMAP4] "mailbox" enc_search = 1*bchar ; encoded version of search_program below enc_section = 1*bchar ; encoded version of section below Formatted and Indexed by: page 9 of 16 instructional media + magic, inc. RFC2192 Web Address: IMAP URL Scheme http://sunsite.cnlab-switch.ch/ By: Newman enc_user = 1*achar ; encoded version of [IMAP4] "userid" imapurl = "imap://" iserver "/" [ icommand ] iauth = ";AUTH=" ( "*" / enc_auth_type ) icommand = imailboxlist / imessagelist / imessagepart imailboxlist = [enc_list_mailbox] ";TYPE=" list_type imessagelist = enc_mailbox [ "?" enc_search ] [uidvalidity] imessagepart = enc_mailbox [uidvalidity] iuid [isection] isection = "/;SECTION=" enc_section iserver = [iuserauth "@"] hostport ; See [BASIC-URL] for "hostport" definition iuid = "/;UID=" nz_number ; See [IMAP4] for "nz_number" definition iuserauth = enc_user [iauth] / [enc_user] iauth list_type = "LIST" / "LSUB" search_program = ["CHARSET" SPACE astring SPACE] search_key *(SPACE search_key) ; IMAP4 literals may not be used ; See [IMAP4] for "astring" and "search_key" section = section_text / (nz_number *["." nz_number] ["." (section_text / "MIME")]) ; See [IMAP4] for "section_text" and "nz_number" uidvalidity = ";UIDVALIDITY=" nz_number ; See [IMAP4] for "nz_number" definition 13. References [BASIC-URL] Berners-Lee, Masinter, McCahill, "Uniform Resource Locators (URL)", RFC 1738, CERN, Xerox Corporation, University of Minnesota, December 1994. Formatted and Indexed by: page 10 of 16 instructional media + magic, inc. RFC2192 Web Address: IMAP URL Scheme http://sunsite.cnlab-switch.ch/ By: Newman [IMAP4] Crispin, M., "Internet Message Access Protocol - Version 4rev1", RFC 2060, University of Washington, December 1996. [IMAP-AUTH] Myers, J., "IMAP4 Authentication Mechanism", RFC 1731, Carnegie-Mellon University, December 1994. [HTTP] Fielding, Gettys, Mogul, Frystyk, Berners-Lee, "Hypertext Transfer Protocol -- HTTP/1.1", RFC 2068, UC Irvine, DEC, MIT/LCS, January 1997. [IMAIL] Crocker, "Standard for the Format of ARPA Internet Text Messages", STD 11, RFC 822, University of Delaware, August 1982. [KEYWORDS] Bradner, "Key words for use in RFCs to Indicate Requirement Levels", RFC 2119, Harvard University, March 1997. [MIME] Freed, N., Borenstein, N., "Multipurpose Internet Mail Extensions", RFC 2045, Innosoft, First Virtual, November 1996. [REL-URL] Fielding, "Relative Uniform Resource Locators", RFC 1808, UC Irvine, June 1995. [UTF8] Yergeau, F. "UTF-8, a transformation format of Unicode and ISO 10646", RFC 2044, Alis Technologies, October 1996. 14. Author's Address Chris Newman Innosoft International, Inc. 1050 Lakes Drive West Covina, CA 91790 USA EMail: [email protected] Formatted and Indexed by: page 11 of 16 instructional media + magic, inc. RFC2192 Web Address: IMAP URL Scheme http://sunsite.cnlab-switch.ch/ By: Newman Appendix A. Sample code Here is sample C source code to convert between URL paths and IMAP mailbox names, taking into account mapping between IMAP's modified UTF-7 [IMAP4] and hex-encoded UTF-8 which is more appropriate for URLs. This code has not been rigorously tested nor does it necessarily behave reasonably with invalid input, but it should serve as a useful example. This code just converts the mailbox portion of the URL and does not deal with parameters, query or server components of the URL. #include /* hexadecimal lookup table */ static char hex[] = "0123456789ABCDEF"; /* URL unsafe printable characters */ static char urlunsafe[] = " \"#%&+:;<=>?@[\\]^`{|}"; /* UTF7 modified base64 alphabet */ static char base64chars[] = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+,"; #define UNDEFINED 64 /* UTF16 definitions */ #define UTF16MASK 0x03FFUL #define UTF16SHIFT 10 #define UTF16BASE 0x10000UL #define UTF16HIGHSTART 0xD800UL #define UTF16HIGHEND 0xDBFFUL #define UTF16LOSTART 0xDC00UL #define UTF16LOEND 0xDFFFUL /* Convert an IMAP mailbox to a URL path * dst needs to have roughly 4 times the storage space of src * Hex encoding can triple the size of the input * UTF-7 can be slightly denser than UTF-8 * (worst case: 8 octets UTF-7 becomes 9 octets UTF-8) */ void MailboxToURL(char *dst, char *src) { unsigned char c, i, bitcount; unsigned long ucs4, utf16, bitbuf; unsigned char base64[256], utf8[6]; Formatted and Indexed by: page 12 of 16 instructional media + magic, inc. RFC2192 Web Address: IMAP URL Scheme http://sunsite.cnlab-switch.ch/ By: Newman /* initialize modified base64 decoding table */ memset(base64, UNDEFINED, sizeof (base64)); for (i = 0; i < sizeof (base64chars); ++i) { base64[base64chars[i]] = i; } /* loop until end of string */ while (*src != '\0') { c = *src++; /* deal with literal characters and &- */ if (c != '&' || *src == '-') { if (c < ' ' || c > '~' || strchr(urlunsafe, c) != NULL) { /* hex encode if necessary */ dst[0] = '%'; dst[1] = hex[c >> 4]; dst[2] = hex[c & 0x0f]; dst += 3; } else { /* encode literally */ *dst++ = c; } /* skip over the '-' if this is an &- sequence */ if (c == '&') ++src; } else { /* convert modified UTF-7 -> UTF-16 -> UCS-4 -> UTF-8 -> HEX */ bitbuf = 0; bitcount = 0; ucs4 = 0; while ((c = base64[(unsigned char) *src]) != UNDEFINED) { ++src; bitbuf = (bitbuf << 6) | c; bitcount += 6; /* enough bits for a UTF-16 character? */ if (bitcount >= 16) { bitcount -= 16; utf16 = (bitcount ? bitbuf >> bitcount : bitbuf) & 0xffff; /* convert UTF16 to UCS4 */ if (utf16 >= UTF16HIGHSTART && utf16 <= UTF16HIGHEND) { ucs4 = (utf16 - UTF16HIGHSTART) << UTF16SHIFT; continue; } else if (utf16 >= UTF16LOSTART && utf16 <= UTF16LOEND) { ucs4 += utf16 - UTF16LOSTART + UTF16BASE; } else { ucs4 = utf16; } Formatted and Indexed by: page 13 of 16 instructional media + magic, inc. RFC2192 Web Address: IMAP URL Scheme http://sunsite.cnlab-switch.ch/ By: Newman /* convert UTF-16 range of UCS4 to UTF-8 */ if (ucs4 <= 0x7fUL) { utf8[0] = ucs4; i = 1; } else if (ucs4 <= 0x7ffUL) { utf8[0] = 0xc0 | (ucs4 >> 6); utf8[1] = 0x80 | (ucs4 & 0x3f); i = 2; } else if (ucs4 <= 0xffffUL) { utf8[0] = 0xe0 | (ucs4 >> 12); utf8[1] = 0x80 | ((ucs4 >> 6) & 0x3f); utf8[2] = 0x80 | (ucs4 & 0x3f); i = 3; } else { utf8[0] = 0xf0 | (ucs4 >> 18); utf8[1] = 0x80 | ((ucs4 >> 12) & 0x3f); utf8[2] = 0x80 | ((ucs4 >> 6) & 0x3f); utf8[3] = 0x80 | (ucs4 & 0x3f); i = 4; } /* convert utf8 to hex */ for (c = 0; c < i; ++c) { dst[0] = '%'; dst[1] = hex[utf8[c] >> 4]; dst[2] = hex[utf8[c] & 0x0f]; dst += 3; } } } /* skip over trailing '-' in modified UTF-7 encoding */ if (*src == '-') ++src; } } /* terminate destination string */ *dst = '\0'; } /* Convert hex coded UTF-8 URL path to modified UTF-7 IMAP mailbox * dst should be about twice the length of src to deal with non-hex * coded URLs */ void URLtoMailbox(char *dst, char *src) { unsigned int utf8pos, utf8total, i, c, utf7mode, bitstogo, utf16flag; unsigned long ucs4, bitbuf; unsigned char hextab[256]; /* initialize hex lookup table */ Formatted and Indexed by: page 14 of 16 instructional media + magic, inc. RFC2192 Web Address: IMAP URL Scheme http://sunsite.cnlab-switch.ch/ By: Newman memset(hextab, 0, sizeof (hextab)); for (i = 0; i < sizeof (hex); ++i) { hextab[hex[i]] = i; if (isupper(hex[i])) hextab[tolower(hex[i])] = i; } utf7mode = 0; utf8total = 0; bitstogo = 0; while ((c = *src) != '\0') { ++src; /* undo hex-encoding */ if (c == '%' && src[0] != '\0' && src[1] != '\0') { c = (hextab[src[0]] << 4) | hextab[src[1]]; src += 2; } /* normal character? */ if (c >= ' ' && c <= '~') { /* switch out of UTF-7 mode */ if (utf7mode) { if (bitstogo) { *dst++ = base64chars[(bitbuf << (6 - bitstogo)) & 0x3F]; } *dst++ = '-'; utf7mode = 0; } *dst++ = c; /* encode '&' as '&-' */ if (c == '&') { *dst++ = '-'; } continue; } /* switch to UTF-7 mode */ if (!utf7mode) { *dst++ = '&'; utf7mode = 1; } /* Encode US-ASCII characters as themselves */ if (c < 0x80) { ucs4 = c; utf8total = 1; } else if (utf8total) { /* save UTF8 bits into UCS4 */ ucs4 = (ucs4 << 6) | (c & 0x3FUL); if (++utf8pos < utf8total) { continue; } Formatted and Indexed by: page 15 of 16 instructional media + magic, inc. RFC2192 Web Address: IMAP URL Scheme http://sunsite.cnlab-switch.ch/ By: Newman } else { utf8pos = 1; if (c < 0xE0) { utf8total = 2; ucs4 = c & 0x1F; } else if (c < 0xF0) { utf8total = 3; ucs4 = c & 0x0F; } else { /* NOTE: can't convert UTF8 sequences longer than 4 */ utf8total = 4; ucs4 = c & 0x03; } continue; } /* loop to split ucs4 into two utf16 chars if necessary */ utf8total = 0; do { if (ucs4 >= UTF16BASE) { ucs4 -= UTF16BASE; bitbuf = (bitbuf << 16) | ((ucs4 >> UTF16SHIFT) + UTF16HIGHSTART); ucs4 = (ucs4 & UTF16MASK) + UTF16LOSTART; utf16flag = 1; } else { bitbuf = (bitbuf << 16) | ucs4; utf16flag = 0; } bitstogo += 16; /* spew out base64 */ while (bitstogo >= 6) { bitstogo -= 6; *dst++ = base64chars[(bitstogo ? (bitbuf >> bitstogo) : bitbuf) & 0x3F]; } } while (utf16flag); } /* if in UTF-7 mode, finish in ASCII */ if (utf7mode) { if (bitstogo) { *dst++ = base64chars[(bitbuf << (6 - bitstogo)) & 0x3F]; } *dst++ = '-'; } /* tie off string */ *dst = '\0'; } Formatted and Indexed by: page 16 of 16 instructional media + magic, inc. RFC2196 Web Address: Site Security Handbook http://sunsite.cnlab-switch.ch/ By: Fraser, Ed. Network Working Group B. Fraser Request for Comments: 2196 Editor FYI: 8 SEI/CMU Obsoletes: 1244 September 1997 Category: Informational Site Security Handbook Status of this Memo This memo provides information for the Internet community. It does not specify an Internet standard of any kind. Distribution of this memo is unlimited. Abstract This handbook is a guide to developing computer security policies and procedures for sites that have systems on the Internet. The purpose of this handbook is to provide practical guidance to administrators trying to secure their information and services. The subjects covered include policy content and formation, a broad range of technical system and network security topics, and security incident response. Table of Contents 1. Introduction...... 2 1.1 Purpose of this Work...... 3 1.2 Audience...... 3 1.3 Definitions...... 3 1.4 Related Work...... 4 1.5 Basic Approach...... 4 1.6 Risk Assessment...... 5 2. Security Policies...... 6 2.1 What is a Security Policy and Why Have One?...... 6 2.2 What Makes a Good Security Policy?...... 9 2.3 Keeping the Policy Flexible...... 11 3. Architecture...... 11 3.1 Objectives...... 11 3.2 Network and Service Configuration...... 14 3.3 Firewalls...... 20 4. Security Services and Procedures...... 24 4.1 Authentication...... 24 4.2 Confidentiality...... 28 4.3 Integrity...... 28 Formatted and Indexed by: page 1 of 75 instructional media + magic, inc. RFC2196 Web Address: Site Security Handbook http://sunsite.cnlab-switch.ch/ By: Fraser, Ed. 4.4 Authorization...... 29 4.5 Access...... 30 4.6 Auditing...... 34 4.7 Securing Backups...... 37 5. Security Incident Handling...... 37 5.1 Preparing and Planning for Incident Handling...... 39 5.2 Notification and Points of Contact...... 42 5.3 Identifying an Incident...... 50 5.4 Handling an Incident...... 52 5.5 Aftermath of an Incident...... 58 5.6 Responsibilities...... 59 6. Ongoing Activities...... 60 7. Tools and Locations...... 60 8. Mailing Lists and Other Resources...... 62 9. References...... 64 1. Introduction This document provides guidance to system and network administrators on how to address security issues within the Internet community. It builds on the foundation provided in RFC 1244 and is the collective work of a number of contributing authors. Those authors include: Jules P. Aronson ([email protected]), Nevil Brownlee ([email protected]), Frank Byrum ([email protected]), Joao Nuno Ferreira ([email protected]), Barbara Fraser ([email protected]), Steve Glass ([email protected]), Erik Guttman ([email protected]), Tom Killalea ([email protected]), Klaus- Peter Kossakowski ([email protected]), Lorna Leone ([email protected]), Edward.P.Lewis ([email protected]), Gary Malkin ([email protected]), Russ Mundy ([email protected]), Philip J. Nesser ([email protected]), and Michael S. Ramsey ([email protected]). In addition to the principle writers, a number of reviewers provided valuable comments. Those reviewers include: Eric Luiijf ([email protected]), Marijke Kaat ([email protected]), Ray Plzak ([email protected]) and Han Pronk ([email protected]). A special thank you goes to Joyce Reynolds, ISI, and Paul Holbrook, CICnet, for their vision, leadership, and effort in the creation of the first version of this handbook. It is the working group's sincere hope that this version will be as helpful to the community as the earlier one was. Formatted and Indexed by: page 2 of 75 instructional media + magic, inc. RFC2196 Web Address: Site Security Handbook http://sunsite.cnlab-switch.ch/ By: Fraser, Ed. 1.1 Purpose of This Work This handbook is a guide to setting computer security policies and procedures for sites that have systems on the Internet (however, the information provided should also be useful to sites not yet connected to the Internet). This guide lists issues and factors that a site must consider when setting their own policies. It makes a number of recommendations and provides discussions of relevant areas. This guide is only a framework for setting security policies and procedures. In order to have an effective set of policies and procedures, a site will have to make many decisions, gain agreement, and then communicate and implement these policies. 1.2 Audience The audience for this document are system and network administrators, and decision makers (typically "middle management") at sites. For brevity, we will use the term "administrator" throughout this document to refer to system and network administrators. This document is not directed at programmers or those trying to create secure programs or systems. The focus of this document is on the policies and procedures that need to be in place to support the technical security features that a site may be implementing. The primary audience for this work are sites that are members of the Internet community. However, this document should be useful to any site that allows communication with other sites. As a general guide to security policies, this document may also be useful to sites with isolated systems. 1.3 Definitions For the purposes of this guide, a "site" is any organization that owns computers or network-related resources. These resources may include host computers that users use, routers, terminal servers, PCs or other devices that have access to the Internet. A site may be an end user of Internet services or a service provider such as a mid- level network. However, most of the focus of this guide is on those end users of Internet services. We assume that the site has the ability to set policies and procedures for itself with the concurrence and support from those who actually own the resources. It will be assumed that sites that are parts of larger organizations will know when they need to consult, collaborate, or take recommendations from, the larger entity. Formatted and Indexed by: page 3 of 75 instructional media + magic, inc. RFC2196 Web Address: Site Security Handbook http://sunsite.cnlab-switch.ch/ By: Fraser, Ed. The "Internet" is a collection of thousands of networks linked by a common set of technical protocols which make it possible for users of any one of the networks to communicate with, or use the services located on, any of the other networks (FYI4, RFC 1594). The term "administrator" is used to cover all those people who are responsible for the day-to-day operation of system and network resources. This may be a number of individuals or an organization. The term "security administrator" is used to cover all those people who are responsible for the security of information and information technology. At some sites this function may be combined with administrator (above); at others, this will be a separate position. The term "decision maker" refers to those people at a site who set or approve policy. These are often (but not always) the people who own the resources. 1.4 Related Work The Site Security Handbook Working Group is working on a User's Guide to Internet Security. It will provide practical guidance to end users to help them protect their information and the resources they use. 1.5 Basic Approach This guide is written to provide basic guidance in developing a security plan for your site. One generally accepted approach to follow is suggested by Fites, et. al. [Fites 1989] and includes the following steps: (1) Identify what you are trying to protect. (2) Determine what you are trying to protect it from. (3) Determine how likely the threats are. (4) Implement measures which will protect your assets in a cost- effective manner. (5) Review the process continuously and make improvements each time a weakness is found. Most of this document is focused on item 4 above, but the other steps cannot be avoided if an effective plan is to be established at your site. One old truism in security is that the cost of protecting yourself against a threat should be less than the cost of recovering if the threat were to strike you. Cost in this context should be remembered to include losses expressed in real currency, reputation, trustworthiness, and other less obvious measures. Without reasonable knowledge of what you are protecting and what the likely threats are, following this rule could be difficult. Formatted and Indexed by: page 4 of 75 instructional media + magic, inc. RFC2196 Web Address: Site Security Handbook http://sunsite.cnlab-switch.ch/ By: Fraser, Ed. 1.6 Risk Assessment 1.6.1 General Discussion One of the most important reasons for creating a computer security policy is to ensure that efforts spent on security yield cost effective benefits. Although this may seem obvious, it is possible to be mislead about where the effort is needed. As an example, there is a great deal of publicity about intruders on computers systems; yet most surveys of computer security show that, for most organizations, the actual loss from "insiders" is much greater. Risk analysis involves determining what you need to protect, what you need to protect it from, and how to protect it. It is the process of examining all of your risks, then ranking those risks by level of severity. This process involves making cost-effective decisions on what you want to protect. As mentioned above, you should probably not spend more to protect something than it is actually worth. A full treatment of risk analysis is outside the scope of this document. [Fites 1989] and [Pfleeger 1989] provide introductions to this topic. However, there are two elements of a risk analysis that will be briefly covered in the next two sections: (1) Identifying the assets (2) Identifying the threats For each asset, the basic goals of security are availability, confidentiality, and integrity. Each threat should be examined with an eye to how the threat could affect these areas. 1.6.2 Identifying the Assets One step in a risk analysis is to identify all the things that need to be protected. Some things are obvious, like valuable proprietary information, intellectual property, and all the various pieces of hardware; but, some are overlooked, such as the people who actually use the systems. The essential point is to list all things that could be affected by a security problem. One list of categories is suggested by Pfleeger [Pfleeger 1989]; this list is adapted from that source: (1) Hardware: CPUs, boards, keyboards, terminals, workstations, personal computers, printers, disk drives, communication lines, terminal servers, routers. Formatted and Indexed by: page 5 of 75 instructional media + magic, inc. RFC2196 Web Address: Site Security Handbook http://sunsite.cnlab-switch.ch/ By: Fraser, Ed. (2) Software: source programs, object programs, utilities, diagnostic programs, operating systems, communication programs. (3) Data: during execution, stored on-line, archived off-line, backups, audit logs, databases, in transit over communication media. (4) People: users, administrators, hardware maintainers. (5) Documentation: on programs, hardware, systems, local administrative procedures. (6) Supplies: paper, forms, ribbons, magnetic media. 1.6.3 Identifying the Threats Once the assets requiring protection are identified, it is necessary to identify threats to those assets. The threats can then be examined to determine what potential for loss exists. It helps to consider from what threats you are trying to protect your assets. The following are classic threats that should be considered. Depending on your site, there will be more specific threats that should be identified and addressed. (1) Unauthorized access to resources and/or information (2) Unintented and/or unauthorized Disclosure of information (3) Denial of service 2. Security Policies Throughout this document there will be many references to policies. Often these references will include recommendations for specific policies. Rather than repeat guidance in how to create and communicate such a policy, the reader should apply the advice presented in this chapter when developing any policy recommended later in this book. 2.1 What is a Security Policy and Why Have One? The security-related decisions you make, or fail to make, as administrator largely determines how secure or insecure your network is, how much functionality your network offers, and how easy your network is to use. However, you cannot make good decisions about security without first determining what your security goals are. Until you determine what your security goals are, you cannot make effective use of any collection of security tools because you simply will not know what to check for and what restrictions to impose. Formatted and Indexed by: page 6 of 75 instructional media + magic, inc. RFC2196 Web Address: Site Security Handbook http://sunsite.cnlab-switch.ch/ By: Fraser, Ed. For example, your goals will probably be very different from the goals of a product vendor. Vendors are trying to make configuration and operation of their products as simple as possible, which implies that the default configurations will often be as open (i.e., insecure) as possible. While this does make it easier to install new products, it also leaves access to those systems, and other systems through them, open to any user who wanders by. Your goals will be largely determined by the following key tradeoffs: (1) services offered versus security provided - Each service offered to users carries its own security risks. For some services the risk outweighs the benefit of the service and the administrator may choose to eliminate the service rather than try to secure it. (2) ease of use versus security - The easiest system to use would allow access to any user and require no passwords; that is, there would be no security. Requiring passwords makes the system a little less convenient, but more secure. Requiring device-generated one-time passwords makes the system even more difficult to use, but much more secure. (3) cost of security versus risk of loss - There are many different costs to security: monetary (i.e., the cost of purchasing security hardware and software like firewalls and one-time password generators), performance (i.e., encryption and decryption take time), and ease of use (as mentioned above). There are also many levels of risk: loss of privacy (i.e., the reading of information by unauthorized individuals), loss of data (i.e., the corruption or erasure of information), and the loss of service (e.g., the filling of data storage space, usage of computational resources, and denial of network access). Each type of cost must be weighed against each type of loss. Your goals should be communicated to all users, operations staff, and managers through a set of security rules, called a "security policy." We are using this term, rather than the narrower "computer security policy" since the scope includes all types of information technology and the information stored and manipulated by the technology. 2.1.1 Definition of a Security Policy A security policy is a formal statement of the rules by which people who are given access to an organization's technology and information assets must abide. Formatted and Indexed by: page 7 of 75 instructional media + magic, inc. RFC2196 Web Address: Site Security Handbook http://sunsite.cnlab-switch.ch/ By: Fraser, Ed. 2.1.2 Purposes of a Security Policy The main purpose of a security policy is to inform users, staff and managers of their obligatory requirements for protecting technology and information assets. The policy should specify the mechanisms through which these requirements can be met. Another purpose is to provide a baseline from which to acquire, configure and audit computer systems and networks for compliance with the policy. Therefore an attempt to use a set of security tools in the absence of at least an implied security policy is meaningless. An Appropriate Use Policy (AUP) may also be part of a security policy. It should spell out what users shall and shall not do on the various components of the system, including the type of traffic allowed on the networks. The AUP should be as explicit as possible to avoid ambiguity or misunderstanding. For example, an AUP might list any prohibited USENET newsgroups. (Note: Appropriate Use Policy is referred to as Acceptable Use Policy by some sites.) 2.1.3 Who Should be Involved When Forming Policy? In order for a security policy to be appropriate and effective, it needs to have the acceptance and support of all levels of employees within the organization. It is especially important that corporate management fully support the security policy process otherwise there is little chance that they will have the intended impact. The following is a list of individuals who should be involved in the creation and review of security policy documents: (1) site security administrator (2) information technology technical staff (e.g., staff from computing center) (3) administrators of large user groups within the organization (e.g., business divisions, computer science department within a university, etc.) (4) security incident response team (5) representatives of the user groups affected by the security policy (6) responsible management (7) legal counsel (if appropriate) The list above is representative of many organizations, but is not necessarily comprehensive. The idea is to bring in representation from key stakeholders, management who have budget and policy authority, technical staff who know what can and cannot be supported, and legal counsel who know the legal ramifications of various policy Formatted and Indexed by: page 8 of 75 instructional media + magic, inc. RFC2196 Web Address: Site Security Handbook http://sunsite.cnlab-switch.ch/ By: Fraser, Ed. choices. In some organizations, it may be appropriate to include EDP audit personnel. Involving this group is important if resulting policy statements are to reach the broadest possible acceptance. It is also relevant to mention that the role of legal counsel will also vary from country to country. 2.2 What Makes a Good Security Policy? The characteristics of a good security policy are: (1) It must be implementable through system administration procedures, publishing of acceptable use guidelines, or other appropriate methods. (2) It must be enforcible with security tools, where appropriate, and with sanctions, where actual prevention is not technically feasible. (3) It must clearly define the areas of responsibility for the users, administrators, and management. The components of a good security policy include: (1) Computer Technology Purchasing Guidelines which specify required, or preferred, security features. These should supplement existing purchasing policies and guidelines. (2) A Privacy Policy which defines reasonable expectations of privacy regarding such issues as monitoring of electronic mail, logging of keystrokes, and access to users' files. (3) An Access Policy which defines access rights and privileges to protect assets from loss or disclosure by specifying acceptable use guidelines for users, operations staff, and management. It should provide guidelines for external connections, data communications, connecting devices to a network, and adding new software to systems. It should also specify any required notification messages (e.g., connect messages should provide warnings about authorized usage and line monitoring, and not simply say "Welcome"). (4) An Accountability Policy which defines the responsibilities of users, operations staff, and management. It should specify an audit capability, and provide incident handling guidelines (i.e., what to do and who to contact if a possible intrusion is detected). Formatted and Indexed by: page 9 of 75 instructional media + magic, inc. RFC2196 Web Address: Site Security Handbook http://sunsite.cnlab-switch.ch/ By: Fraser, Ed. (5) An Authentication Policy which establishes trust through an effective password policy, and by setting guidelines for remote location authentication and the use of authentication devices (e.g., one-time passwords and the devices that generate them). (6) An Availability statement which sets users' expectations for the availability of resources. It should address redundancy and recovery issues, as well as specify operating hours and maintenance down-time periods. It should also include contact information for reporting system and network failures. (7) An Information Technology System & Network Maintenance Policy which describes how both internal and external maintenance people are allowed to handle and access technology. One important topic to be addressed here is whether remote maintenance is allowed and how such access is controlled. Another area for consideration here is outsourcing and how it is managed. (8) A Violations Reporting Policy that indicates which types of violations (e.g., privacy and security, internal and external) must be reported and to whom the reports are made. A non- threatening atmosphere and the possibility of anonymous reporting will result in a greater probability that a violation will be reported if it is detected. (9) Supporting Information which provides users, staff, and management with contact information for each type of policy violation; guidelines on how to handle outside queries about a security incident, or information which may be considered confidential or proprietary; and cross-references to security procedures and related information, such as company policies and governmental laws and regulations. There may be regulatory requirements that affect some aspects of your security policy (e.g., line monitoring). The creators of the security policy should consider seeking legal assistance in the creation of the policy. At a minimum, the policy should be reviewed by legal counsel. Once your security policy has been established it should be clearly communicated to users, staff, and management. Having all personnel sign a statement indicating that they have read, understood, and agreed to abide by the policy is an important part of the process. Finally, your policy should be reviewed on a regular basis to see if it is successfully supporting your security needs. Formatted and Indexed by: page 10 of 75 instructional media + magic, inc. RFC2196 Web Address: Site Security Handbook http://sunsite.cnlab-switch.ch/ By: Fraser, Ed. 2.3 Keeping the Policy Flexible In order for a security policy to be viable for the long term, it requires a lot of flexibility based upon an architectural security concept. A security policy should be (largely) independent from specific hardware and software situations (as specific systems tend to be replaced or moved overnight). The mechanisms for updating the policy should be clearly spelled out. This includes the process, the people involved, and the people who must sign-off on the changes. It is also important to recognize that there are exceptions to every rule. Whenever possible, the policy should spell out what exceptions to the general policy exist. For example, under what conditions is a system administrator allowed to go through a user's files. Also, there may be some cases when multiple users will have access to the same userid. For example, on systems with a "root" user, multiple system administrators may know the password and use the root account. Another consideration is called the "Garbage Truck Syndrome." This refers to what would happen to a site if a key person was suddenly unavailable for his/her job function (e.g., was suddenly ill or left the company unexpectedly). While the greatest security resides in the minimum dissemination of information, the risk of losing critical information increases when that information is not shared. It is important to determine what the proper balance is for your site. 3. Architecture 3.1 Objectives 3.1.1 Completely Defined Security Plans All sites should define a comprehensive security plan. This plan should be at a higher level than the specific policies discussed in chapter 2, and it should be crafted as a framework of broad guidelines into which specific policies will fit. It is important to have this framework in place so that individual policies can be consistent with the overall site security architecture. For example, having a strong policy with regard to Internet access and having weak restrictions on modem usage is inconsistent with an overall philosophy of strong security restrictions on external access. A security plan should define: the list of network services that will be provided; which areas of the organization will provide the services; who will have access to those services; how access will be provided; who will administer those services; etc. Formatted and Indexed by: page 11 of 75 instructional media + magic, inc. RFC2196 Web Address: Site Security Handbook http://sunsite.cnlab-switch.ch/ By: Fraser, Ed. The plan should also address how incident will be handled. Chapter 5 provides an in-depth discussion of this topic, but it is important for each site to define classes of incidents and corresponding responses. For example, sites with firewalls should set a threshold on the number of attempts made to foil the firewall before triggering a response? Escallation levels should be defined for both attacks and responses. Sites without firewalls will have to determine if a single attempt to connect to a host constitutes an incident? What about a systematic scan of systems? For sites connected to the Internet, the rampant media magnification of Internet related security incidents can overshadow a (potentially) more serious internal security problem. Likewise, companies who have never been connected to the Internet may have strong, well defined, internal policies but fail to adequately address an external connection policy. 3.1.2 Separation of Services There are many services which a site may wish to provide for its users, some of which may be external. There are a variety of security reasons to attempt to isolate services onto dedicated host computers. There are also performance reasons in most cases, but a detailed discussion is beyond to scope of this document. The services which a site may provide will, in most cases, have different levels of access needs and models of trust. Services which are essential to the security or smooth operation of a site would be better off being placed on a dedicated machine with very limited access (see Section 3.1.3 "deny all" model), rather than on a machine that provides a service (or services) which has traditionally been less secure, or requires greater accessability by users who may accidentally suborn security. It is also important to distinguish between hosts which operate within different models of trust (e.g., all the hosts inside of a firewall and any host on an exposed network). Some of the services which should be examined for potential separation are outlined in section 3.2.3. It is important to remember that security is only as strong as the weakest link in the chain. Several of the most publicized penetrations in recent years have been through the exploitation of vulnerabilities in electronic mail systems. The intruders were not trying to steal electronic mail, but they used the vulnerability in that service to gain access to other systems. Formatted and Indexed by: page 12 of 75 instructional media + magic, inc. RFC2196 Web Address: Site Security Handbook http://sunsite.cnlab-switch.ch/ By: Fraser, Ed. If possible, each service should be running on a different machine whose only duty is to provide a specific service. This helps to isolate intruders and limit potential harm. 3.1.3 Deny all/ Allow all There are two diametrically opposed underlying philosophies which can be adopted when defining a security plan. Both alternatives are legitimate models to adopt, and the choice between them will depend on the site and its needs for security. The first option is to turn off all services and then selectively enable services on a case by case basis as they are needed. This can be done at the host or network level as appropriate. This model, which will here after be referred to as the "deny all" model, is generally more secure than the other model described in the next paragraph. More work is required to successfully implement a "deny all" configuration as well as a better understanding of services. Allowing only known services provides for a better analysis of a particular service/protocol and the design of a security mechanism suited to the security level of the site. The other model, which will here after be referred to as the "allow all" model, is much easier to implement, but is generally less secure than the "deny all" model. Simply turn on all services, usually the default at the host level, and allow all protocols to travel across network boundaries, usually the default at the router level. As security holes become apparent, they are restricted or patched at either the host or network level. Each of these models can be applied to different portions of the site, depending on functionality requirements, administrative control, site policy, etc. For example, the policy may be to use the "allow all" model when setting up workstations for general use, but adopt a "deny all" model when setting up information servers, like an email hub. Likewise, an "allow all" policy may be adopted for traffic between LAN's internal to the site, but a "deny all" policy can be adopted between the site and the Internet. Be careful when mixing philosophies as in the examples above. Many sites adopt the theory of a hard "crunchy" shell and a soft "squishy" middle. They are willing to pay the cost of security for their external traffic and require strong security measures, but are unwilling or unable to provide similar protections internally. This works fine as long as the outer defenses are never breached and the internal users can be trusted. Once the outer shell (firewall) is breached, subverting the internal network is trivial. Formatted and Indexed by: page 13 of 75 instructional media + magic, inc. RFC2196 Web Address: Site Security Handbook http://sunsite.cnlab-switch.ch/ By: Fraser, Ed. 3.1.4 Identify Real Needs for Services There is a large variety of services which may be provided, both internally and on the Internet at large. Managing security is, in many ways, managing access to services internal to the site and managing how internal users access information at remote sites. Services tend to rush like waves over the Internet. Over the years many sites have established anonymous FTP servers, gopher servers, wais servers, WWW servers, etc. as they became popular, but not particularly needed, at all sites. Evaluate all new services that are established with a skeptical attitude to determine if they are actually needed or just the current fad sweeping the Internet. Bear in mind that security complexity can grow exponentially with the number of services provided. Filtering routers need to be modified to support the new protocols. Some protocols are inherently difficult to filter safely (e.g., RPC and UDP services), thus providing more openings to the internal network. Services provided on the same machine can interact in catastrophic ways. For example, allowing anonymous FTP on the same machine as the WWW server may allow an intruder to place a file in the anonymous FTP area and cause the HTTP server to execute it. 3.2 Network and Service Configuration 3.2.1 Protecting the Infrastructure Many network administrators go to great lengths to protect the hosts on their networks. Few administrators make any effort to protect the networks themselves. There is some rationale to this. For example, it is far easier to protect a host than a network. Also, intruders are likely to be after data on the hosts; damaging the network would not serve their purposes. That said, there are still reasons to protect the networks. For example, an intruder might divert network traffic through an outside host in order to examine the data (i.e., to search for passwords). Also, infrastructure includes more than the networks and the routers which interconnect them. Infrastructure also includes network management (e.g., SNMP), services (e.g., DNS, NFS, NTP, WWW), and security (i.e., user authentication and access restrictions). The infrastructure also needs protection against human error. When an administrator misconfigures a host, that host may offer degraded service. This only affects users who require that host and, unless Formatted and Indexed by: page 14 of 75 instructional media + magic, inc. RFC2196 Web Address: Site Security Handbook http://sunsite.cnlab-switch.ch/ By: Fraser, Ed. that host is a primary server, the number of affected users will therefore be limited. However, if a router is misconfigured, all users who require the network will be affected. Obviously, this is a far larger number of users than those depending on any one host. 3.2.2 Protecting the Network There are several problems to which networks are vulnerable. The classic problem is a "denial of service" attack. In this case, the network is brought to a state in which it can no longer carry legitimate users' data. There are two common ways this can be done: by attacking the routers and by flooding the network with extraneous traffic. Please note that the term "router" in this section is used as an example of a larger class of active network interconnection components that also includes components like firewalls, proxy- servers, etc. An attack on the router is designed to cause it to stop forwarding packets, or to forward them improperly. The former case may be due to a misconfiguration, the injection of a spurious routing update, or a "flood attack" (i.e., the router is bombarded with unroutable packets, causing its performance to degrade). A flood attack on a network is similar to a flood attack on a router, except that the flood packets are usually broadcast. An ideal flood attack would be the injection of a single packet which exploits some known flaw in the network nodes and causes them to retransmit the packet, or generate error packets, each of which is picked up and repeated by another host. A well chosen attack packet can even generate an exponential explosion of transmissions. Another classic problem is "spoofing." In this case, spurious routing updates are sent to one or more routers causing them to misroute packets. This differs from a denial of service attack only in the purpose behind the spurious route. In denial of service, the object is to make the router unusable; a state which will be quickly detected by network users. In spoofing, the spurious route will cause packets to be routed to a host from which an intruder may monitor the data in the packets. These packets are then re-routed to their correct destinations. However, the intruder may or may not have altered the contents of the packets. The solution to most of these problems is to protect the routing update packets sent by the routing protocols in use (e.g., RIP-2, OSPF). There are three levels of protection: clear-text password, cryptographic checksum, and encryption. Passwords offer only minimal protection against intruders who do not have direct access to the physical networks. Passwords also offer some protection against misconfigured routers (i.e, routers which, out of the box, attempt to Formatted and Indexed by: page 15 of 75 instructional media + magic, inc. RFC2196 Web Address: Site Security Handbook http://sunsite.cnlab-switch.ch/ By: Fraser, Ed. route packets). The advantage of passwords is that they have a very low overhead, in both bandwidth and CPU consumption. Checksums protect against the injection of spurious packets, even if the intruder has direct access to the physical network. Combined with a sequence number, or other unique identifier, a checksum can also protect again "replay" attacks, wherein an old (but valid at the time) routing update is retransmitted by either an intruder or a misbehaving router. The most security is provided by complete encryption of sequenced, or uniquely identified, routing updates. This prevents an intruder from determining the topology of the network. The disadvantage to encryption is the overhead involved in processing the updates. RIP-2 (RFC 1723) and OSPF (RFC 1583) both support clear-text passwords in their base design specifications. In addition, there are extensions to each base protocol to support MD5 encryption. Unfortunately, there is no adequate protection against a flooding attack, or a misbehaving host or router which is flooding the network. Fortunately, this type of attack is obvious when it occurs and can usually be terminated relatively simply. 3.2.3 Protecting the Services There are many types of services and each has its own security requirements. These requirements will vary based on the intended use of the service. For example, a service which should only be usable within a site (e.g., NFS) may require different protection mechanisms than a service provided for external use. It may be sufficient to protect the internal server from external access. However, a WWW server, which provides a home page intended for viewing by users anywhere on the Internet, requires built-in protection. That is, the service/protocol/server must provide whatever security may be required to prevent unauthorized access and modification of the Web database. Internal services (i.e., services meant to be used only by users within a site) and external services (i.e., services deliberately made available to users outside a site) will, in general, have protection requirements which differ as previously described. It is therefore wise to isolate the internal services to one set of server host computers and the external services to another set of server host computers. That is, internal and external servers should not be co-located on the same host computer. In fact, many sites go so far Formatted and Indexed by: page 16 of 75 instructional media + magic, inc. RFC2196 Web Address: Site Security Handbook http://sunsite.cnlab-switch.ch/ By: Fraser, Ed. as to have one set of subnets (or even different networks) which are accessible from the outside and another set which may be accessed only within the site. Of course, there is usually a firewall which connects these partitions. Great care must be taken to ensure that such a firewall is operating properly. There is increasing interest in using intranets to connect different parts of a organization (e.g., divisions of a company). While this document generally differentiates between external and internal (public and private), sites using intranets should be aware that they will need to consider three separations and take appropriate actions when designing and offering services. A service offered to an intranet would be neither public, nor as completely private as a service to a single organizational subunit. Therefore, the service would need its own supporting system, separated from both external and internal services and networks. One form of external service deserves some special consideration, and that is anonymous, or guest, access. This may be either anonymous FTP or guest (unauthenticated) login. It is extremely important to ensure that anonymous FTP servers and guest login userids are carefully isolated from any hosts and file systems from which outside users should be kept. Another area to which special attention must be paid concerns anonymous, writable access. A site may be legally responsible for the content of publicly available information, so careful monitoring of the information deposited by anonymous users is advised. Now we shall consider some of the most popular services: name service, password/key service, authentication/proxy service, electronic mail, WWW, file transfer, and NFS. Since these are the most frequently used services, they are the most obvious points of attack. Also, a successful attack on one of these services can produce disaster all out of proportion to the innocence of the basic service. 3.2.3.1 Name Servers (DNS and NIS(+)) The Internet uses the Domain Name System (DNS) to perform address resolution for host and network names. The Network Information Service (NIS) and NIS+ are not used on the global Internet, but are subject to the same risks as a DNS server. Name-to-address resolution is critical to the secure operation of any network. An attacker who can successfully control or impersonate a DNS server can re-route traffic to subvert security protections. For example, routine traffic can be diverted to a compromised system to be monitored; or, users can be tricked into providing authentication secrets. An organization should create well known, protected sites Formatted and Indexed by: page 17 of 75 instructional media + magic, inc. RFC2196 Web Address: Site Security Handbook http://sunsite.cnlab-switch.ch/ By: Fraser, Ed. to act as secondary name servers and protect their DNS masters from denial of service attacks using filtering routers. Traditionally, DNS has had no security capabilities. In particular, the information returned from a query could not be checked for modification or verified that it had come from the name server in question. Work has been done to incorporate digital signatures into the protocol which, when deployed, will allow the integrity of the information to be cryptographically verified (see RFC 2065). 3.2.3.2 Password/Key Servers (NIS(+) and KDC) Password and key servers generally protect their vital information (i.e., the passwords and keys) with encryption algorithms. However, even a one-way encrypted password can be determined by a dictionary attack (wherein common words are encrypted to see if they match the stored encryption). It is therefore necessary to ensure that these servers are not accessable by hosts which do not plan to use them for the service, and even those hosts should only be able to access the service (i.e., general services, such as Telnet and FTP, should not be allowed by anyone other than administrators). 3.2.3.3 Authentication/Proxy Servers (SOCKS, FWTK) A proxy server provides a number of security enhancements. It allows sites to concentrate services through a specific host to allow monitoring, hiding of internal structure, etc. This funnelling of services creates an attractive target for a potential intruder. The type of protection required for a proxy server depends greatly on the proxy protocol in use and the services being proxied. The general rule of limiting access only to those hosts which need the services, and limiting access by those hosts to only those services, is a good starting point. 3.2.3.4 Electronic Mail Electronic mail (email) systems have long been a source for intruder break-ins because email protocols are among the oldest and most widely deployed services. Also, by it's very nature, an email server requires access to the outside world; most email servers accept input from any source. An email server generally consists of two parts: a receiving/sending agent and a processing agent. Since email is delivered to all users, and is usually private, the processing agent typically requires system (root) privileges to deliver the mail. Most email implementations perform both portions of the service, which means the receiving agent also has system privileges. This opens several security holes which this document will not describe. There are some implementations available which allow a separation of Formatted and Indexed by: page 18 of 75 instructional media + magic, inc. RFC2196 Web Address: Site Security Handbook http://sunsite.cnlab-switch.ch/ By: Fraser, Ed. the two agents. Such implementations are generally considered more secure, but still require careful installation to avoid creating a security problem. 3.2.3.5 World Wide Web (WWW) The Web is growing in popularity exponentially because of its ease of use and the powerful ability to concentrate information services. Most WWW servers accept some type of direction and action from the persons accessing their services. The most common example is taking a request from a remote user and passing the provided information to a program running on the server to process the request. Some of these programs are not written with security in mind and can create security holes. If a Web server is available to the Internet community, it is especially important that confidential information not be co-located on the same host as that server. In fact, it is recommended that the server have a dedicated host which is not "trusted" by other internal hosts. Many sites may want to co-locate FTP service with their WWW service. But this should only occur for anon-ftp servers that only provide information (ftp-get). Anon-ftp puts, in combination with WWW, might be dangerous (e.g., they could result in modifications to the information your site is publishing to the web) and in themselves make the security considerations for each service different. 3.2.3.6 File Transfer (FTP, TFTP) FTP and TFTP both allow users to receive and send electronic files in a point-to-point manner. However, FTP requires authentication while TFTP requires none. For this reason, TFTP should be avoided as much as possible. Improperly configured FTP servers can allow intruders to copy, replace and delete files at will, anywhere on a host, so it is very important to configure this service correctly. Access to encrypted passwords and proprietary data, and the introduction of Trojan horses are just a few of the potential security holes that can occur when the service is configured incorrectly. FTP servers should reside on their own host. Some sites choose to co-locate FTP with a Web server, since the two protocols share common security considerations However, the the practice isn't recommended, especially when the FTP service allows the deposit of files (see section on WWW above). As mentioned in the opening paragraphs of section 3.2.3, services offered internally to your site should not be co-located with services offered externally. Each should have its own host. Formatted and Indexed by: page 19 of 75 instructional media + magic, inc. RFC2196 Web Address: Site Security Handbook http://sunsite.cnlab-switch.ch/ By: Fraser, Ed. TFTP does not support the same range of functions as FTP, and has no security whatsoever. This service should only be considered for internal use, and then it should be configured in a restricted way so that the server only has access to a set of predetermined files (instead of every world-readable file on the system). Probably the most common usage of TFTP is for downloading router configuration files to a router. TFTP should reside on its own host, and should not be installed on hosts supporting external FTP or Web access. 3.2.3.7 NFS The Network File Service allows hosts to share common disks. NFS is frequently used by diskless hosts who depend on a disk server for all of their storage needs. Unfortunately, NFS has no built-in security. It is therefore necessary that the NFS server be accessable only by those hosts which are using it for service. This is achieved by specifying which hosts the file system is being exported to and in what manner (e.g., read-only, read-write, etc.). Filesystems should not be exported to any hosts outside the local network since this will require that the NFS service be accessible externally. Ideally, external access to NFS service should be stopped by a firewall. 3.2.4 Protecting the Protection It is amazing how often a site will overlook the most obvious weakness in its security by leaving the security server itself open to attack. Based on considerations previously discussed, it should be clear that: the security server should not be accessible from off-site; should offer minimum access, except for the authentication function, to users on-site; and should not be co-located with any other servers. Further, all access to the node, including access to the service itself, should be logged to provide a "paper trail" in the event of a security breach. 3.3 Firewalls One of the most widely deployed and publicized security measures in use on the Internet is a "firewall." Firewalls have been given the reputation of a general panacea for many, if not all, of the Internet security issues. They are not. Firewalls are just another tool in the quest for system security. They provide a certain level of protection and are, in general, a way of implementing security policy at the network level. The level of security that a firewall provides can vary as much as the level of security on a particular machine. There are the traditional trade-offs between security, ease of use, cost, complexity, etc. Formatted and Indexed by: page 20 of 75 instructional media + magic, inc. RFC2196 Web Address: Site Security Handbook http://sunsite.cnlab-switch.ch/ By: Fraser, Ed. A firewall is any one of several mechanisms used to control and watch access to and from a network for the purpose of protecting it. A firewall acts as a gateway through which all traffic to and from the protected network and/or systems passes. Firewalls help to place limitations on the amount and type of communication that takes place between the protected network and the another network (e.g., the Internet, or another piece of the site's network). A firewall is generally a way to build a wall between one part of a network, a company's internal network, for example, and another part, the global Internet, for example. The unique feature about this wall is that there needs to be ways for some traffic with particular characteristics to pass through carefully monitored doors ("gateways"). The difficult part is establishing the criteria by which the packets are allowed or denied access through the doors. Books written on firewalls use different terminology to describe the various forms of firewalls. This can be confusing to system administrators who are not familiar with firewalls. The thing to note here is that there is no fixed terminology for the description of firewalls. Firewalls are not always, or even typically, a single machine. Rather, firewalls are often a combination of routers, network segments, and host computers. Therefore, for the purposes of this discussion, the term "firewall" can consist of more than one physical device. Firewalls are typically built using two different components, filtering routers and proxy servers. Filtering routers are the easiest component to conceptualize in a firewall. A router moves data back and forth between two (or more) different networks. A "normal" router takes a packet from network A and "routes" it to its destination on network B. A filtering router does the same thing but decides not only how to route the packet, but whether it should route the packet. This is done by installing a series of filters by which the router decides what to do with any given packet of data. A discussion concerning capabilities of a particular brand of router, running a particular software version is outside the scope of this document. However, when evaluating a router to be used for filtering packets, the following criteria can be important when implementing a filtering policy: source and destination IP address, source and destination TCP port numbers, state of the TCP "ack" bit, UDP source and destination port numbers, and direction of packet flow (i.e.. A- >B or B->A). Other information necessary to construct a secure filtering scheme are whether the router reorders filter instructions (designed to optimize filters, this can sometimes change the meaning and cause unintended access), and whether it is possible to apply Formatted and Indexed by: page 21 of 75 instructional media + magic, inc. RFC2196 Web Address: Site Security Handbook http://sunsite.cnlab-switch.ch/ By: Fraser, Ed. filters for inbound and outbound packets on each interface (if the router filters only outbound packets then the router is "outside" of its filters and may be more vulnerable to attack). In addition to the router being vulnerable, this distinction between applying filters on inbound or outbound packets is especially relevant for routers with more than 2 interfaces. Other important issues are the ability to create filters based on IP header options and the fragment state of a packet. Building a good filter can be very difficult and requires a good understanding of the type of services (protocols) that will be filtered. For better security, the filters usually restrict access between the two connected nets to just one host, the bastion host. It is only possible to access the other network via this bastion host. As only this host, rather than a few hundred hosts, can get attacked, it is easier to maintain a certain level of security because only this host has to be protected very carefully. To make resources available to legitimate users across this firewall, services have to be forwarded by the bastion host. Some servers have forwarding built in (like DNS-servers or SMTP-servers), for other services (e.g., Telnet, FTP, etc.), proxy servers can be used to allow access to the resources across the firewall in a secure way. A proxy server is way to concentrate application services through a single machine. There is typically a single machine (the bastion host) that acts as a proxy server for a variety of protocols (Telnet, SMTP, FTP, HTTP, etc.) but there can be individual host computers for each service. Instead of connecting directly to an external server, the client connects to the proxy server which in turn initiates a connection to the requested external server. Depending on the type of proxy server used, it is possible to configure internal clients to perform this redirection automatically, without knowledge to the user, others might require that the user connect directly to the proxy server and then initiate the connection through a specified format. There are significant security benefits which can be derived from using proxy servers. It is possible to add access control lists to protocols, requiring users or systems to provide some level of authentication before access is granted. Smarter proxy servers, sometimes called Application Layer Gateways (ALGs), can be written which understand specific protocols and can be configured to block only subsections of the protocol. For example, an ALG for FTP can tell the difference between the "put" command and the "get" command; an organization may wish to allow users to "get" files from the Internet, but not be able to "put" internal files on a remote server. By contrast, a filtering router could either block all FTP access, or none, but not a subset. Formatted and Indexed by: page 22 of 75 instructional media + magic, inc. RFC2196 Web Address: Site Security Handbook http://sunsite.cnlab-switch.ch/ By: Fraser, Ed. Proxy servers can also be configured to encrypt data streams based on a variety of parameters. An organization might use this feature to allow encrypted connections between two locations whose sole access points are on the Internet. Firewalls are typically thought of as a way to keep intruders out, but they are also often used as a way to let legitimate users into a site. There are many examples where a valid user might need to regularly access the "home" site while on travel to trade shows and conferences, etc. Access to the Internet is often available but may be through an untrusted machine or network. A correctly configured proxy server can allow the correct users into the site while still denying access to other users. The current best effort in firewall techniques is found using a combination of a pair of screening routers with one or more proxy servers on a network between the two routers. This setup allows the external router to block off any attempts to use the underlying IP layer to break security (IP spoofing, source routing, packet fragments), while allowing the proxy server to handle potential security holes in the higher layer protocols. The internal router's purpose is to block all traffic except to the proxy server. If this setup is rigidly implemented, a high level of security can be achieved. Most firewalls provide logging which can be tuned to make security administration of the network more convenient. Logging may be centralized and the system may be configured to send out alerts for abnormal conditions. It is important to regularly monitor these logs for any signs of intrusions or break-in attempts. Since some intruders will attempt to cover their tracks by editing logs, it is desirable to protect these logs. A variety of methods is available, including: write once, read many (WORM) drives; papers logs; and centralized logging via the "syslog" utility. Another technique is to use a "fake" serial printer, but have the serial port connected to an isolated machine or PC which keeps the logs. Firewalls are available in a wide range of quality and strengths. Commercial packages start at approximately $10,000US and go up to over $250,000US. "Home grown" firewalls can be built for smaller amounts of capital. It should be remembered that the correct setup of a firewall (commercial or homegrown) requires a significant amount of skill and knowledge of TCP/IP. Both types require regular maintenance, installation of software patches and updates, and regular monitoring. When budgeting for a firewall, these additional costs should be considered in addition to the cost of the physical elements of the firewall. Formatted and Indexed by: page 23 of 75 instructional media + magic, inc. RFC2196 Web Address: Site Security Handbook http://sunsite.cnlab-switch.ch/ By: Fraser, Ed. As an aside, building a "home grown" firewall requires a significant amount of skill and knowledge of TCP/IP. It should not be trivially attempted because a perceived sense of security is worse in the long run than knowing that there is no security. As with all security measures, it is important to decide on the threat, the value of the assets to be protected, and the costs to implement security. A final note about firewalls. They can be a great aid when implementing security for a site and they protect against a large variety of attacks. But it is important to keep in mind that they are only one part of the solution. They cannot protect your site against all types of attack. 4. Security Services and Procedures This chapter guides the reader through a number of topics that should be addressed when securing a site. Each section touches on a security service or capability that may be required to protect the information and systems at a site. The topics are presented at a fairly high-level to introduce the reader to the concepts. Throughout the chapter, you will find significant mention of cryptography. It is outside the scope of this document to delve into details concerning cryptography, but the interested reader can obtain more information from books and articles listed in the reference section of this document. 4.1 Authentication For many years, the prescribed method for authenticating users has been through the use of standard, reusable passwords. Originally, these passwords were used by users at terminals to authenticate themselves to a central computer. At the time, there were no networks (internally or externally), so the risk of disclosure of the clear text password was minimal. Today, systems are connected together through local networks, and these local networks are further connected together and to the Internet. Users are logging in from all over the globe; their reusable passwords are often transmitted across those same networks in clear text, ripe for anyone in-between to capture. And indeed, the CERT* Coordination Center and other response teams are seeing a tremendous number of incidents involving packet sniffers which are capturing the clear text passwords. With the advent of newer technologies like one-time passwords (e.g., S/Key), PGP, and token-based authentication devices, people are using password-like strings as secret tokens and pins. If these secret tokens and pins are not properly selected and protected, the authentication will be easily subverted. Formatted and Indexed by: page 24 of 75 instructional media + magic, inc. RFC2196 Web Address: Site Security Handbook http://sunsite.cnlab-switch.ch/ By: Fraser, Ed. 4.1.1 One-Time passwords As mentioned above, given today's networked environments, it is recommended that sites concerned about the security and integrity of their systems and networks consider moving away from standard, reusable passwords. There have been many incidents involving Trojan network programs (e.g., telnet and rlogin) and network packet sniffing programs. These programs capture clear text hostname/account name/password triplets. Intruders can use the captured information for subsequent access to those hosts and accounts. This is possible because 1) the password is used over and over (hence the term "reusable"), and 2) the password passes across the network in clear text. Several authentication techniques have been developed that address this problem. Among these techniques are challenge-response technologies that provide passwords that are only used once (commonly called one-time passwords). There are a number of products available that sites should consider using. The decision to use a product is the responsibility of each organization, and each organization should perform its own evaluation and selection. 4.1.2 Kerberos Kerberos is a distributed network security system which provides for authentication across unsecured networks. If requested by the application, integrity and encryption can also be provided. Kerberos was originally developed at the Massachusetts Institute of Technology (MIT) in the mid 1980s. There are two major releases of Kerberos, version 4 and 5, which are for practical purposes, incompatible. Kerberos relies on a symmetric key database using a key distribution center (KDC) which is known as the Kerberos server. A user or service (known as "principals") are granted electronic "tickets" after properly communicating with the KDC. These tickets are used for authentication between principals. All tickets include a time stamp which limits the time period for which the ticket is valid. Therefore, Kerberos clients and server must have a secure time source, and be able to keep time accurately. The practical side of Kerberos is its integration with the application level. Typical applications like FTP, telnet, POP, and NFS have been integrated with the Kerberos system. There are a variety of implementations which have varying levels of integration. Please see the Kerberos FAQ available at http://www.ov.com/misc/krb- faq.html for the latest information. Formatted and Indexed by: page 25 of 75 instructional media + magic, inc. RFC2196 Web Address: Site Security Handbook http://sunsite.cnlab-switch.ch/ By: Fraser, Ed. 4.1.3 Choosing and Protecting Secret Tokens and PINs When selecting secret tokens, take care to choose them carefully. Like the selection of passwords, they should be robust against brute force efforts to guess them. That is, they should not be single words in any language, any common, industry, or cultural acronyms, etc. Ideally, they will be longer rather than shorter and consist of pass phrases that combine upper and lower case character, digits, and other characters. Once chosen, the protection of these secret tokens is very important. Some are used as pins to hardware devices (like token cards) and these should not be written down or placed in the same location as the device with which they are associated. Others, such as a secret Pretty Good Privacy (PGP) key, should be protected from unauthorized access. One final word on this subject. When using cryptography products, like PGP, take care to determine the proper key length and ensure that your users are trained to do likewise. As technology advances, the minimum safe key length continues to grow. Make sure your site keeps up with the latest knowledge on the technology so that you can ensure that any cryptography in use is providing the protection you believe it is. 4.1.4 Password Assurance While the need to eliminate the use of standard, reusable passwords cannot be overstated, it is recognized that some organizations may still be using them. While it's recommended that these organizations transition to the use of better technology, in the mean time, we have the following advice to help with the selection and maintenance of traditional passwords. But remember, none of these measures provides protection against disclosure due to sniffer programs. (1) The importance of robust passwords - In many (if not most) cases of system penetration, the intruder needs to gain access to an account on the system. One way that goal is typically accomplished is through guessing the password of a legitimate user. This is often accomplished by running an automated password cracking program, which utilizes a very large dictionary, against the system's password file. The only way to guard against passwords being disclosed in this manner is through the careful selection of passwords which cannot be easily guessed (i.e., combinations of numbers, letters, and punctuation characters). Passwords should also be as long as the system supports and users can tolerate. Formatted and Indexed by: page 26 of 75 instructional media + magic, inc. RFC2196 Web Address: Site Security Handbook http://sunsite.cnlab-switch.ch/ By: Fraser, Ed. (2) Changing default passwords - Many operating systems and application programs are installed with default accounts and passwords. These must be changed immediately to something that cannot be guessed or cracked. (3) Restricting access to the password file - In particular, a site wants to protect the encrypted password portion of the file so that would-be intruders don't have them available for cracking. One effective technique is to use shadow passwords where the password field of the standard file contains a dummy or false password. The file containing the legitimate passwords are protected elsewhere on the system. (4) Password aging - When and how to expire passwords is still a subject of controversy among the security community. It is generally accepted that a password should not be maintained once an account is no longer in use, but it is hotly debated whether a user should be forced to change a good password that's in active use. The arguments for changing passwords relate to the prevention of the continued use of penetrated accounts. However, the opposition claims that frequent password changes lead to users writing down their passwords in visible areas (such as pasting them to a terminal), or to users selecting very simple passwords that are easy to guess. It should also be stated that an intruder will probably use a captured or guessed password sooner rather than later, in which case password aging provides little if any protection. While there is no definitive answer to this dilemma, a password policy should directly address the issue and provide guidelines for how often a user should change the password. Certainly, an annual change in their password is usually not difficult for most users, and you should consider requiring it. It is recommended that passwords be changed at least whenever a privileged account is compromised, there is a critical change in personnel (especially if it is an administrator!), or when an account has been compromised. In addition, if a privileged account password is compromised, all passwords on the system should be changed. (5) Password/account blocking - Some sites find it useful to disable accounts after a predefined number of failed attempts to authenticate. If your site decides to employ this mechanism, it is recommended that the mechanism not "advertise" itself. After Formatted and Indexed by: page 27 of 75 instructional media + magic, inc. RFC2196 Web Address: Site Security Handbook http://sunsite.cnlab-switch.ch/ By: Fraser, Ed. disabling, even if the correct password is presented, the message displayed should remain that of a failed login attempt. Implementing this mechanism will require that legitimate users contact their system administrator to request that their account be reactivated. (6) A word about the finger daemon - By default, the finger daemon displays considerable system and user information. For example, it can display a list of all users currently using a system, or all the contents of a specific user's .plan file. This information can be used by would-be intruders to identify usernames and guess their passwords. It is recommended that sites consider modifying finger to restrict the information displayed. 4.2 Confidentiality There will be information assets that your site will want to protect from disclosure to unauthorized entities. Operating systems often have built-in file protection mechanisms that allow an administrator to control who on the system can access, or "see," the contents of a given file. A stronger way to provide confidentiality is through encryption. Encryption is accomplished by scrambling data so that it is very difficult and time consuming for anyone other than the authorized recipients or owners to obtain the plain text. Authorized recipients and the owner of the information will possess the corresponding decryption keys that allow them to easily unscramble the text to a readable (clear text) form. We recommend that sites use encryption to provide confidentiality and protect valuable information. The use of encryption is sometimes controlled by governmental and site regulations, so we encourage administrators to become informed of laws or policies that regulate its use before employing it. It is outside the scope of this document to discuss the various algorithms and programs available for this purpose, but we do caution against the casual use of the UNIX crypt program as it has been found to be easily broken. We also encourage everyone to take time to understand the strength of the encryption in any given algorithm/product before using it. Most well-known products are well-documented in the literature, so this should be a fairly easy task. 4.3 Integrity As an administrator, you will want to make sure that information (e.g., operating system files, company data, etc.) has not been altered in an unauthorized fashion. This means you will want to provide some assurance as to the integrity of the information on your Formatted and Indexed by: page 28 of 75 instructional media + magic, inc. RFC2196 Web Address: Site Security Handbook http://sunsite.cnlab-switch.ch/ By: Fraser, Ed. systems. One way to provide this is to produce a checksum of the unaltered file, store that checksum offline, and periodically (or when desired) check to make sure the checksum of the online file hasn't changed (which would indicate the data has been modified). Some operating systems come with checksumming programs, such as the UNIX sum program. However, these may not provide the protection you actually need. Files can be modified in such a way as to preserve the result of the UNIX sum program! Therefore, we suggest that you use a cryptographically strong program, such as the message digesting program MD5 [ref], to produce the checksums you will be using to assure integrity. There are other applications where integrity will need to be assured, such as when transmitting an email message between two parties. There are products available that can provide this capability. Once you identify that this is a capability you need, you can go about identifying technologies that will provide it. 4.4 Authorization Authorization refers to the process of granting privileges to processes and, ultimately, users. This differs from authentication in that authentication is the process used to identify a user. Once identified (reliably), the privileges, rights, property, and permissible actions of the user are determined by authorization. Explicitly listing the authorized activities of each user (and user process) with respect to all resources (objects) is impossible in a reasonable system. In a real system certain techniques are used to simplify the process of granting and checking authorization(s). One approach, popularized in UNIX systems, is to assign to each object three classes of user: owner, group and world. The owner is either the creator of the object or the user assigned as owner by the super-user. The owner permissions (read, write and execute) apply only to the owner. A group is a collection of users which share access rights to an object. The group permissions (read, write and execute) apply to all users in the group (except the owner). The world refers to everybody else with access to the system. The world permissions (read, write and execute) apply to all users (except the owner and members of the group). Another approach is to attach to an object a list which explicitly contains the identity of all permitted users (or groups). This is an Access Control List (ACL). The advantage of ACLs are that they are Formatted and Indexed by: page 29 of 75 instructional media + magic, inc. RFC2196 Web Address: Site Security Handbook http://sunsite.cnlab-switch.ch/ By: Fraser, Ed. easily maintained (one central list per object) and it's very easy to visually check who has access to what. The disadvantages are the extra resources required to store such lists, as well as the vast number of such lists required for large systems. 4.5 Access 4.5.1 Physical Access Restrict physical access to hosts, allowing access only to those people who are supposed to use the hosts. Hosts include "trusted" terminals (i.e., terminals which allow unauthenticated use such as system consoles, operator terminals and terminals dedicated to special tasks), and individual microcomputers and workstations, especially those connected to your network. Make sure people's work areas mesh well with access restrictions; otherwise they will find ways to circumvent your physical security (e.g., jamming doors open). Keep original and backup copies of data and programs safe. Apart from keeping them in good condition for backup purposes, they must be protected from theft. It is important to keep backups in a separate location from the originals, not only for damage considerations, but also to guard against thefts. Portable hosts are a particular risk. Make sure it won't cause problems if one of your staff's portable computer is stolen. Consider developing guidelines for the kinds of data that should be allowed to reside on the disks of portable computers as well as how the data should be protected (e.g., encryption) when it is on a portable computer. Other areas where physical access should be restricted is the wiring closets and important network elements like file servers, name server hosts, and routers. 4.5.2 Walk-up Network Connections By "walk-up" connections, we mean network connection points located to provide a convenient way for users to connect a portable host to your network. Consider whether you need to provide this service, bearing in mind that it allows any user to attach an unauthorized host to your network. This increases the risk of attacks via techniques such as Formatted and Indexed by: page 30 of 75 instructional media + magic, inc. RFC2196 Web Address: Site Security Handbook http://sunsite.cnlab-switch.ch/ By: Fraser, Ed. IP address spoofing, packet sniffing, etc. Users and site management must appreciate the risks involved. If you decide to provide walk-up connections, plan the service carefully and define precisely where you will provide it so that you can ensure the necessary physical access security. A walk-up host should be authenticated before its user is permitted to access resources on your network. As an alternative, it may be possible to control physical access. For example, if the service is to be used by students, you might only provide walk-up connection sockets in student laboratories. If you are providing walk-up access for visitors to connect back to their home networks (e.g., to read e-mail, etc.) in your facility, consider using a separate subnet that has no connectivity to the internal network. Keep an eye on any area that contains unmonitored access to the network, such as vacant offices. It may be sensible to disconnect such areas at the wiring closet, and consider using secure hubs and monitoring attempts to connect unauthorized hosts. 4.5.3 Other Network Technologies Technologies considered here include X.25, ISDN, SMDS, DDS and Frame Relay. All are provided via physical links which go through telephone exchanges, providing the potential for them to be diverted. Crackers are certainly interested in telephone switches as well as in data networks! With switched technologies, use Permanent Virtual Circuits or Closed User Groups whenever this is possible. Technologies which provide authentication and/or encryption (such as IPv6) are evolving rapidly; consider using them on links where security is important. 4.5.4 Modems 4.5.4.1 Modem Lines Must Be Managed Although they provide convenient access to a site for its users, they can also provide an effective detour around the site's firewalls. For this reason it is essential to maintain proper control of modems. Don't allow users to install a modem line without proper authorization. This includes temporary installations (e.g., plugging a modem into a facsimile or telephone line overnight). Formatted and Indexed by: page 31 of 75 instructional media + magic, inc. RFC2196 Web Address: Site Security Handbook http://sunsite.cnlab-switch.ch/ By: Fraser, Ed. Maintain a register of all your modem lines and keep your register up to date. Conduct regular (ideally automated) site checks for unauthorized modems. 4.5.4.2 Dial-in Users Must Be Authenticated A username and password check should be completed before a user can access anything on your network. Normal password security considerations are particularly important (see section 4.1.1). Remember that telephone lines can be tapped, and that it is quite easy to intercept messages to cellular phones. Modern high-speed modems use more sophisticated modulation techniques, which makes them somewhat more difficult to monitor, but it is prudent to assume that hackers know how to eavesdrop on your lines. For this reason, you should use one-time passwords if at all possible. It is helpful to have a single dial-in point (e.g., a single large modem pool) so that all users are authenticated in the same way. Users will occasionally mis-type a password. Set a short delay - say two seconds - after the first and second failed logins, and force a disconnect after the third. This will slow down automated password attacks. Don't tell the user whether the username, the password, or both, were incorrect. 4.5.4.3 Call-back Capability Some dial-in servers offer call-back facilities (i.e., the user dials in and is authenticated, then the system disconnects the call and calls back on a specified number). Call-back is useful since if someone were to guess a username and password, they are disconnected, and the system then calls back the actual user whose password was cracked; random calls from a server are suspicious, at best. This does mean users may only log in from one location (where the server is configured to dial them back), and of course there may be phone charges associated with there call-back location. This feature should be used with caution; it can easily be bypassed. At a minimum, make sure that the return call is never made from the same modem as the incoming one. Overall, although call-back can improve modem security, you should not depend on it alone. 4.5.4.4 All Logins Should Be Logged All logins, whether successful or unsuccessful should be logged. However, do not keep correct passwords in the log. Rather, log them simply as a successful login attempt. Since most bad passwords are Formatted and Indexed by: page 32 of 75 instructional media + magic, inc. RFC2196 Web Address: Site Security Handbook http://sunsite.cnlab-switch.ch/ By: Fraser, Ed. mistyped by authorized users, they only vary by a single character from the actual password. Therefore if you can't keep such a log secure, don't log it at all. If Calling Line Identification is available, take advantage of it by recording the calling number for each login attempt. Be sensitive to the privacy issues raised by Calling Line Identification. Also be aware that Calling Line Identification is not to be trusted (since intruders have been known to break into phone switches and forward phone numbers or make other changes); use the data for informational purposes only, not for authentication. 4.5.4.5 Choose Your Opening Banner Carefully Many sites use a system default contained in a message of the day file for their opening banner. Unfortunately, this often includes the type of host hardware or operating system present on the host. This can provide valuable information to a would-be intruder. Instead, each site should create its own specific login banner, taking care to only include necessary information. Display a short banner, but don't offer an "inviting" name (e.g., University of XYZ, Student Records System). Instead, give your site name, a short warning that sessions may be monitored, and a username/password prompt. Verify possible legal issues related to the text you put into the banner. For high-security applications, consider using a "blind" password (i.e., give no response to an incoming call until the user has typed in a password). This effectively simulates a dead modem. 4.5.4.6 Dial-out Authentication Dial-out users should also be authenticated, particularly since your site will have to pay their telephone charges. Never allow dial-out from an unauthenticated dial-in call, and consider whether you will allow it from an authenticated one. The goal here is to prevent callers using your modem pool as part of a chain of logins. This can be hard to detect, particularly if a hacker sets up a path through several hosts on your site. At a minimum, don't allow the same modems and phone lines to be used for both dial-in and dial-out. This can be implemented easily if you run separate dial-in and dial-out modem pools. Formatted and Indexed by: page 33 of 75 instructional media + magic, inc. RFC2196 Web Address: Site Security Handbook http://sunsite.cnlab-switch.ch/ By: Fraser, Ed. 4.5.4.7 Make Your Modem Programming as "Bullet-proof" as Possible Be sure modems can't be reprogrammed while they're in service. At a minimum, make sure that three plus signs won't put your dial-in modems into command mode! Program your modems to reset to your standard configuration at the start of each new call. Failing this, make them reset at the end of each call. This precaution will protect you against accidental reprogramming of your modems. Resetting at both the end and the beginning of each call will assure an even higher level of confidence that a new caller will not inherit a previous caller's session. Check that your modems terminate calls cleanly. When a user logs out from an access server, verify that the server hangs up the phone line properly. It is equally important that the server forces logouts from whatever sessions were active if the user hangs up unexpectedly. 4.6 Auditing This section covers the procedures for collecting data generated by network activity, which may be useful in analyzing the security of a network and responding to security incidents. 4.6.1 What to Collect Audit data should include any attempt to achieve a different security level by any person, process, or other entity in the network. This includes login and logout, super user access (or the non-UNIX equivalent), ticket generation (for Kerberos, for example), and any other change of access or status. It is especially important to note "anonymous" or "guest" access to public servers. The actual data to collect will differ for different sites and for different types of access changes within a site. In general, the information you want to collect includes: username and hostname, for login and logout; previous and new access rights, for a change of access rights; and a timestamp. Of course, there is much more information which might be gathered, depending on what the system makes available and how much space is available to store that information. One very important note: do not gather passwords. This creates an enormous potential security breach if the audit records should be improperly accessed. Do not gather incorrect passwords either, as they often differ from valid passwords by only a single character or transposition. Formatted and Indexed by: page 34 of 75 instructional media + magic, inc. RFC2196 Web Address: Site Security Handbook http://sunsite.cnlab-switch.ch/ By: Fraser, Ed. 4.6.2 Collection Process The collection process should be enacted by the host or resource being accessed. Depending on the importance of the data and the need to have it local in instances in which services are being denied, data could be kept local to the resource until needed or be transmitted to storage after each event. There are basically three ways to store audit records: in a read/write file on a host, on a write-once/read-many device (e.g., a CD-ROM or a specially configured tape drive), or on a write-only device (e.g., a line printer). Each method has advantages and disadvantages. File system logging is the least resource intensive of the three methods and the easiest to configure. It allows instant access to the records for analysis, which may be important if an attack is in progress. File system logging is also the least reliable method. If the logging host has been compromised, the file system is usually the first thing to go; an intruder could easily cover up traces of the intrusion. Collecting audit data on a write-once device is slightly more effort to configure than a simple file, but it has the significant advantage of greatly increased security because an intruder could not alter the data showing that an intrusion has occurred. The disadvantage of this method is the need to maintain a supply of storage media and the cost of that media. Also, the data may not be instantly available. Line printer logging is useful in system where permanent and immediate logs are required. A real time system is an example of this, where the exact point of a failure or attack must be recorded. A laser printer, or other device which buffers data (e.g., a print server), may suffer from lost data if buffers contain the needed data at a critical instant. The disadvantage of, literally, "paper trails" is the need to keep the printer fed and the need to scan records by hand. There is also the issue of where to store the, potentially, enormous volume of paper which may be generated. For each of the logging methods described, there is also the issue of securing the path between the device generating the log and actual logging device (i.e., the file server, tape/CD-ROM drive, printer). If that path is compromised, logging can be stopped or spoofed or both. In an ideal world, the logging device would be directly Formatted and Indexed by: page 35 of 75 instructional media + magic, inc. RFC2196 Web Address: Site Security Handbook http://sunsite.cnlab-switch.ch/ By: Fraser, Ed. attached by a single, simple, point-to-point cable. Since that is usually impractical, the path should pass through the minimum number of networks and routers. Even if logs can be blocked, spoofing can be prevented with cryptographic checksums (it probably isn't necessary to encrypt the logs because they should not contain sensitive information in the first place). 4.6.3 Collection Load Collecting audit data may result in a rapid accumulation of bytes so storage availability for this information must be considered in advance. There are a few ways to reduce the required storage space. First, data can be compressed, using one of many methods. Or, the required space can be minimized by keeping data for a shorter period of time with only summaries of that data kept in long-term archives. One major drawback to the latter method involves incident response. Often, an incident has been ongoing for some period of time when a site notices it and begins to investigate. At that point in time, it's very helpful to have detailed audit logs available. If these are just summaries, there may not be sufficient detail to fully handle the incident. 4.6.4 Handling and Preserving Audit Data Audit data should be some of the most carefully secured data at the site and in the backups. If an intruder were to gain access to audit logs, the systems themselves, in addition to the data, would be at risk. Audit data may also become key to the investigation, apprehension, and prosecution of the perpetrator of an incident. For this reason, it is advisable to seek the advice of legal council when deciding how audit data should be treated. This should happen before an incident occurs. If a data handling plan is not adequately defined prior to an incident, it may mean that there is no recourse in the aftermath of an event, and it may create liability resulting from improper treatment of the data. 4.6.5 Legal Considerations Due to the content of audit data, there are a number of legal questions that arise which might need to be addressed by your legal counsel. If you collect and save audit data, you need to be prepared for consequences resulting both from its existence and its content. Formatted and Indexed by: page 36 of 75 instructional media + magic, inc. RFC2196 Web Address: Site Security Handbook http://sunsite.cnlab-switch.ch/ By: Fraser, Ed. One area concerns the privacy of individuals. In certain instances, audit data may contain personal information. Searching through the data, even for a routine check of the system's security, could represent an invasion of privacy. A second area of concern involves knowledge of intrusive behavior originating from your site. If an organization keeps audit data, is it responsible for examining it to search for incidents? If a host in one organization is used as a launching point for an attack against another organization, can the second organization use the audit data of the first organization to prove negligence on the part of that organization? The above examples are meant to be comprehensive, but should motivate your organization to consider the legal issues involved with audit data. 4.7 Securing Backups The procedure of creating backups is a classic part of operating a computer system. Within the context of this document, backups are addressed as part of the overall security plan of a site. There are several aspects to backups that are important within this context: (1) Make sure your site is creating backups (2) Make sure your site is using offsite storage for backups. The storage site should be carefully selected for both its security and its availability. (3) Consider encrypting your backups to provide additional protection of the information once it is off-site. However, be aware that you will need a good key management scheme so that you'll be able to recover data at any point in the future. Also, make sure you will have access to the necessary decryption programs at such time in the future as you need to perform the decryption. (4) Don't always assume that your backups are good. There have been many instances of computer security incidents that have gone on for long periods of time before a site has noticed the incident. In such cases, backups of the affected systems are also tainted. (5) Periodically verify the correctness and completeness of your backups. 5. Security Incident Handling This chapter of the document will supply guidance to be used before, during, and after a computer security incident occurs on a host, network, site, or multi-site environment. The operative philosophy in the event of a breach of computer security is to react according Formatted and Indexed by: page 37 of 75 instructional media + magic, inc. RFC2196 Web Address: Site Security Handbook http://sunsite.cnlab-switch.ch/ By: Fraser, Ed. to a plan. This is true whether the breach is the result of an external intruder attack, unintentional damage, a student testing some new program to exploit a software vulnerability, or a disgruntled employee. Each of the possible types of events, such as those just listed, should be addressed in advance by adequate contingency plans. Traditional computer security, while quite important in the overall site security plan, usually pays little attention to how to actually handle an attack once one occurs. The result is that when an attack is in progress, many decisions are made in haste and can be damaging to tracking down the source of the incident, collecting evidence to be used in prosecution efforts, preparing for the recovery of the system, and protecting the valuable data contained on the system. One of the most important, but often overlooked, benefits for efficient incident handling is an economic one. Having both technical and managerial personnel respond to an incident requires considerable resources. If trained to handle incidents efficiently, less staff time is required when one occurs. Due to the world-wide network most incidents are not restricted to a single site. Operating systems vulnerabilities apply (in some cases) to several millions of systems, and many vulnerabilities are exploited within the network itself. Therefore, it is vital that all sites with involved parties be informed as soon as possible. Another benefit is related to public relations. News about computer security incidents tends to be damaging to an organization's stature among current or potential clients. Efficient incident handling minimizes the potential for negative exposure. A final benefit of efficient incident handling is related to legal issues. It is possible that in the near future organizations may be held responsible because one of their nodes was used to launch a network attack. In a similar vein, people who develop patches or workarounds may be sued if the patches or workarounds are ineffective, resulting in compromise of the systems, or, if the patches or workarounds themselves damage systems. Knowing about operating system vulnerabilities and patterns of attacks, and then taking appropriate measures to counter these potential threats, is critical to circumventing possible legal problems. Formatted and Indexed by: page 38 of 75 instructional media + magic, inc. RFC2196 Web Address: Site Security Handbook http://sunsite.cnlab-switch.ch/ By: Fraser, Ed. The sections in this chapter provide an outline and starting point for creating your site's policy for handling security incidents. The sections are: (1) Preparing and planning (what are the goals and objectives in handling an incident). (2) Notification (who should be contacted in the case of an incident). - Local managers and personnel - Law enforcement and investigative agencies - Computer security incidents handling teams - Affected and involved sites - Internal communications - Public relations and press releases (3) Identifying an incident (is it an incident and how serious is it). (4) Handling (what should be done when an incident occurs). - Notification (who should be notified about the incident) - Protecting evidence and activity logs (what records should be kept from before, during, and after the incident) - Containment (how can the damage be limited) - Eradication (how to eliminate the reasons for the incident) - Recovery (how to reestablish service and systems) - Follow Up (what actions should be taken after the incident) (5) Aftermath (what are the implications of past incidents). (6) Administrative response to incidents. The remainder of this chapter will detail the issues involved in each of the important topics listed above, and provide some guidance as to what should be included in a site policy for handling incidents. 5.1 Preparing and Planning for Incident Handling Part of handling an incident is being prepared to respond to an incident before the incident occurs in the first place. This includes establishing a suitable level of protections as explained in the preceding chapters. Doing this should help your site prevent incidents as well as limit potential damage resulting from them when they do occur. Protection also includes preparing incident handling guidelines as part of a contingency plan for your organization or site. Having written plans eliminates much of the ambiguity which occurs during an incident, and will lead to a more appropriate and thorough set of responses. It is vitally important to test the proposed plan before an incident occurs through "dry runs". A team might even consider hiring a tiger team to act in parallel with the dry run. (Note: a tiger team is a team of specialists that try to penetrate the security of a system.) Formatted and Indexed by: page 39 of 75 instructional media + magic, inc. RFC2196 Web Address: Site Security Handbook http://sunsite.cnlab-switch.ch/ By: Fraser, Ed. Learning to respond efficiently to an incident is important for a number of reasons: (1) Protecting the assets which could be compromised (2) Protecting resources which could be utilized more profitably if an incident did not require their services (3) Complying with (government or other) regulations (4) Preventing the use of your systems in attacks against other systems (which could cause you to incur legal liability) (5) Minimizing the potential for negative exposure As in any set of pre-planned procedures, attention must be paid to a set of goals for handling an incident. These goals will be prioritized differently depending on the site. A specific set of objectives can be identified for dealing with incidents: (1) Figure out how it happened. (2) Find out how to avoid further exploitation of the same vulnerability. (3) Avoid escalation and further incidents. (4) Assess the impact and damage of the incident. (5) Recover from the incident. (6) Update policies and procedures as needed. (7) Find out who did it (if appropriate and possible). Due to the nature of the incident, there might be a conflict between analyzing the original source of a problem and restoring systems and services. Overall goals (like assuring the integrity of critical systems) might be the reason for not analyzing an incident. Of course, this is an important management decision; but all involved parties must be aware that without analysis the same incident may happen again. It is also important to prioritize the actions to be taken during an incident well in advance of the time an incident occurs. Sometimes an incident may be so complex that it is impossible to do everything at once to respond to it; priorities are essential. Although priorities will vary from institution to institution, the following suggested priorities may serve as a starting point for defining your organization's response: (1) Priority one -- protect human life and people's safety; human life always has precedence over all other considerations. (2) Priority two -- protect classified and/or sensitive data. Prevent exploitation of classified and/or sensitive systems, networks or sites. Inform affected Formatted and Indexed by: page 40 of 75 instructional media + magic, inc. RFC2196 Web Address: Site Security Handbook http://sunsite.cnlab-switch.ch/ By: Fraser, Ed. classified and/or sensitive systems, networks or sites about already occurred penetrations. (Be aware of regulations by your site or by government) (3) Priority three -- protect other data, including proprietary, scientific, managerial and other data, because loss of data is costly in terms of resources. Prevent exploitations of other systems, networks or sites and inform already affected systems, networks or sites about successful penetrations. (4) Priority four -- prevent damage to systems (e.g., loss or alteration of system files, damage to disk drives, etc.). Damage to systems can result in costly down time and recovery. (5) Priority five -- minimize disruption of computing resources (including processes). It is better in many cases to shut a system down or disconnect from a network than to risk damage to data or systems. Sites will have to evaluate the trade-offs between shutting down and disconnecting, and staying up. There may be service agreements in place that may require keeping systems up even in light of further damage occurring. However, the damage and scope of an incident may be so extensive that service agreements may have to be over-ridden. An important implication for defining priorities is that once human life and national security considerations have been addressed, it is generally more important to save data than system software and hardware. Although it is undesirable to have any damage or loss during an incident, systems can be replaced. However, the loss or compromise of data (especially classified or proprietary data) is usually not an acceptable outcome under any circumstances. Another important concern is the effect on others, beyond the systems and networks where the incident occurs. Within the limits imposed by government regulations it is always important to inform affected parties as soon as possible. Due to the legal implications of this topic, it should be included in the planned procedures to avoid further delays and uncertainties for the administrators. Any plan for responding to security incidents should be guided by local policies and regulations. Government and private sites that deal with classified material have specific rules that they must follow. Formatted and Indexed by: page 41 of 75 instructional media + magic, inc. RFC2196 Web Address: Site Security Handbook http://sunsite.cnlab-switch.ch/ By: Fraser, Ed. The policies chosen by your site on how it reacts to incidents will shape your response. For example, it may make little sense to create mechanisms to monitor and trace intruders if your site does not plan to take action against the intruders if they are caught. Other organizations may have policies that affect your plans. Telephone companies often release information about telephone traces only to law enforcement agencies. Handling incidents can be tedious and require any number of routine tasks that could be handled by support personnel. To free the technical staff it may be helpful to identify support staff who will help with tasks like: photocopying, fax'ing, etc. 5.2 Notification and Points of Contact It is important to establish contacts with various personnel before a real incident occurs. Many times, incidents are not real emergencies. Indeed, often you will be able to handle the activities internally. However, there will also be many times when others outside your immediate department will need to be included in the incident handling. These additional contacts include local managers and system administrators, administrative contacts for other sites on the Internet, and various investigative organizations. Getting to know these contacts before incidents occurs will help to make your incident handling process more efficient. For each type of communication contact, specific "Points of Contact" (POC) should be defined. These may be technical or administrative in nature and may include legal or investigative agencies as well as service providers and vendors. When establishing these contact, it is important to decide how much information will be shared with each class of contact. It is especially important to define, ahead of time, what information will be shared with the users at a site, with the public (including the press), and with other sites. Settling these issues are especially important for the local person responsible for handling the incident, since that is the person responsible for the actual notification of others. A list of contacts in each of these categories is an important time saver for this person during an incident. It can be quite difficult to find an appropriate person during an incident when many urgent events are ongoing. It is strongly recommended that all relevant telephone numbers (also electronic mail addresses and fax numbers) be included in the site security policy. The names and contact information of all individuals who will be directly involved in the handling of an incident should be placed at the top of this list. Formatted and Indexed by: page 42 of 75 instructional media + magic, inc. RFC2196 Web Address: Site Security Handbook http://sunsite.cnlab-switch.ch/ By: Fraser, Ed. 5.2.1 Local Managers and Personnel When an incident is under way, a major issue is deciding who is in charge of coordinating the activity of the multitude of players. A major mistake that can be made is to have a number of people who are each working independently, but are not working together. This will only add to the confusion of the event and will probably lead to wasted or ineffective effort. The single POC may or may not be the person responsible for handling the incident. There are two distinct roles to fill when deciding who shall be the POC and who will be the person in charge of the incident. The person in charge of the incident will make decisions as to the interpretation of policy applied to the event. In contrast, the POC must coordinate the effort of all the parties involved with handling the event. The POC must be a person with the technical expertise to successfully coordinate the efforts of the system managers and users involved in monitoring and reacting to the attack. Care should be taken when identifying who this person will be. It should not necessarily be the same person who has administrative responsibility for the compromised systems since often such administrators have knowledge only sufficient for the day to day use of the computers, and lack in depth technical expertise. Another important function of the POC is to maintain contact with law enforcement and other external agencies to assure that multi-agency involvement occurs. The level of involvement will be determined by management decisions as well as legal constraints. A single POC should also be the single person in charge of collecting evidence, since as a rule of thumb, the more people that touch a potential piece of evidence, the greater the possibility that it will be inadmissible in court. To ensure that evidence will be acceptable to the legal community, collecting evidence should be done following predefined procedures in accordance with local laws and legal regulations. One of the most critical tasks for the POC is the coordination of all relevant processes. Responsibilities may be distributed over the whole site, involving multiple independent departments or groups. This will require a well coordinated effort in order to achieve overall success. The situation becomes even more complex if multiple sites are involved. When this happens, rarely will a single POC at one site be able to adequately coordinate the handling of the entire incident. Instead, appropriate incident response teams should be involved. Formatted and Indexed by: page 43 of 75 instructional media + magic, inc. RFC2196 Web Address: Site Security Handbook http://sunsite.cnlab-switch.ch/ By: Fraser, Ed. The incident handling process should provide some escalation mechanisms. In order to define such a mechanism, sites will need to create an internal classification scheme for incidents. Associated with each level of incident will be the appropriate POC and procedures. As an incident is escalated, there may be a change in the POC which will need to be communicated to all others involved in handling the incident. When a change in the POC occurs, old POC should brief the new POC in all background information. Lastly, users must know how to report suspected incidents. Sites should establish reporting procedures that will work both during and outside normal working hours. Help desks are often used to receive these reports during normal working hours, while beepers and telephones can be used for out of hours reporting. 5.2.2 Law Enforcement and Investigative Agencies In the event of an incident that has legal consequences, it is important to establish contact with investigative agencies (e.g, the FBI and Secret Service in the U.S.) as soon as possible. Local law enforcement, local security offices, and campus police departments should also be informed as appropriate. This section describes many of the issues that will be confronted, but it is acknowledged that each organization will have its own local and governmental laws and regulations that will impact how they interact with law enforcement and investigative agencies. The most important point to make is that each site needs to work through these issues. A primary reason for determining these point of contact well in advance of an incident is that once a major attack is in progress, there is little time to call these agencies to determine exactly who the correct point of contact is. Another reason is that it is important to cooperate with these agencies in a manner that will foster a good working relationship, and that will be in accordance with the working procedures of these agencies. Knowing the working procedures in advance, and the expectations of your point of contact is a big step in this direction. For example, it is important to gather evidence that will be admissible in any subsequent legal proceedings, and this will require prior knowledge of how to gather such evidence. A final reason for establishing contacts as soon as possible is that it is impossible to know the particular agency that will assume jurisdiction in any given incident. Making contacts and finding the proper channels early on will make responding to an incident go considerably more smoothly. Formatted and Indexed by: page 44 of 75 instructional media + magic, inc. RFC2196 Web Address: Site Security Handbook http://sunsite.cnlab-switch.ch/ By: Fraser, Ed. If your organization or site has a legal counsel, you need to notify this office soon after you learn that an incident is in progress. At a minimum, your legal counsel needs to be involved to protect the legal and financial interests of your site or organization. There are many legal and practical issues, a few of which are: (1) Whether your site or organization is willing to risk negative publicity or exposure to cooperate with legal prosecution efforts. (2) Downstream liability--if you leave a compromised system as is so it can be monitored and another computer is damaged because the attack originated from your system, your site or organization may be liable for damages incurred. (3) Distribution of information--if your site or organization distributes information about an attack in which another site or organization may be involved or the vulnerability in a product that may affect ability to market that product, your site or organization may again be liable for any damages (including damage of reputation). (4) Liabilities due to monitoring--your site or organization may be sued if users at your site or elsewhere discover that your site is monitoring account activity without informing users. Unfortunately, there are no clear precedents yet on the liabilities or responsibilities of organizations involved in a security incident or who might be involved in supporting an investigative effort. Investigators will often encourage organizations to help trace and monitor intruders. Indeed, most investigators cannot pursue computer intrusions without extensive support from the organizations involved. However, investigators cannot provide protection from liability claims, and these kinds of efforts may drag out for months and may take a lot of effort. On the other hand, an organization's legal council may advise extreme caution and suggest that tracing activities be halted and an intruder shut out of the system. This, in itself, may not provide protection from liability, and may prevent investigators from identifying the perpetrator. The balance between supporting investigative activity and limiting liability is tricky. You'll need to consider the advice of your legal counsel and the damage the intruder is causing (if any) when making your decision about what to do during any particular incident. Formatted and Indexed by: page 45 of 75 instructional media + magic, inc. RFC2196 Web Address: Site Security Handbook http://sunsite.cnlab-switch.ch/ By: Fraser, Ed. Your legal counsel should also be involved in any decision to contact investigative agencies when an incident occurs at your site. The decision to coordinate efforts with investigative agencies is most properly that of your site or organization. Involving your legal counsel will also foster the multi-level coordination between your site and the particular investigative agency involved, which in turn results in an efficient division of labor. Another result is that you are likely to obtain guidance that will help you avoid future legal mistakes. Finally, your legal counsel should evaluate your site's written procedures for responding to incidents. It is essential to obtain a "clean bill of health" from a legal perspective before you actually carry out these procedures. It is vital, when dealing with investigative agencies, to verify that the person who calls asking for information is a legitimate representative from the agency in question. Unfortunately, many well intentioned people have unknowingly leaked sensitive details about incidents, allowed unauthorized people into their systems, etc., because a caller has masqueraded as a representative of a government agency. (Note: this word of caution actually applies to all external contacts.) A similar consideration is using a secure means of communication. Because many network attackers can easily re-route electronic mail, avoid using electronic mail to communicate with other agencies (as well as others dealing with the incident at hand). Non-secured phone lines (the phones normally used in the business world) are also frequent targets for tapping by network intruders, so be careful! There is no one established set of rules for responding to an incident when the local government becomes involved. Normally (in the U.S.), except by legal order, no agency can force you to monitor, to disconnect from the network, to avoid telephone contact with the suspected attackers, etc. Each organization will have a set of local and national laws and regulations that must be adhered to when handling incidents. It is recommended that each site be familiar with those laws and regulations, and identify and get know the contacts for agencies with jurisdiction well in advance of handling an incident. 5.2.3 Computer Security Incident Handling Teams There are currently a number of of Computer Security Incident Response teams (CSIRTs) such as the CERT Coordination Center, the German DFN-CERT, and other teams around the globe. Teams exist for many major government agencies and large corporations. If such a Formatted and Indexed by: page 46 of 75 instructional media + magic, inc. RFC2196 Web Address: Site Security Handbook http://sunsite.cnlab-switch.ch/ By: Fraser, Ed. team is available, notifying it should be of primary consideration during the early stages of an incident. These teams are responsible for coordinating computer security incidents over a range of sites and larger entities. Even if the incident is believed to be contained within a single site, it is possible that the information available through a response team could help in fully resolving the incident. If it is determined that the breach occurred due to a flaw in the system's hardware or software, the vendor (or supplier) and a Computer Security Incident Handling team should be notified as soon as possible. This is especially important because many other systems are vulnerable, and these vendor and response team organizations can help disseminate help to other affected sites. In setting up a site policy for incident handling, it may be desirable to create a subgroup, much like those teams that already exist, that will be responsible for handling computer security incidents for the site (or organization). If such a team is created, it is essential that communication lines be opened between this team and other teams. Once an incident is under way, it is difficult to open a trusted dialogue between other teams if none has existed before. 5.2.4 Affected and Involved Sites If an incident has an impact on other sites, it is good practice to inform them. It may be obvious from the beginning that the incident is not limited to the local site, or it may emerge only after further analysis. Each site may choose to contact other sites directly or they can pass the information to an appropriate incident response team. It is often very difficult to find the responsible POC at remote sites and the incident response team will be able to facilitate contact by making use of already established channels. The legal and liability issues arising from a security incident will differ from site to site. It is important to define a policy for the sharing and logging of information about other sites before an incident occurs. Information about specific people is especially sensitive, and may be subject to privacy laws. To avoid problems in this area, irrelevant information should be deleted and a statement of how to handle the remaining information should be included. A clear statement of how this information is to be used is essential. No one who informs a site of a security incident wants to read about it in the public Formatted and Indexed by: page 47 of 75 instructional media + magic, inc. RFC2196 Web Address: Site Security Handbook http://sunsite.cnlab-switch.ch/ By: Fraser, Ed. press. Incident response teams are valuable in this respect. When they pass information to responsible POCs, they are able to protect the anonymity of the original source. But, be aware that, in many cases, the analysis of logs and information at other sites will reveal addresses of your site. All the problems discussed above should be not taken as reasons not to involve other sites. In fact, the experiences of existing teams reveal that most sites informed about security problems are not even aware that their site had been compromised. Without timely information, other sites are often unable to take action against intruders. 5.2.5 Internal Communications It is crucial during a major incident to communicate why certain actions are being taken, and how the users (or departments) are expected to behave. In particular, it should be made very clear to users what they are allowed to say (and not say) to the outside world (including other departments). For example, it wouldn't be good for an organization if users replied to customers with something like, "I'm sorry the systems are down, we've had an intruder and we are trying to clean things up." It would be much better if they were instructed to respond with a prepared statement like, "I'm sorry our systems are unavailable, they are being maintained for better service in the future." Communications with customers and contract partners should be handled in a sensible, but sensitive way. One can prepare for the main issues by preparing a checklist. When an incident occurs, the checklist can be used with the addition of a sentence or two for the specific circumstances of the incident. Public relations departments can be very helpful during incidents. They should be involved in all planning and can provide well constructed responses for use when contact with outside departments and organizations is necessary. 5.2.6 Public Relations - Press Releases There has been a tremendous growth in the amount of media coverage dedicated to computer security incidents in the United States. Such press coverage is bound to extend to other countries as the Internet continues to grow and expand internationally. Readers from countries where such media attention has not yet occurred, can learn from the experiences in the U.S. and should be forwarned and prepared. Formatted and Indexed by: page 48 of 75 instructional media + magic, inc. RFC2196 Web Address: Site Security Handbook http://sunsite.cnlab-switch.ch/ By: Fraser, Ed. One of the most important issues to consider is when, who, and how much to release to the general public through the press. There are many issues to consider when deciding this particular issue. First and foremost, if a public relations office exists for the site, it is important to use this office as liaison to the press. The public relations office is trained in the type and wording of information released, and will help to assure that the image of the site is protected during and after the incident (if possible). A public relations office has the advantage that you can communicate candidly with them, and provide a buffer between the constant press attention and the need of the POC to maintain control over the incident. If a public relations office is not available, the information released to the press must be carefully considered. If the information is sensitive, it may be advantageous to provide only minimal or overview information to the press. It is quite possible that any information provided to the press will be quickly reviewed by the perpetrator of the incident. Also note that misleading the press can often backfire and cause more damage than releasing sensitive information. While it is difficult to determine in advance what level of detail to provide to the press, some guidelines to keep in mind are: (1) Keep the technical level of detail low. Detailed information about the incident may provide enough information for others to launch similar attacks on other sites, or even damage the site's ability to prosecute the guilty party once the event is over. (2) Keep the speculation out of press statements. Speculation of who is causing the incident or the motives are very likely to be in error and may cause an inflamed view of the incident. (3) Work with law enforcement professionals to assure that evidence is protected. If prosecution is involved, assure that the evidence collected is not divulged to the press. (4) Try not to be forced into a press interview before you are prepared. The popular press is famous for the "2 am" interview, where the hope is to catch the interviewee off guard and obtain information otherwise not available. (5) Do not allow the press attention to detract from the handling of the event. Always remember that the successful closure of an incident is of primary importance. Formatted and Indexed by: page 49 of 75 instructional media + magic, inc. RFC2196 Web Address: Site Security Handbook http://sunsite.cnlab-switch.ch/ By: Fraser, Ed. 5.3 Identifying an Incident 5.3.1 Is It Real? This stage involves determining if a problem really exists. Of course many if not most signs often associated with virus infection, system intrusions, malicious users, etc., are simply anomalies such as hardware failures or suspicious system/user behavior. To assist in identifying whether there really is an incident, it is usually helpful to obtain and use any detection software which may be available. Audit information is also extremely useful, especially in determining whether there is a network attack. It is extremely important to obtain a system snapshot as soon as one suspects that something is wrong. Many incidents cause a dynamic chain of events to occur, and an initial system snapshot may be the most valuable tool for identifying the problem and any source of attack. Finally, it is important to start a log book. Recording system events, telephone conversations, time stamps, etc., can lead to a more rapid and systematic identification of the problem, and is the basis for subsequent stages of incident handling. There are certain indications or "symptoms" of an incident that deserve special attention: (1) System crashes. (2) New user accounts (the account RUMPLESTILTSKIN has been unexpectedly created), or high activity on a previously low usage account. (3) New files (usually with novel or strange file names, such as data.xx or k or .xx ). (4) Accounting discrepancies (in a UNIX system you might notice the shrinking of an accounting file called /usr/admin/lastlog, something that should make you very suspicious that there may be an intruder). (5) Changes in file lengths or dates (a user should be suspicious if .EXE files in an MS DOS computer have unexplainedly grown by over 1800 bytes). (6) Attempts to write to system (a system manager notices that a privileged user in a VMS system is attempting to alter RIGHTSLIST.DAT). (7) Data modification or deletion (files start to disappear). (8) Denial of service (a system manager and all other users become locked out of a UNIX system, now in single user mode). (9) Unexplained, poor system performance (10) Anomalies ("GOTCHA" is displayed on the console or there are frequent unexplained "beeps"). (11) Suspicious probes (there are numerous unsuccessful login attempts from another node). Formatted and Indexed by: page 50 of 75 instructional media + magic, inc. RFC2196 Web Address: Site Security Handbook http://sunsite.cnlab-switch.ch/ By: Fraser, Ed. (12) Suspicious browsing (someone becomes a root user on a UNIX system and accesses file after file on many user accounts.) (13) Inability of a user to log in due to modifications of his/her account. By no means is this list comprehensive; we have just listed a number of common indicators. It is best to collaborate with other technical and computer security personnel to make a decision as a group about whether an incident is occurring. 5.3.2 Types and Scope of Incidents Along with the identification of the incident is the evaluation of the scope and impact of the problem. It is important to correctly identify the boundaries of the incident in order to effectively deal with it and prioritize responses. In order to identify the scope and impact a set of criteria should be defined which is appropriate to the site and to the type of connections available. Some of the issues include: (1) Is this a multi-site incident? (2) Are many computers at your site affected by this incident? (3) Is sensitive information involved? (4) What is the entry point of the incident (network, phone line, local terminal, etc.)? (5) Is the press involved? (6) What is the potential damage of the incident? (7) What is the estimated time to close out the incident? (8) What resources could be required to handle the incident? (9) Is law enforcement involved? 5.3.3 Assessing the Damage and Extent The analysis of the damage and extent of the incident can be quite time consuming, but should lead to some insight into the nature of the incident, and aid investigation and prosecution. As soon as the breach has occurred, the entire system and all of its components should be considered suspect. System software is the most probable target. Preparation is key to be able to detect all changes for a possibly tainted system. This includes checksumming all media from the vendor using a algorithm which is resistant to tampering. (See sections 4.3) Assuming original vendor distribution media are available, an analysis of all system files should commence, and any irregularities should be noted and referred to all parties involved in handling the incident. It can be very difficult, in some cases, to decide which Formatted and Indexed by: page 51 of 75 instructional media + magic, inc. RFC2196 Web Address: Site Security Handbook http://sunsite.cnlab-switch.ch/ By: Fraser, Ed. backup media are showing a correct system status. Consider, for example, that the incident may have continued for months or years before discovery, and the suspect may be an employee of the site, or otherwise have intimate knowledge or access to the systems. In all cases, the pre-incident preparation will determine what recovery is possible. If the system supports centralized logging (most do), go back over the logs and look for abnormalities. If process accounting and connect time accounting is enabled, look for patterns of system usage. To a lesser extent, disk usage may shed light on the incident. Accounting can provide much helpful information in an analysis of an incident and subsequent prosecution. Your ability to address all aspects of a specific incident strongly depends on the success of this analysis. 5.4 Handling an Incident Certain steps are necessary to take during the handling of an incident. In all security related activities, the most important point to be made is that all sites should have policies in place. Without defined policies and goals, activities undertaken will remain without focus. The goals should be defined by management and legal counsel in advance. One of the most fundamental objectives is to restore control of the affected systems and to limit the impact and damage. In the worst case scenario, shutting down the system, or disconnecting the system from the network, may the only practical solution. As the activities involved are complex, try to get as much help as necessary. While trying to solve the problem alone, real damage might occur due to delays or missing information. Most administrators take the discovery of an intruder as a personal challenge. By proceeding this way, other objectives as outlined in the local policies may not always be considered. Trying to catch intruders may be a very low priority, compared to system integrity, for example. Monitoring a hacker's activity is useful, but it might not be considered worth the risk to allow the continued access. 5.4.1 Types of Notification and Exchange of Information When you have confirmed that an incident is occurring, the appropriate personnel must be notified. How this notification is achieved is very important to keeping the event under control both from a technical and emotional standpoint. The circumstances should be described in as much detail as possible, in order to aid prompt acknowledgment and understanding of the problem. Great care should Formatted and Indexed by: page 52 of 75 instructional media + magic, inc. RFC2196 Web Address: Site Security Handbook http://sunsite.cnlab-switch.ch/ By: Fraser, Ed. be taken when determining to which groups detailed technical information is given during the notification. For example, it is helpful to pass this kind of information to an incident handling team as they can assist you by providing helpful hints for eradicating the vulnerabilities involved in an incident. On the other hand, putting the critical knowledge into the public domain (e.g., via USENET newsgroups or mailing lists) may potentially put a large number of systems at risk of intrusion. It is invalid to assume that all administrators reading a particular newsgroup have access to operating system source code, or can even understand an advisory well enough to take adequate steps. First of all, any notification to either local or off-site personnel must be explicit. This requires that any statement (be it an electronic mail message, phone call, fax, beeper, or semaphone) providing information about the incident be clear, concise, and fully qualified. When you are notifying others that will help you handle an event, a "smoke screen" will only divide the effort and create confusion. If a division of labor is suggested, it is helpful to provide information to each participant about what is being accomplished in other efforts. This will not only reduce duplication of effort, but allow people working on parts of the problem to know where to obtain information relevant to their part of the incident. Another important consideration when communicating about the incident is to be factual. Attempting to hide aspects of the incident by providing false or incomplete information may not only prevent a successful resolution to the incident, but may even worsen the situation. The choice of language used when notifying people about the incident can have a profound effect on the way that information is received. When you use emotional or inflammatory terms, you raise the potential for damage and negative outcomes of the incident. It is important to remain calm both in written and spoken communications. Another consideration is that not all people speak the same language. Due to this fact, misunderstandings and delay may arise, especially if it is a multi-national incident. Other international concerns include differing legal implications of a security incident and cultural differences. However, cultural differences do not only exist between countries. They even exist within countries, between different social or user groups. For example, an administrator of a university system might be very relaxed about attempts to connect to the system via telnet, but the administrator of a military system is likely to consider the same action as a possible attack. Formatted and Indexed by: page 53 of 75 instructional media + magic, inc. RFC2196 Web Address: Site Security Handbook http://sunsite.cnlab-switch.ch/ By: Fraser, Ed. Another issue associated with the choice of language is the notification of non-technical or off-site personnel. It is important to accurately describe the incident without generating undue alarm or confusion. While it is more difficult to describe the incident to a non-technical audience, it is often more important. A non-technical description may be required for upper-level management, the press, or law enforcement liaisons. The importance of these communications cannot be underestimated and may make the difference between resolving the incident properly and escalating to some higher level of damage. If an incident response team becomes involved, it might be necessary to fill out a template for the information exchange. Although this may seem to be an additional burden and adds a certain delay, it helps the team to act on this minimum set of information. The response team may be able to respond to aspects of the incident of which the local administrator is unaware. If information is given out to someone else, the following minimum information should be provided: (1) timezone of logs, ... in GMT or local time (2) information about the remote system, including host names, IP addresses and (perhaps) user IDs (3) all log entries relevant for the remote site (4) type of incident (what happened, why should you care) If local information (i.e., local user IDs) is included in the log entries, it will be necessary to sanitize the entries beforehand to avoid privacy issues. In general, all information which might assist a remote site in resolving an incident should be given out, unless local policies prohibit this. 5.4.2 Protecting Evidence and Activity Logs When you respond to an incident, document all details related to the incident. This will provide valuable information to yourself and others as you try to unravel the course of events. Documenting all details will ultimately save you time. If you don't document every relevant phone call, for example, you are likely to forget a significant portion of information you obtain, requiring you to contact the source of information again. At the same time, recording details will provide evidence for prosecution efforts, providing the case moves in that direction. Documenting an incident will also help you perform a final assessment of damage (something your management, as well as law enforcement officers, will want to know), and will provide the basis for later phases of the handling process: eradication, recovery, and follow-up "lessons learned." Formatted and Indexed by: page 54 of 75 instructional media + magic, inc. RFC2196 Web Address: Site Security Handbook http://sunsite.cnlab-switch.ch/ By: Fraser, Ed. During the initial stages of an incident, it is often infeasible to determine whether prosecution is viable, so you should document as if you are gathering evidence for a court case. At a minimum, you should record: (1) all system events (audit records) (2) all actions you take (time tagged) (3) all external conversations (including the person with whom you talked, the date and time, and the content of the conversation) The most straightforward way to maintain documentation is keeping a log book. This allows you to go to a centralized, chronological source of information when you need it, instead of requiring you to page through individual sheets of paper. Much of this information is potential evidence in a court of law. Thus, when a legal follow-up is a possibility, one should follow the prepared procedures and avoid jeopardizing the legal follow-up by improper handling of possible evidence. If appropriate, the following steps may be taken. (1) Regularly (e.g., every day) turn in photocopied, signed copies of your logbook (as well as media you use to record system events) to a document custodian. (2) The custodian should store these copied pages in a secure place (e.g., a safe). (3) When you submit information for storage, you should receive a signed, dated receipt from the document custodian. Failure to observe these procedures can result in invalidation of any evidence you obtain in a court of law. 5.4.3 Containment The purpose of containment is to limit the extent of an attack. An essential part of containment is decision making (e.g., determining whether to shut a system down, disconnect from a network, monitor system or network activity, set traps, disable functions such as remote file transfer, etc.). Sometimes this decision is trivial; shut the system down if the information is classified, sensitive, or proprietary. Bear in mind that removing all access while an incident is in progress obviously notifies all users, including the alleged problem users, that the administrators are aware of a problem; this may have a deleterious Formatted and Indexed by: page 55 of 75 instructional media + magic, inc. RFC2196 Web Address: Site Security Handbook http://sunsite.cnlab-switch.ch/ By: Fraser, Ed. effect on an investigation. In some cases, it is prudent to remove all access or functionality as soon as possible, then restore normal operation in limited stages. In other cases, it is worthwhile to risk some damage to the system if keeping the system up might enable you to identify an intruder. This stage should involve carrying out predetermined procedures. Your organization or site should, for example, define acceptable risks in dealing with an incident, and should prescribe specific actions and strategies accordingly. This is especially important when a quick decision is necessary and it is not possible to first contact all involved parties to discuss the decision. In the absence of predefined procedures, the person in charge of the incident will often not have the power to make difficult management decisions (like to lose the results of a costly experiment by shutting down a system). A final activity that should occur during this stage of incident handling is the notification of appropriate authorities. 5.4.4 Eradication Once the incident has been contained, it is time to eradicate the cause. But before eradicating the cause, great care should be taken to collect all necessary information about the compromised system(s) and the cause of the incident as they will likely be lost when cleaning up the system. Software may be available to help you in the eradication process, such as anti-virus software. If any bogus files have been created, archive them before deleting them. In the case of virus infections, it is important to clean and reformat any media containing infected files. Finally, ensure that all backups are clean. Many systems infected with viruses become periodically re-infected simply because people do not systematically eradicate the virus from backups. After eradication, a new backup should be taken. Removing all vulnerabilities once an incident has occurred is difficult. The key to removing vulnerabilities is knowledge and understanding of the breach. It may be necessary to go back to the original distribution media and re-customize the system. To facilitate this worst case scenario, a record of the original system setup and each customization change should be maintained. In the case of a network-based attack, it is important to install patches for each operating system vulnerability which was exploited. Formatted and Indexed by: page 56 of 75 instructional media + magic, inc. RFC2196 Web Address: Site Security Handbook http://sunsite.cnlab-switch.ch/ By: Fraser, Ed. As discussed in section 5.4.2, a security log can be most valuable during this phase of removing vulnerabilities. The logs showing how the incident was discovered and contained can be used later to help determine how extensive the damage was from a given incident. The steps taken can be used in the future to make sure the problem does not resurface. Ideally, one should automate and regularly apply the same test as was used to detect the security incident. If a particular vulnerability is isolated as having been exploited, the next step is to find a mechanism to protect your system. The security mailing lists and bulletins would be a good place to search for this information, and you can get advice from incident response teams. 5.4.5 Recovery Once the cause of an incident has been eradicated, the recovery phase defines the next stage of action. The goal of recovery is to return the system to normal. In general, bringing up services in the order of demand to allow a minimum of user inconvenience is the best practice. Understand that the proper recovery procedures for the system are extremely important and should be specific to the site. 5.4.6 Follow-Up Once you believe that a system has been restored to a "safe" state, it is still possible that holes, and even traps, could be lurking in the system. One of the most important stages of responding to incidents is also the most often omitted, the follow-up stage. In the follow-up stage, the system should be monitored for items that may have been missed during the cleanup stage. It would be prudent to utilize some of the tools mentioned in chapter 7 as a start. Remember, these tools don't replace continual system monitoring and good systems administration practices. The most important element of the follow-up stage is performing a postmortem analysis. Exactly what happened, and at what times? How well did the staff involved with the incident perform? What kind of information did the staff need quickly, and how could they have gotten that information as soon as possible? What would the staff do differently next time? After an incident, it is prudent to write a report describing the exact sequence of events: the method of discovery, correction procedure, monitoring procedure, and a summary of lesson learned. This will aid in the clear understanding of the problem. Creating a formal chronology of events (including time stamps) is also important for legal reasons. Formatted and Indexed by: page 57 of 75 instructional media + magic, inc. RFC2196 Web Address: Site Security Handbook http://sunsite.cnlab-switch.ch/ By: Fraser, Ed. A follow-up report is valuable for many reasons. It provides a reference to be used in case of other similar incidents. It is also important to, as quickly as possible obtain a monetary estimate of the amount of damage the incident caused. This estimate should include costs associated with any loss of software and files (especially the value of proprietary data that may have been disclosed), hardware damage, and manpower costs to restore altered files, reconfigure affected systems, and so forth. This estimate may become the basis for subsequent prosecution activity. The report can also help justify an organization's computer security effort to management. 5.5 Aftermath of an Incident In the wake of an incident, several actions should take place. These actions can be summarized as follows: (1) An inventory should be taken of the systems' assets, (i.e., a careful examination should determine how the system was affected by the incident). (2) The lessons learned as a result of the incident should be included in revised security plan to prevent the incident from re-occurring. (3) A new risk analysis should be developed in light of the incident. (4) An investigation and prosecution of the individuals who caused the incident should commence, if it is deemed desirable. If an incident is based on poor policy, and unless the policy is changed, then one is doomed to repeat the past. Once a site has recovered from and incident, site policy and procedures should be reviewed to encompass changes to prevent similar incidents. Even without an incident, it would be prudent to review policies and procedures on a regular basis. Reviews are imperative due to today's changing computing environments. The whole purpose of this post mortem process is to improve all security measures to protect the site against future attacks. As a result of an incident, a site or organization should gain practical knowledge from the experience. A concrete goal of the post mortem is to develop new proactive methods. Another important facet of the aftermath may be end user and administrator education to prevent a reoccurrence of the security problem. Formatted and Indexed by: page 58 of 75 instructional media + magic, inc. RFC2196 Web Address: Site Security Handbook http://sunsite.cnlab-switch.ch/ By: Fraser, Ed. 5.6 Responsibilities 5.6.1 Not Crossing the Line It is one thing to protect one's own network, but quite another to assume that one should protect other networks. During the handling of an incident, certain system vulnerabilities of one's own systems and the systems of others become apparent. It is quite easy and may even be tempting to pursue the intruders in order to track them. Keep in mind that at a certain point it is possible to "cross the line," and, with the best of intentions, become no better than the intruder. The best rule when it comes to propriety is to not use any facility of remote sites which is not public. This clearly excludes any entry onto a system (such as a remote shell or login session) which is not expressly permitted. This may be very tempting; after a breach of security is detected, a system administrator may have the means to "follow it up," to ascertain what damage is being done to the remote site. Don't do it! Instead, attempt to reach the appropriate point of contact for the affected site. 5.6.2 Good Internet Citizenship During a security incident there are two choices one can make. First, a site can choose to watch the intruder in the hopes of catching him; or, the site can go about cleaning up after the incident and shut the intruder out of the systems. This is a decision that must be made very thoughtfully, as there may be legal liabilities if you choose to leave your site open, knowing that an intruder is using your site as a launching pad to reach out to other sites. Being a good Internet citizen means that you should try to alert other sites that may have been impacted by the intruder. These affected sites may be readily apparent after a thorough review of your log files. 5.6.3 Administrative Response to Incidents When a security incident involves a user, the site's security policy should describe what action is to be taken. The transgression should be taken seriously, but it is very important to be sure of the role the user played. Was the user naive? Could there be a mistake in attributing the security breach to the user? Applying administrative action that assumes the user intentionally caused the incident may Formatted and Indexed by: page 59 of 75 instructional media + magic, inc. RFC2196 Web Address: Site Security Handbook http://sunsite.cnlab-switch.ch/ By: Fraser, Ed. not be appropriate for a user who simply made a mistake. It may be appropriate to include sanctions more suitable for such a situation in your policies (e.g., education or reprimand of a user) in addition to more stern measures for intentional acts of intrusion and system misuse. 6. Ongoing Activities At this point in time, your site has hopefully developed a complete security policy and has developed procedures to assist in the configuration and management of your technology in support of those policies. How nice it would be if you could sit back and relax at this point and know that you were finished with the job of security. Unfortunately, that isn't possible. Your systems and networks are not a static environment, so you will need to review policies and procedures on a regular basis. There are a number of steps you can take to help you keep up with the changes around you so that you can initiate corresponding actions to address those changes. The following is a starter set and you may add others as appropriate for your site. (1) Subscribe to advisories that are issued by various security incident response teams, like those of the CERT Coordination Center, and update your systems against those threats that apply to your site's technology. (2) Monitor security patches that are produced by the vendors of your equipment, and obtain and install all that apply. (3) Actively watch the configurations of your systems to identify any changes that may have occurred, and investigate all anomalies. (4) Review all security policies and procedures annually (at a minimum). (5) Read relevant mailing lists and USENET newsgroups to keep up to date with the latest information being shared by fellow administrators. (6) Regularly check for compliance with policies and procedures. This audit should be performed by someone other than the people who define or implement the policies and procedures. 7. Tools and Locations This chapter provides a brief list of publicly available security technology which can be downloaded from the Internet. Many of the items described below will undoubtedly be surpassed or made obsolete before this document is published. Formatted and Indexed by: page 60 of 75 instructional media + magic, inc. RFC2196 Web Address: Site Security Handbook http://sunsite.cnlab-switch.ch/ By: Fraser, Ed. Some of the tools listed are applications such as end user programs (clients) and their supporting system infrastructure (servers). Others are tools that a general user will never see or need to use, but may be used by applications, or by administrators to troubleshoot security problems or to guard against intruders. A sad fact is that there are very few security conscious applications currently available. Primarily, this is caused by the need for a security infrastructure which must first be put into place for most applications to operate securely. There is considerable effort currently taking place to build this infrastructure so that applications can take advantage of secure communications. Most of the tools and applications described below can be found in one of the following archive sites: (1) CERT Coordination Center ftp://info.cert.org:/pub/tools (2) DFN-CERT ftp://ftp.cert.dfn.de/pub/tools/ (3) Computer Operations, Audit, and Security Tools (COAST) coast.cs.purdue.edu:/pub/tools It is important to note that many sites, including CERT and COAST are mirrored throughout the Internet. Be careful to use a "well known" mirror site to retrieve software, and to use verification tools (md5 checksums, etc.) to validate that software. A clever cracker might advertise security software that has intentionally been designed to provide access to data or systems. Tools COPS DES Drawbridge identd (not really a security tool) ISS Kerberos logdaemon lsof MD5 PEM PGP rpcbind/portmapper replacement SATAN sfingerd S/KEY smrsh Formatted and Indexed by: page 61 of 75 instructional media + magic, inc. RFC2196 Web Address: Site Security Handbook http://sunsite.cnlab-switch.ch/ By: Fraser, Ed. ssh swatch TCP-Wrapper tiger Tripwire* TROJAN.PL 8. Mailing Lists and Other Resources It would be impossible to list all of the mail-lists and other resources dealing with site security. However, these are some "jump- points" from which the reader can begin. All of these references are for the "INTERNET" constituency. More specific (vendor and geographical) resources can be found through these references. Mailing Lists (1) CERT(TM) Advisory Send mail to: [email protected] Message Body: subscribe cert A CERT advisory provides information on how to obtain a patch or details of a workaround for a known computer security problem. The CERT Coordination Center works with vendors to produce a workaround or a patch for a problem, and does not publish vulnerability information until a workaround or a patch is available. A CERT advisory may also be a warning to our constituency about ongoing attacks (e.g., "CA-91:18.Active.Internet.tftp.Attacks"). CERT advisories are also published on the USENET newsgroup: comp.security.announce CERT advisory archives are available via anonymous FTP from info.cert.org in the /pub/cert_advisories directory. (2) VIRUS-L List Send mail to: listserv%[email protected] Message Body: subscribe virus-L FIRSTNAME LASTNAME VIRUS-L is a moderated mailing list with a focus on computer virus issues. For more information, including a copy of the posting guidelines, see the file "virus-l.README", available by anonymous FTP from cs.ucr.edu. Formatted and Indexed by: page 62 of 75 instructional media + magic, inc. RFC2196 Web Address: Site Security Handbook http://sunsite.cnlab-switch.ch/ By: Fraser, Ed. (3) Internet Firewalls Send mail to: [email protected] Message Body: subscribe firewalls user@host The Firewalls mailing list is a discussion forum for firewall administrators and implementors. USENET newsgroups (1) comp.security.announce The comp.security.announce newsgroup is moderated and is used solely for the distribution of CERT advisories. (2) comp.security.misc The comp.security.misc is a forum for the discussion of computer security, especially as it relates to the UNIX(r) Operating System. (3) alt.security The alt.security newsgroup is also a forum for the discussion of computer security, as well as other issues such as car locks and alarm systems. (4) comp.virus The comp.virus newsgroup is a moderated newsgroup with a focus on computer virus issues. For more information, including a copy of the posting guidelines, see the file "virus-l.README", available via anonymous FTP on info.cert.org in the /pub/virus-l directory. (5) comp.risks The comp.risks newsgroup is a moderated forum on the risks to the public in computers and related systems. World-Wide Web Pages (1) http://www.first.org/ Computer Security Resource Clearinghouse. The main focus is on crisis response information; information on computer security-related threats, vulnerabilities, and solutions. At the same time, the Clearinghouse strives to be a general index to computer security information on a broad variety of subjects, including general risks, privacy, legal issues, viruses, assurance, policy, and training. Formatted and Indexed by: page 63 of 75 instructional media + magic, inc. RFC2196 Web Address: Site Security Handbook http://sunsite.cnlab-switch.ch/ By: Fraser, Ed. (2) http://www.telstra.com.au/info/security.html This Reference Index contains a list of links to information sources on Network and Computer Security. There is no implied fitness to the Tools, Techniques and Documents contained within this archive. Many if not all of these items work well, but we do not guarantee that this will be so. This information is for the education and legitimate use of computer security techniques only. (3) http://www.alw.nih.gov/Security/security.html This page features general information about computer security. Information is organized by source and each section is organized by topic. Recent modifications are noted in What's New page. (4) http://csrc.ncsl.nist.gov This archive at the National Institute of Standards and Technology's Computer Security Resource Clearinghouse page contains a number of announcements, programs, and documents related to computer security. * CERT and Tripwire are registered in the U.S. Patent and Trademark Office 9. References The following references may not be available in all countries. [Appelman, et. al., 1995] Appelman, Heller, Ehrman, White, and McAuliffe, "The Law and The Internet", USENIX 1995 Technical Conference on UNIX and Advanced Computing, New Orleans, LA, January 16-20, 1995. [ABA, 1989] American Bar Association, Section of Science and Technology, "Guide to the Prosecution of Telecommunication Fraud by the Use of Computer Crime Statutes", American Bar Association, 1989. [Aucoin, 1989] R. Aucoin, "Computer Viruses: Checklist for Recovery", Computers in Libraries, Vol. 9, No. 2, Pg. 4, February 1989. [Barrett, 1996] D. Barrett, "Bandits on the Information Superhighway", O'Reilly & Associates, Sebastopol, CA, 1996. [Bates, 1992] R. Bates, "Disaster Recovery Planning: Networks, Telecommunications and Data Communications", McGraw-Hill, 1992. [Bellovin, 1989] S. Bellovin, "Security Problems in the TCP/IP Protocol Suite", Computer Communication Review, Vol 19, 2, pp. 32-48, April 1989. Formatted and Indexed by: page 64 of 75 instructional media + magic, inc. RFC2196 Web Address: Site Security Handbook http://sunsite.cnlab-switch.ch/ By: Fraser, Ed. [Bellovin, 1990] S. Bellovin, and M. Merritt, "Limitations of the Kerberos Authentication System", Computer Communications Review, October 1990. [Bellovin, 1992] S. Bellovin, "There Be Dragon", USENIX: Proceedings of the Third Usenix Security Symposium, Baltimore, MD. September, 1992. [Bender, 1894] D. Bender, "Computer Law: Evidence and Procedure", M. Bender, New York, NY, 1978-present. [Bloombecker, 1990] B. Bloombecker, "Spectacular Computer Crimes", Dow Jones- Irwin, Homewood, IL. 1990. [Brand, 1990] R. Brand, "Coping with the Threat of Computer Security Incidents: A Primer from Prevention through Recovery", R. Brand, 8 June 1990. [Brock, 1989] J. Brock, "November 1988 Internet Computer Virus and the Vulnerability of National Telecommunications Networks to Computer Viruses", GAO/T-IMTEC-89-10, Washington, DC, 20 July 1989. [BS 7799] British Standard, BS Tech Cttee BSFD/12, Info. Sec. Mgmt, "BS 7799 : 1995 Code of Practice for Information Security Management", British Standards Institution, London, 54, Effective 15 February 1995. [Caelli, 1988] W. Caelli, Editor, "Computer Security in the Age of Information", Proceedings of the Fifth IFIP International Conference on Computer Security, IFIP/Sec '88. [Carroll, 1987] J. Carroll, "Computer Security", 2nd Edition, Butterworth Publishers, Stoneham, MA, 1987. [Cavazos and Morin, 1995] E. Cavazos and G. Morin, "Cyber-Space and The Law", MIT Press, Cambridge, MA, 1995. [CCH, 1989] Commerce Clearing House, "Guide to Computer Law", (Topical Law Reports), Chicago, IL., 1989. [Chapman, 1992] B. Chapman, "Network(In) Security Through IP Packet Filtering", USENIX: Proceedings of the Third UNIX Security Symposium, Baltimore, MD, September 1992. [Chapman and Zwicky, 1995] B. Chapman and E. Zwicky, "Building Internet Firewalls", O'Reilly and Associates, Sebastopol, CA, 1995. Formatted and Indexed by: page 65 of 75 instructional media + magic, inc. RFC2196 Web Address: Site Security Handbook http://sunsite.cnlab-switch.ch/ By: Fraser, Ed. [Cheswick, 1990] B. Cheswick, "The Design of a Secure Internet Gateway", Proceedings of the Summer Usenix Conference, Anaheim, CA, June 1990. [Cheswick1] W. Cheswick, "An Evening with Berferd In Which a Cracker is Lured, Endured, and Studied", AT&T Bell Laboratories. [Cheswick and Bellovin, 1994] W. Cheswick and S. Bellovin, "Firewalls and Internet Security: Repelling the Wily Hacker", Addison-Wesley, Reading, MA, 1994. [Conly, 1989] C. Conly, "Organizing for Computer Crime Investigation and Prosecution", U.S. Dept. of Justice, Office of Justice Programs, Under Contract Number OJP-86-C-002, National Institute of Justice, Washington, DC, July 1989. [Cooper, 1989] J. Cooper, "Computer and Communications Security: Strategies for the 1990s", McGraw-Hill, 1989. [CPSR, 1989] Computer Professionals for Social Responsibility, "CPSR Statement on the Computer Virus", CPSR, Communications of the ACM, Vol. 32, No. 6, Pg. 699, June 1989. [CSC-STD-002-85, 1985] Department of Defense, "Password Management Guideline", CSC-STD-002-85, 12 April 1985, 31 pages. [Curry, 1990] D. Curry, "Improving the Security of Your UNIX System", SRI International Report ITSTD-721-FR-90-21, April 1990. [Curry, 1992] D. Curry, "UNIX System Security: A Guide for Users and Systems Administrators", Addision-Wesley, Reading, MA, 1992. [DDN88] Defense Data Network, "BSD 4.2 and 4.3 Software Problem Resolution", DDN MGT Bulletin #43, DDN Network Information Center, 3 November 1988. [DDN89] DCA DDN Defense Communications System, "DDN Security Bulletin 03", DDN Security Coordination Center, 17 October 1989. [Denning, 1990] P. Denning, Editor, "Computers Under Attack: Intruders, Worms, and Viruses", ACM Press, 1990. [Eichin and Rochlis, 1989] M. Eichin, and J. Rochlis, "With Microscope and Tweezers: An Analysis of the Internet Virus of November 1988", Massachusetts Institute of Technology, February 1989. Formatted and Indexed by: page 66 of 75 instructional media + magic, inc. RFC2196 Web Address: Site Security Handbook http://sunsite.cnlab-switch.ch/ By: Fraser, Ed. [Eisenberg, et. al., 89] T. Eisenberg, D. Gries, J. Hartmanis, D. Holcomb, M. Lynn, and T. Santoro, "The Computer Worm", Cornell University, 6 February 1989. [Ermann, Willians, and Gutierrez, 1990] D. Ermann, M. Williams, and C. Gutierrez, Editors, "Computers, Ethics, and Society", Oxford University Press, NY, 1990. (376 pages, includes bibliographical references). [Farmer and Spafford, 1990] D. Farmer and E. Spafford, "The COPS Security Checker System", Proceedings of the Summer 1990 USENIX Conference, Anaheim, CA, Pgs. 165-170, June 1990. [Farrow, 1991] Rik Farrow, "UNIX Systems Security", Addison-Wesley, Reading, MA, 1991. [Fenwick, 1985] W. Fenwick, Chair, "Computer Litigation, 1985: Trial Tactics and Techniques", Litigation Course Handbook Series No. 280, Prepared for distribution at the Computer Litigation, 1985: Trial Tactics and Techniques Program, February-March 1985. [Fites 1989] M. Fites, P. Kratz, and A. Brebner, "Control and Security of Computer Information Systems", Computer Science Press, 1989. [Fites, Johnson, and Kratz, 1992] Fites, Johnson, and Kratz, "The Computer Virus Crisis", Van Hostrand Reinhold, 2nd edition, 1992. [Forester and Morrison, 1990] T. Forester, and P. Morrison, "Computer Ethics: Tales and Ethical Dilemmas in Computing", MIT Press, Cambridge, MA, 1990. [Foster and Morrision, 1990] T. Forester, and P. Morrison, "Computer Ethics: Tales and Ethical Dilemmas in Computing", MIT Press, Cambridge, MA, 1990. (192 pages including index.) [GAO/IMTEX-89-57, 1989] U.S. General Accounting Office, "Computer Security - Virus Highlights Need for Improved Internet Management", United States General Accounting Office, Washington, DC, 1989. [Garfinkel and Spafford, 1991] S. Garfinkel, and E. Spafford, "Practical Unix Security", O'Reilly & Associates, ISBN 0-937175-72-2, May 1991. [Garfinkel, 1995] S. Garfinkel, "PGP:Pretty Good Privacy", O'Reilly & Associates, Sebastopol, CA, 1996. Formatted and Indexed by: page 67 of 75 instructional media + magic, inc. RFC2196 Web Address: Site Security Handbook http://sunsite.cnlab-switch.ch/ By: Fraser, Ed. [Garfinkel and Spafford, 1996] S. Garfinkel and E. Spafford, "Practical UNIX and Internet Security", O'Reilly & Associates, Sebastopol, CA, 1996. [Gemignani, 1989] M. Gemignani, "Viruses and Criminal Law", Communications of the ACM, Vol. 32, No. 6, Pgs. 669-671, June 1989. [Goodell, 1996] J. Goodell, "The Cyberthief and the Samurai: The True Story of Kevin Mitnick-And The Man Who Hunted Him Down", Dell Publishing, 1996. [Gould, 1989] C. Gould, Editor, "The Information Web: Ethical and Social Implications of Computer Networking", Westview Press, Boulder, CO, 1989. [Greenia, 1989] M. Greenia, "Computer Security Information Sourcebook", Lexikon Services, Sacramento, CA, 1989. [Hafner and Markoff, 1991] K. Hafner and J. Markoff, "Cyberpunk: Outlaws and Hackers on the Computer Frontier", Touchstone, Simon & Schuster, 1991. [Hess, Safford, and Pooch] D. Hess, D. Safford, and U. Pooch, "A Unix Network Protocol Security Study: Network Information Service", Texas A&M University. [Hoffman, 1990] L. Hoffman, "Rogue Programs: Viruses, Worms, and Trojan Horses", Van Nostrand Reinhold, NY, 1990. (384 pages, includes bibliographical references and index.) [Howard, 1995] G. Howard, "Introduction to Internet Security: From Basics to Beyond", Prima Publishing, Rocklin, CA, 1995. [Huband and Shelton, 1986] F. Huband, and R. Shelton, Editors, "Protection of Computer Systems and Software: New Approaches for Combating Theft of Software and Unauthorized Intrusion", Papers presented at a workshop sponsored by the National Science Foundation, 1986. [Hughes, 1995] L. Hughes Jr., "Actually Useful Internet Security Techniques", New Riders Publishing, Indianapolis, IN, 1995. [IAB-RFC1087, 1989] Internet Activities Board, "Ethics and the Internet", RFC 1087, IAB, January 1989. Also appears in the Communications of the ACM, Vol. 32, No. 6, Pg. 710, June 1989. Formatted and Indexed by: page 68 of 75 instructional media + magic, inc. RFC2196 Web Address: Site Security Handbook http://sunsite.cnlab-switch.ch/ By: Fraser, Ed. [Icove, Seger, and VonStorch, 1995] D. Icove, K. Seger, and W. VonStorch, "Computer Crime: A Crimefighter's Handbook", O'Reilly & Associates, Sebastopol, CA, 1995. [IVPC, 1996] IVPC, "International Virus Prevention Conference '96 Proceedings", NCSA, 1996. [Johnson and Podesta] D. Johnson, and J. Podesta, "Formulating A Company Policy on Access to and Use and Disclosure of Electronic Mail on Company Computer Systems". [Kane, 1994] P. Kane, "PC Security and Virus Protection Handbook: The Ongoing War Against Information Sabotage", M&T Books, 1994. [Kaufman, Perlman, and Speciner, 1995] C. Kaufman, R. Perlman, and M. Speciner, "Network Security: PRIVATE Communication in a PUBLIC World", Prentice Hall, Englewood Cliffs, NJ, 1995. [Kent, 1990] S. Kent, "E-Mail Privacy for the Internet: New Software and Strict Registration Procedures will be Implemented this Year", Business Communications Review, Vol. 20, No. 1, Pg. 55, 1 January 1990. [Levy, 1984] S. Levy, "Hacker: Heroes of the Computer Revolution", Delta, 1984. [Lewis, 1996] S. Lewis, "Disaster Recovery Yellow Pages", The Systems Audit Group, 1996. [Littleman, 1996] J. Littleman, "The Fugitive Game: Online with Kevin Mitnick", Little, Brown, Boston, MA., 1996. [Lu and Sundareshan, 1989] W. Lu and M. Sundareshan, "Secure Communication in Internet Environments: A Hierarchical Key Management Scheme for End-to-End Encryption", IEEE Transactions on Communications, Vol. 37, No. 10, Pg. 1014, 1 October 1989. [Lu and Sundareshan, 1990] W. Lu and M. Sundareshan, "A Model for Multilevel Security in Computer Networks", IEEE Transactions on Software Engineering, Vol. 16, No. 6, Page 647, 1 June 1990. [Martin and Schinzinger, 1989] M. Martin, and R. Schinzinger, "Ethics in Engineering", McGraw Hill, 2nd Edition, 1989. [Merkle] R. Merkle, "A Fast Software One Way Hash Function", Journal of Cryptology, Vol. 3, No. 1. Formatted and Indexed by: page 69 of 75 instructional media + magic, inc. RFC2196 Web Address: Site Security Handbook http://sunsite.cnlab-switch.ch/ By: Fraser, Ed. [McEwen, 1989] J. McEwen, "Dedicated Computer Crime Units", Report Contributors: D. Fester and H. Nugent, Prepared for the National Institute of Justice, U.S. Department of Justice, by Institute for Law and Justice, Inc., under contract number OJP-85-C-006, Washington, DC, 1989. [MIT, 1989] Massachusetts Institute of Technology, "Teaching Students About Responsible Use of Computers", MIT, 1985-1986. Also reprinted in the Communications of the ACM, Vol. 32, No. 6, Pg. 704, Athena Project, MIT, June 1989. [Mogel, 1989] Mogul, J., "Simple and Flexible Datagram Access Controls for UNIX-based Gateways", Digital Western Research Laboratory Research Report 89/4, March 1989. [Muffett, 1992] A. Muffett, "Crack Version 4.1: A Sensible Password Checker for Unix" [NCSA1, 1995] NCSA, "NCSA Firewall Policy Guide", 1995. [NCSA2, 1995] NCSA, "NCSA's Corporate Computer Virus Prevention Policy Model", NCSA, 1995. [NCSA, 1996] NCSA, "Firewalls & Internet Security Conference '96 Proceedings", 1996. [NCSC-89-660-P, 1990] National Computer Security Center, "Guidelines for Formal Verification Systems", Shipping list no.: 89-660-P, The Center, Fort George G. Meade, MD, 1 April 1990. [NCSC-89-254-P, 1988] National Computer Security Center, "Glossary of Computer Security Terms", Shipping list no.: 89-254-P, The Center, Fort George G. Meade, MD, 21 October 1988. [NCSC-C1-001-89, 1989] Tinto, M., "Computer Viruses: Prevention, Detection, and Treatment", National Computer Security Center C1 Technical Report C1-001-89, June 1989. [NCSC Conference, 1989] National Computer Security Conference, "12th National Computer Security Conference: Baltimore Convention Center, Baltimore, MD, 10-13 October, 1989: Information Systems Security, Solutions for Today - Concepts for Tomorrow", National Institute of Standards and National Computer Security Center, 1989. [NCSC-CSC-STD-003-85, 1985] National Computer Security Center, "Guidance for Applying the Department of Defense Trusted Computer System Evaluation Criteria in Specific Environments", CSC-STD-003-85, NCSC, 25 June 1985. Formatted and Indexed by: page 70 of 75 instructional media + magic, inc. RFC2196 Web Address: Site Security Handbook http://sunsite.cnlab-switch.ch/ By: Fraser, Ed. [NCSC-STD-004-85, 1985] National Computer Security Center, "Technical Rationale Behind CSC-STD-003-85: Computer Security Requirements", CSC-STD-004-85, NCSC, 25 June 1985. [NCSC-STD-005-85, 1985] National Computer Security Center, "Magnetic Remanence Security Guideline", CSC-STD-005-85, NCSC, 15 November 1985. [NCSC-TCSEC, 1985] National Computer Security Center, "Trusted Computer System Evaluation Criteria", DoD 5200.28-STD, CSC-STD-001- 83, NCSC, December 1985. [NCSC-TG-003, 1987] NCSC, "A Guide to Understanding DISCRETIONARY ACCESS CONTROL in Trusted Systems", NCSC-TG-003, Version-1, 30 September 1987, 29 pages. [NCSC-TG-001, 1988] NCSC, "A Guide to Understanding AUDIT in Trusted Systems", NCSC-TG-001, Version-2, 1 June 1988, 25 pages. [NCSC-TG-004, 1988] National Computer Security Center, "Glossary of Computer Security Terms", NCSC-TG-004, NCSC, 21 October 1988. [NCSC-TG-005, 1987] National Computer Security Center, "Trusted Network Interpretation", NCSC-TG-005, NCSC, 31 July 1987. [NCSC-TG-006, 1988] NCSC, "A Guide to Understanding CONFIGURATION MANAGEMENT in Trusted Systems", NCSC-TG-006, Version-1, 28 March 1988, 31 pages. [NCSC-TRUSIX, 1990] National Computer Security Center, "Trusted UNIX Working Group (TRUSIX) rationale for selecting access control list features for the UNIX system", Shipping list no.: 90-076-P, The Center, Fort George G. Meade, MD, 1990. [NRC, 1991] National Research Council, "Computers at Risk: Safe Computing in the Information Age", National Academy Press, 1991. [Nemeth, et. al, 1995] E. Nemeth, G. Snyder, S. Seebass, and T. Hein, "UNIX Systems Administration Handbook", Prentice Hall PTR, Englewood Cliffs, NJ, 2nd ed. 1995. [NIST, 1989] National Institute of Standards and Technology, "Computer Viruses and Related Threats: A Management Guide", NIST Special Publication 500-166, August 1989. [NSA] National Security Agency, "Information Systems Security Products and Services Catalog", NSA, Quarterly Publication. Formatted and Indexed by: page 71 of 75 instructional media + magic, inc. RFC2196 Web Address: Site Security Handbook http://sunsite.cnlab-switch.ch/ By: Fraser, Ed. [NSF, 1988] National Science Foundation, "NSF Poses Code of Networking Ethics", Communications of the ACM, Vol. 32, No. 6, Pg. 688, June 1989. Also appears in the minutes of the regular meeting of the Division Advisory Panel for Networking and Communications Research and Infrastructure, Dave Farber, Chair, November 29-30, 1988. [NTISSAM, 1987] NTISS, "Advisory Memorandum on Office Automation Security Guideline", NTISSAM COMPUSEC/1-87, 16 January 1987, 58 pages. [OTA-CIT-310, 1987] United States Congress, Office of Technology Assessment, "Defending Secrets, Sharing Data: New Locks and Keys for Electronic Information", OTA-CIT-310, October 1987. [OTA-TCT-606] Congress of the United States, Office of Technology Assessment, "Information Security and Privacy in Network Environments", OTA-TCT-606, September 1994. [Palmer and Potter, 1989] I. Palmer, and G. Potter, "Computer Security Risk Management", Van Nostrand Reinhold, NY, 1989. [Parker, 1989] D. Parker, "Computer Crime: Criminal Justice Resource Manual", U.S. Dept. of Justice, National Institute of Justice, Office of Justice Programs, Under Contract Number OJP-86-C-002, Washington, D.C., August 1989. [Parker, Swope, and Baker, 1990] D. Parker, S. Swope, and B. Baker, "Ethical Conflicts: Information and Computer Science, Technology and Business", QED Information Sciences, Inc., Wellesley, MA. (245 pages). [Pfleeger, 1989] C. Pfleeger, "Security in Computing", Prentice-Hall, Englewood Cliffs, NJ, 1989. [Quarterman, 1990] J. Quarterman, J., "The Matrix: Computer Networks and Conferencing Systems Worldwide", Digital Press, Bedford, MA, 1990. [Ranum1, 1992] M. Ranum, "An Internet Firewall", Proceedings of World Conference on Systems Management and Security, 1992. [Ranum2, 1992] M. Ranum, "A Network Firewall", Digital Equipment Corporation Washington Open Systems Resource Center, June 12, 1992. [Ranum, 1993] M. Ranum, "Thinking About Firewalls", 1993. Formatted and Indexed by: page 72 of 75 instructional media + magic, inc. RFC2196 Web Address: Site Security Handbook http://sunsite.cnlab-switch.ch/ By: Fraser, Ed. [Ranum and Avolio, 1994] M. Ranum and F. Avolio, "A Toolkit and Methods for Internet Firewalls", Trustest Information Systems, 1994. [Reinhardt, 1992] R. Reinhardt, "An Architectural Overview of UNIX Network Security" [Reinhardt, 1993] R. Reinhardt, "An Architectural Overview of UNIX Network Security", ARINC Research Corporation, February 18, 1993. [Reynolds-RFC1135, 1989] The Helminthiasis of the Internet, RFC 1135, USC/Information Sciences Institute, Marina del Rey, CA, December 1989. [Russell and Gangemi, 1991] D. Russell and G. Gangemi, "Computer Security Basics" O'Reilly & Associates, Sebastopol, CA, 1991. [Schneier 1996] B. Schneier, "Applied Cryptography: Protocols, Algorithms, and Source Code in C", John Wiley & Sons, New York, second edition, 1996. [Seeley, 1989] D. Seeley, "A Tour of the Worm", Proceedings of 1989 Winter USENIX Conference, Usenix Association, San Diego, CA, February 1989. [Shaw, 1986] E. Shaw Jr., "Computer Fraud and Abuse Act of 1986", Congressional Record (3 June 1986), Washington, D.C., 3 June 1986. [Shimomura, 1996] T. Shimomura with J. Markoff, "Takedown:The Pursuit and Capture of Kevin Mitnick, America's Most Wanted Computer Outlaw- by the Man Who Did It", Hyperion, 1996. [Shirey, 1990] R. Shirey, "Defense Data Network Security Architecture", Computer Communication Review, Vol. 20, No. 2, Page 66, 1 April 1990. [Slatalla and Quittner, 1995] M. Slatalla and J. Quittner, "Masters of Deception: The Gang that Ruled Cyberspace", Harper Collins Publishers, 1995. [Smith, 1989] M. Smith, "Commonsense Computer Security: Your Practical Guide to Preventing Accidental and Deliberate Electronic Data Loss", McGraw-Hill, New York, NY, 1989. [Smith, 1995] D. Smith, "Forming an Incident Response Team", Sixth Annual Computer Security Incident Handling Workshop, Boston, MA, July 25-29, 1995. Formatted and Indexed by: page 73 of 75 instructional media + magic, inc. RFC2196 Web Address: Site Security Handbook http://sunsite.cnlab-switch.ch/ By: Fraser, Ed. [Spafford, 1988] E. Spafford, "The Internet Worm Program: An Analysis", Computer Communication Review, Vol. 19, No. 1, ACM SIGCOM, January 1989. Also issued as Purdue CS Technical Report CSD-TR-823, 28 November 1988. [Spafford, 1989] G. Spafford, "An Analysis of the Internet Worm", Proceedings of the European Software Engineering Conference 1989, Warwick England, September 1989. Proceedings published by Springer- Verlag as: Lecture Notes in Computer Science #387. Also issued as Purdue Technical Report #CSD-TR-933. [Spafford, Keaphy, and Ferbrache, 1989] E. Spafford, K. Heaphy, and D. Ferbrache, "Computer Viruses: Dealing with Electronic Vandalism and Programmed Threats", ADAPSO, 1989. (109 pages.) [Stallings1, 1995] W. Stallings, "Internet Security Handbook", IDG Books, Foster City CA, 1995. [Stallings2, 1995] W. Stallings, "Network and InterNetwork Security", Prentice Hall, , 1995. [Stallings3, 1995] W. Stallings, "Protect Your Privacy: A Guide for PGP Users" PTR Prentice Hall, 1995. [Stoll, 1988] C. Stoll, "Stalking the Wily Hacker", Communications of the ACM, Vol. 31, No. 5, Pgs. 484-497, ACM, New York, NY, May 1988. [Stoll, 1989] C. Stoll, "The Cuckoo's Egg", ISBN 00385-24946-2, Doubleday, 1989. [Treese and Wolman, 1993] G. Treese and A. Wolman, "X Through the Firewall, and Other Applications Relays", Digital Equipment Corporation, Cambridge Research Laboratory, CRL 93/10, May 3, 1993. [Trible, 1986] P. Trible, "The Computer Fraud and Abuse Act of 1986", U.S. Senate Committee on the Judiciary, 1986. [Venema] W. Venema, "TCP WRAPPER: Network monitoring, access control, and booby traps", Mathematics and Computing Science, Eindhoven University of Technology, The Netherlands. [USENIX, 1988] USENIX, "USENIX Proceedings: UNIX Security Workshop", Portland, OR, August 29-30, 1988. [USENIX, 1990] USENIX, "USENIX Proceedings: UNIX Security II Workshop", Portland, OR, August 27-28, 1990. Formatted and Indexed by: page 74 of 75 instructional media + magic, inc. RFC2196 Web Address: Site Security Handbook http://sunsite.cnlab-switch.ch/ By: Fraser, Ed. [USENIX, 1992] USENIX, "USENIX Symposium Proceedings: UNIX Security III", Baltimore, MD, September 14-16, 1992. [USENIX, 1993] USENIX, "USENIX Symposium Proceedings: UNIX Security IV", Santa Clara, CA, October 4-6, 1993. [USENIX, 1995] USENIX, "The Fifth USENIX UNIX Security Symposium", Salt Lake City, UT, June 5-7, 1995. [Wood, et.al., 1987] C. Wood, W. Banks, S. Guarro, A. Garcia, V. Hampel, and H. Sartorio, "Computer Security: A Comprehensive Controls Checklist", John Wiley and Sons, Interscience Publication, 1987. [Wrobel, 1993] L. Wrobel, "Writing Disaster Recovery Plans for Telecommunications Networks and LANS", Artech House, 1993. [Vallabhaneni, 1989] S. Vallabhaneni, "Auditing Computer Security: A Manual with Case Studies", Wiley, New York, NY, 1989. Security Considerations This entire document discusses security issues. Editor Information Barbara Y. Fraser Software Engineering Institute Carnegie Mellon University 5000 Forbes Avenue Pittsburgh, PA 15213 Phone: (412) 268-5010 Fax: (412) 268-6989 EMail: [email protected] _ Formatted and Indexed by: page 75 of 75 instructional media + magic, inc.