ISO 28500:2017
Information and documentation — WARC file format
ISO 28500:2017 specifies the WARC file format:
- to store both the payload content and control information from mainstream Internet application layer protocols, such as the HTTP, DNS, and FTP;
- to store arbitrary metadata linked to other stored data (e.g. subject classifier, discovered language, encoding);
- to support data compression and maintain data record integrity;
- to store all control information from the harvesting protocol (e.g. request headers), not just response information;
- to store the results of data transformations linked to other stored data;
- to store a duplicate detection event linked to other stored data (to reduce storage in the presence of identical or substantially similar resources);
- to be extended without disruption to existing functionality;
- to support handling of overly long records by truncation or segmentation, where desired.
| Get this standard | Prices exclude GST | |
|---|---|---|
| PDF ( Single user document) |
$256.00 NZD
|
|
| HardCopy |
$303.00 NZD
|
|
| Networkable PDF |
Price varies
|
Foreword
Introduction
1 Scope
2 Normative references
3 Terms, definitions and abbreviated terms
4 File and record model
5 Named fields
5.1 General
5.2 WARC-Record-ID (mandatory)
5.3 Content-Length (mandatory)
5.4 WARC-Date (mandatory)
5.5 WARC-Type (mandatory)
5.6 Content-Type
5.7 WARC-Concurrent-To
5.8 WARC-Block-Digest
5.9 WARC-Payload-Digest
5.10 WARC-IP-Address
5.11 WARC-Refers-To
5.12 WARC-Refers-To-Target-URI
5.13 WARC-Refers-To-Date
5.14 WARC-Target-URI
5.15 WARC-Truncated
5.16 WARC-Warcinfo-ID
5.17 WARC-Filename
5.18 WARC-Profile
5.19 WARC-Identified-Payload-Type
5.20 WARC-Segment-Number
5.21 WARC-Segment-Origin-ID
5.22 WARC-Segment-Total-Length
6 WARC record types
6.1 General
6.2 ‘warcinfo’
6.3 ‘response’
6.3.1 General
6.3.2 ‘http’ and ‘https’ schemes
6.3.3 Other URI schemes
6.4 ‘resource’
6.4.1 General
6.4.2 ‘http’ and ‘https’ schemes
6.4.3 ‘ftp’ scheme
6.4.4 ‘dns’ scheme
6.4.5 Other URI schemes
6.5 ‘request’
6.5.1 General
6.5.2 ‘http’ and ‘https’ schemes
6.5.3 Other URI schemes
6.6 ‘metadata’
6.7 ‘revisit’
6.7.1 General
6.7.2 Profile: Identical Payload Digest
6.7.3 Profile: Server Not Modified
6.7.4 Other profiles
6.8 ‘conversion’
6.9 ‘continuation’
7 Record segmentation
8 WARC file name, size and compression
Annex A (informative) Use cases for writing WARC records
Annex B (informative) Examples of WARC records
Annex C (informative) WARC file size and name recommendations
Annex D (informative) Compression recommendations
Bibliography
Previous versions
Keep me up-to-date
Sign up to receive updates when there are changes to this standard
Related Information
Similar Standards
-
23/30382233DC
BS EN 9300-00. Aerospace series. LOTAR -LOng Term Archiving and Retrieval of digital technical product documentation such as 3D, CAD and PDM data, Part 007. Terms and references
-
AS/NZS 23950:1999
Information and documentation - Information retrieval (Z30.50) - Application service definition and protocol specification
-
AS/NZS 3968.0:1994
Information technology - Database languages - SQL - Definition of data structures and basic operations
-
AS/NZS 3968.0:1994 A1
Information technology - Database languages - SQL - Definition of data structures and basic operations: Amendment 1
Foreword
Introduction
1 Scope
2 Normative references
3 Terms, definitions and abbreviated terms
4 File and record model
5 Named fields
5.1 General
5.2 WARC-Record-ID (mandatory)
5.3 Content-Length (mandatory)
5.4 WARC-Date (mandatory)
5.5 WARC-Type (mandatory)
5.6 Content-Type
5.7 WARC-Concurrent-To
5.8 WARC-Block-Digest
5.9 WARC-Payload-Digest
5.10 WARC-IP-Address
5.11 WARC-Refers-To
5.12 WARC-Refers-To-Target-URI
5.13 WARC-Refers-To-Date
5.14 WARC-Target-URI
5.15 WARC-Truncated
5.16 WARC-Warcinfo-ID
5.17 WARC-Filename
5.18 WARC-Profile
5.19 WARC-Identified-Payload-Type
5.20 WARC-Segment-Number
5.21 WARC-Segment-Origin-ID
5.22 WARC-Segment-Total-Length
6 WARC record types
6.1 General
6.2 ‘warcinfo’
6.3 ‘response’
6.3.1 General
6.3.2 ‘http’ and ‘https’ schemes
6.3.3 Other URI schemes
6.4 ‘resource’
6.4.1 General
6.4.2 ‘http’ and ‘https’ schemes
6.4.3 ‘ftp’ scheme
6.4.4 ‘dns’ scheme
6.4.5 Other URI schemes
6.5 ‘request’
6.5.1 General
6.5.2 ‘http’ and ‘https’ schemes
6.5.3 Other URI schemes
6.6 ‘metadata’
6.7 ‘revisit’
6.7.1 General
6.7.2 Profile: Identical Payload Digest
6.7.3 Profile: Server Not Modified
6.7.4 Other profiles
6.8 ‘conversion’
6.9 ‘continuation’
7 Record segmentation
8 WARC file name, size and compression
Annex A (informative) Use cases for writing WARC records
Annex B (informative) Examples of WARC records
Annex C (informative) WARC file size and name recommendations
Annex D (informative) Compression recommendations
Bibliography