Zstd Window Size: Window Sizing for Zstandard Content Encoding
A hard 8 MB window limit so zstd-compressed web content decodes everywhere.
Jaju, N., Ed. and W. F. Handte, Ed., "Window Sizing for Zstandard Content Encoding", RFC 9659, DOI 10.17487/RFC9659, September 2024, <https://www.rfc-editor.org/info/rfc9659>.
@misc{rfc9659,
series = {Request for Comments},
number = 9659,
howpublished = {RFC 9659},
publisher = {RFC Editor},
doi = {10.17487/RFC9659},
url = {https://www.rfc-editor.org/info/rfc9659},
author = {N. Jaju and W. F. Handte},
title = {{Window Sizing for Zstandard Content Encoding}},
pagetotal = 4,
year = 2024,
month = sep,
} RFC text
Contents
Internet Engineering Task Force (IETF) N. Jaju, Ed.
Request for Comments: 9659 Google
Updates: 8878 W. F. Handte, Ed.
Category: Informational Meta Platforms, Inc.
ISSN: 2070-1721 September 2024
Window Sizing for Zstandard Content Encoding
Abstract
Deployments of Zstandard, or "zstd", can use different window sizes
to limit memory usage during compression and decompression. Some
browsers and user agents limit window sizes to mitigate memory usage
concerns, thereby causing interoperability issues. This document
updates the window size limit in RFC 8878 from a recommendation to a
requirement in HTTP contexts.
Status of This Memo
This document is not an Internet Standards Track specification; it is
published for informational purposes.
This document is a product of the Internet Engineering Task Force
(IETF). It represents the consensus of the IETF community. It has
received public review and has been approved for publication by the
Internet Engineering Steering Group (IESG). Not all documents
approved by the IESG are candidates for any level of Internet
Standard; see Section 2 of RFC 7841.
Information about the current status of this document, any errata,
and how to provide feedback on it may be obtained at
https://www.rfc-editor.org/info/rfc9659.
Copyright Notice
Copyright (c) 2024 IETF Trust and the persons identified as the
document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents
(https://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with respect
to this document. Code Components extracted from this document must
include Revised BSD License text as described in Section 4.e of the
Trust Legal Provisions and are provided without warranty as described
in the Revised BSD License.
Table of Contents
1. Introduction
2. Conventions and Definitions
3. Window Size
4. Security Considerations
5. IANA Considerations
5.1. Content Encoding
6. Normative References
Acknowledgments
Authors' Addresses
1. Introduction
Zstandard, or "zstd", specified in [RFC8878], is a lossless data
compression mechanism similar to gzip. When used with HTTP, the
"zstd" content coding token signals to the decoder that the content
is Zstandard-compressed.
An important property of Zstandard-compressed content is its
Window_Size ([RFC8878], Section 3.1.1.1.2), which describes the
maximum distance for back-references and therefore how much of the
content must be kept in memory during decompression.
The minimum Window_Size is 1 KB. The maximum Window_Size is (1<<41)
+ 7*(1<<38) bytes, where "<<" denotes a bitwise left shift, which is
3.75 TB. Larger Window_Size values tend to improve the compression
ratio but at the cost of increased memory usage.
To protect against unreasonable memory usage, some browsers and user
agents limit the maximum Window_Size they will handle. This causes
failures to decode responses when the content is compressed with a
larger Window_Size than the recipient allows, leading to decreased
interoperability.
[RFC8878], Section 3.1.1.1.2 recommends that decoders support a
Window_Size of up to 8 MB, and that encoders not generate frames
using a Window_Size larger than 8 MB. However, it imposes no
requirements.
This document updates [RFC8878] to enforce Window_Size limits on the
encoder and decoder for the "zstd" HTTP content coding.
2. Conventions and Definitions
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and
"OPTIONAL" in this document are to be interpreted as described in
BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all
capitals, as shown here.
3. Window Size
To ensure interoperability, when using the "zstd" content coding,
decoders MUST support a Window_Size of up to and including 8 MB, and
encoders MUST NOT generate frames requiring a Window_Size larger than
8 MB (see Section 5.1).
4. Security Considerations
This document introduces no new security considerations beyond those
discussed in [RFC8878].
Note that decoders still need to take into account that they can
receive oversized frames that do not follow the window size limit
specified in this document and fail decoding when such invalid frames
are received.
5. IANA Considerations
5.1. Content Encoding
This document updates the following entry in the "HTTP Content Coding
Registry" in the "Hypertext Transfer Protocol (HTTP) Parameters"
registry group (https://www.iana.org/assignments/http-parameters):
Name: zstd
Description: A stream of bytes compressed using the Zstandard
protocol with a Window_Size of not more than 8 MB.
Reference: This document and [RFC8878]
6. Normative References
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119,
DOI 10.17487/RFC2119, March 1997,
<https://www.rfc-editor.org/info/rfc2119>.
[RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC
2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174,
May 2017, <https://www.rfc-editor.org/info/rfc8174>.
[RFC8878] Collet, Y. and M. Kucherawy, Ed., "Zstandard Compression
and the 'application/zstd' Media Type", RFC 8878,
DOI 10.17487/RFC8878, February 2021,
<https://www.rfc-editor.org/info/rfc8878>.
Acknowledgments
Zstandard was developed by Yann Collet.
The authors would like to thank Yann Collet, Klaus Post, Adam Rice,
and members of the Web Performance Working Group in the W3C for
collaborating on the window size issue and helping to formulate a
solution.
Authors' Addresses
Nidhi Jaju (editor)
Google
Shibuya Stream, 3 Chome-21-3 Shibuya, Shibuya City, Tokyo
150-0002
Japan
Email: nidhijaju@google.com
W. Felix P. Handte (editor)
Meta Platforms, Inc.
380 W 33rd St
New York, NY 10001
United States of America
Email: felixh@meta.com
AI ELI5
On this page
§1The Window Size Problem
Zstandard ("zstd"), specified in RFC 8878, is a lossless compression mechanism similar to gzip. In HTTP, the zstd content coding token tells the decoder that the content is Zstandard-compressed.
Compressed data carries a Window_Size: the maximum distance for back-references, and so how much of the content must stay in memory while decompressing. The minimum is 1 KB and the maximum is (1<<41) + 7*(1<<38) bytes, which is 3.75 TB. Bigger windows tend to compress better but use more memory.
To protect against unreasonable memory use, some browsers and user agents limit the Window_Size they will handle. Content compressed with a larger window then fails to decode, which hurts interoperability.
A bigger workspace squishes files smaller but needs more room. Some programs only have a small desk, so they cannot open files that were squished on a giant one.
§3The 8 MB Requirement
RFC 8878, Section 3.1.1.1.2, recommends that decoders support a Window_Size of up to 8 MB and that encoders not generate frames with a larger one, but it imposes no requirements. This document turns that into a rule for the zstd HTTP content coding.
Decoders MUST support a Window_Size of up to and including 8 MB. Encoders MUST NOT generate frames requiring a Window_Size larger than 8 MB. With both sides bound, a sender can rely on any receiver being able to decode its output.
Everyone agrees on one desk size: 8 MB. Senders must not use a bigger one, and receivers must be able to handle at least that much.
§4Security and Registry Notes
The document introduces no new security considerations beyond those in RFC 8878. It does warn that decoders still need to expect oversized frames that break the limit, and must fail decoding when they receive such invalid frames.
On the IANA side, the entry for zstd in the HTTP Content Coding Registry is updated. Its description becomes a stream of bytes compressed using the Zstandard protocol with a Window_Size of not more than 8 MB, and its reference is this document and RFC 8878.
A careful receiver still checks the size, and refuses any file that breaks the rule. The official list of names is updated to say the limit too.