Skip to main content

Data Compression

Use of Gzip Compression by the API​

Some large API responses are automatically compressed with gzip to reduce size and speed up downloads.

Accepting Gzipped Responses​

Most HTTP clients and browsers support gzip by default. To ensure gzip is accepted, add the following header to your request:

Accept-Encoding: gzip

If the response is large, the server will send it gzipped.

Reading Gzipped Responses​

  • Browsers: Handled automatically.

  • curl: Use --compressed:

    curl --compressed https://api.example.com/data

  • Code: If not automatic, check Content-Encoding: gzip (or X-Content-Encoding: gzip) and decompress with your language's gzip/zlib library.

Small responses may not be gzipped, this is normal.

Sending Gzipped Request Data​

If your request body is large (greater than 6MB), you should also compress it before sending. To do this:

  • Compress your request body using gzip.

  • Add the following header to indicate compression:

    X-Content-Encoding: gzip

  • Our server endpoints automatically decompress gzipped request bodies.

Example with curl:

echo '{"large":"payload"}' | gzip | \
curl \
  -X POST https://api.example.com/upload \
  -H 'X-Content-Encoding: gzip' \
  -H 'Content-Type: application/json' \
  --data-binary @-

This helps reduce upload size for large JSON or binary payloads.

You can read more about Compression in HTTP & gzip compression in the MDN Web Docs.

Appendix: Content-Encoding versus X-Content-Encoding​

It is very important to use the X-Content-Encoding header and not the standard Content-Encoding, this is because AWS API Gateway will automatically decompress gzip data based on the Content-Encoding header. We want to decompress & compress your data using our own middleware instead of API Gateway, hence the "custom" header needs to be used.

Appendix: Gzip Compression Analysis​

Measured on the 10 largest diverse production documents per collection (March 2026). Compression ratio = raw size / gzip size.

ClustyRequest​

PicksRaw JSONGzipCompression Ratio
34,4279.9 MB420.8 KB24x
33,4819.6 MB413.3 KB24x
31,3916.3 MB580.7 KB11x
30,2286.0 MB553.6 KB11x
29,8055.9 MB545.8 KB11x
27,8104.6 MB441.3 KB11x
26,1124.2 MB423.4 KB10x
24,7977.1 MB300.6 KB24x
23,6413.8 MB414.0 KB9x
23,1706.7 MB284.1 KB24x

Average: 15x compression

RoutyRequest​

PicksRaw JSONGzipCompression Ratio
3,717689.1 KB22.7 KB30x
2,406502.4 KB17.4 KB29x
1,515281.9 KB24.3 KB12x
1,059141.1 KB11.9 KB12x
791105.4 KB10.0 KB11x
68491.5 KB8.9 KB10x
66989.3 KB8.9 KB10x
66889.1 KB8.9 KB10x
610139.8 KB10.0 KB14x
60981.1 KB7.6 KB11x

Average: 17x compression

Takeaway​

Gzip achieves 10-30x compression on request payloads, with an overall average of ~15x. The repetitive structure of picks arrays compresses extremely well.