misa77 (0.2.0) misa77 is an LZ-based codec that targets the write-once, read-many niche. In particular, it aims to satisfy the following criteria:

Extremely high decompression throughput (single-threaded). Modest compression ratios (it has no entropy backend, so one can obviously not compare it to something like zstd, but LZ4 at high effort levels is a good reference point). Constant memory use, regardless of input size (<= 5 MB across all compression modes, and 0 MB for decompression).

Slow compression is the obvious tradeoff that one makes to achieve the above. In addition, misa77 has a somewhat synergizing tendency to decompress highly compressed files faster, leading to the following results:

It offers particularly high decompression throughput on highly compressible files. Even for moderately compressible files, spending more effort during compression to get a more compressed result leads to better decompression throughput (alongside the natural advantage of better ratios).

This makes high-effort compression particularly attractive for misa77, and inspires some experimental compression modes (refer to src/experimental/) that aim to spend more effort at compression time to produce a compressed stream that is friendlier to the microarchitectures of most CPUs when decompressing said streams. misa77 also has two compression effort levels as of v0.2.0:

level 0: offers better decode throughput, slightly worse ratio, similar encode throughput level 1 (default): offers slightly worse decode throughput, better ratio, similar encode throughput

Benchmarks Detailed results are listed ahead, but here's a terse summary:

misa77 lies on the pareto frontier for decompression throughput vs compression ratio on most shapes of data. It very frequently decompresses faster even when competitors have a significantly worse ratio. It is quite slow at compression.

Cross-Platform results for v0.1.0 (across Intel x86-64, AMD x86-64, ARM64) can be found here. Note that these are likely outdated at the time of you reading this. Some detailed results for Intel x86-64 follow. Setup:

CPU: Intel(R) Core(TM) i7-14650HX (@2.2 GHz) (Intel Turbo disabled). Single threaded, pinned to a single performance core. CPU governor set to performance. The benchmarking harness is a public fork of lzbench, and can be accessed here.

Results for the Silesia Corpus. misa77 rows first, then competitors are sorted by decompression speed.

Compressor name Compression Decompress. Ratio Filename

misa77 0.2.0 -0 54.5 MB/s 5219 MB/s 42.64 silesia.tar

misa77 0.2.0 -1 51.2 MB/s 4274 MB/s 39.65 silesia.tar

zxc 0.12.0 -3 115 MB/s 2841 MB/s 45.46 silesia.tar

zxc 0.12.0 -4 80.8 MB/s 2726 MB/s 42.63 silesia.tar

lzsse8fast 2019-04-18 183 MB/s 2661 MB/s 44.80 silesia.tar

zxc 0.12.0 -5 48.6 MB/s 2599 MB/s 40.25 silesia.tar

lz4hc 1.10.0 -12 7.31 MB/s 2531 MB/s 36.45 silesia.tar

lzsse4fast 2019-04-18 186 MB/s 2522 MB/s 45.26 silesia.tar

lz4 1.10.0 371 MB/s 2505 MB/s 47.59 silesia.tar

lizard 2.1 -10 320 MB/s 2452 MB/s 48.79 silesia.tar

zstd 1.5.7 -1 297 MB/s 901 MB/s 34.54 silesia.tar

snappy 1.2.2 375 MB/s 855 MB/s 47.89 silesia.tar

Results for enwik8.

Compressor name Compression Decompress. Ratio Filename

misa77 0.2.0 -0 38.7 MB/s 4802 MB/s 48.59 enwik8

misa77 0.2.0 -1 40.6 MB/s 4134 MB/s 44.05 enwik8

zxc 0.12.0 -3 70.1 MB/s 2674 MB/s 52.05 enwik8

zxc 0.12.0 -5 38.7 MB/s 2627 MB/s 46.85 enwik8

zxc 0.12.0 -4 52.7 MB/s 2593 MB/s 48.60 enwik8

lzsse4fast 2019-04-18 141 MB/s 2581 MB/s 47.11 enwik8

lzsse8fast 2019-04-18 135 MB/s 2553 MB/s 47.25 enwik8

lizard 2.1 -10 240 MB/s 2462 MB/s 57.31 enwik8

lz4 1.10.0 276 MB/s 2355 MB/s 57.26 enwik8

lz4hc 1.10.0 -12 9.89 MB/s 2167 MB/s 41.91 enwik8

zstd 1.5.7 -1 225 MB/s 831 MB/s 40.66 enwik8

snappy 1.2.2 240 MB/s 568 MB/s 55.93 enwik8

As misa77's performance is quite "spiky" (depending on the shape of the data being compressed), a file-level breakdown for the silesia corpus yields some interesting insights into its performance. Note:

The visuals that follow are derived from the benchmark results at misc/lzbench-results-archive/0.2.0/intel.txt These results are with the same x86-64 (Intel) setup mentioned previously.

Decode speed relative to lz4 At level 0, misa77 decodes faster than lz4 on all 12 files (some by huge margins). All other levels decode faster on 11/12 files. The exception is x-ray, which is highly incompressible (lz4 has a ratio of nearly 1.0 on this file and essentially devolves to a memcpy).

Throughput vs ratio, against popular fast-decode codecs On the compressible files, misa77 sits on the decode-throughput/ratio Pareto frontier: it decodes fastest while ~matching or beating the ratio of the other fast-LZ codecs. sao and x-ray are exceptions due to the reasons stated before.

Requirements

A C++20 compiler (both GCC and Clang are fine). CMake >= 3.20. A little-endian 64-bit system. The misa CLI needs POSIX (Linux, macOS).

Note: On x86-64, AVX2/SSE2 are selected at runtime. Other architectures use a portable path that has no explicit intrinsics, but is easily auto-vectorizable by compilers (and from my testing, does auto-vectorize on Apple ARM at the very least). Building cmake -B build -DCMAKE_BUILD_TYPE=Release cmake --build build This produces the misa CLI at build/misa. For a binary tuned to the exact machine you'll run it on, add -DMISA77_MARCH=native (I recommend this). To run the round-trip test:

Library Usage The build produces a static library (CMake target misa77) with a small C++ API in misa77/misa77.h. The easiest integration is a git submodule (or CMake FetchContent) plus: add_subdirectory(misa77) target_link_libraries(your_app PRIVATE misa77) Sample usage: #include <misa77/misa77.h> #include

// compress (pick a level with misa77::config(0/1); default is 1), returns 0 on failure std::vector<uint8_t> compressed(misa77::compress_bound(input_size)); uint64_t csize = misa77::compress(input, input_size, compressed.data(), compressed.size()); compressed.resize(csize);

// decompress, returns original_size on success uint64_t original_size = misa77::decompressed_size(compressed.data()); std::vector<uint8_t> output(misa77::decompressed_buffer_bound(original_size)); uint64_t written = misa77::decompress(compressed.data(), csize, output.data(), output.size()); Two things to keep in mind:

You must size the destination buffers with compress_bound / decompressed_buffer_bound. As of v0.2.x, decompress assumes that the input is valid (see Status). An input-safe decoder will be added in v0.3.0.

The experimental modes are declared in misa77/experimental.h, with usage documented in comments (these keep changing frequently so I don't wanna "formally" document them here just yet). CLI Usage misa is a single, dependency-free binary with three file-based subcommands. It operates on single files only (there's no directory or pipe support, so tar first if you need those). misa compress FILE # -> FILE.misa77 misa decompress FILE.misa77 # -> FILE misa suggest FILE # -> FILE.misap (tuned params) misa compress takes -l N / --level N to pick the compression level (default is 1). There are also some experimental compression modes (at most one at a time, not combinable with --level):

Flag Effect

--adaptive autotune the compressor based on the input for decode speed (only use this with homogeneous data)

--params F.misap compress with a vector from misa suggest

--yolo high-effort, decode-optimized

--adaptive and suggest also take --tune loose / --tune tight (similar tradeoffs as level 0/1, and the default is loose) and --sample MB (how much input to sample when picking params, default is 2 MB). Everywhere, -o PATH sets the output path and -f overwrites without asking. misa compress -l 0 enwik8 # enwik8 -> enwik8.misa77, fastest-decode level misa decompress enwik8.misa77 # back to enwik8

tune on a sample, then reuse the params:

misa suggest --tune tight data.bin # -> data.misap misa compress --params data.misap data.bin Documentation The underlying stream format (used by the library functions) and the container format for .misa77 files (produced by the CLI) can be found in docs/. Status

misa77's format may change unexpectedly as it's still v0.x.y. The decoder assumes that the input is a valid misa77 stream. Invalid input is UB and I offer no guarantees for whatever misa77 does in this case. It's been through some local fuzzing but is not hardened, so treat it as experimental.

Note: misa77 has evolved out of a less polished endeavour to learn performance engineering, and its history can be found in this archived repository. Acknowledgement Inspiration has been taken from:

LZ4 zxc lizard

Lastly, Claude Opus 4.8 and Fable 5 helped a lot with scripting, tooling, and building the CLI. License MIT (see LICENSE).