diff options
author | Gao Xiang <hsiangkao@linux.alibaba.com> | 2023-03-09 19:26:29 +0800 |
---|---|---|
committer | Gao Xiang <hsiangkao@linux.alibaba.com> | 2023-03-10 00:39:22 +0800 |
commit | 53ee2c7a279080937aea649183567abcd26f9d2b (patch) | |
tree | b2b3dcb769a303c4ee2299b0e7c9b8697d01d6ce | |
parent | 281c097dcfe8b6572149d5ef2e2c80b9e160844e (diff) | |
download | erofs-utils-53ee2c7a279080937aea649183567abcd26f9d2b.tar.gz |
erofs-utils: improve documentation for upcoming 1.6
Refine README as well as add docs/INSTALL.md and docs/PERFORMANCE.md.
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230309112630.74230-2-hsiangkao@linux.alibaba.com
-rw-r--r-- | README | 241 | ||||
-rw-r--r-- | docs/INSTALL.md | 71 | ||||
-rw-r--r-- | docs/PERFORMANCE.md | 183 |
3 files changed, 341 insertions, 154 deletions
@@ -1,7 +1,7 @@ erofs-utils =========== -userspace tools for EROFS filesystem, currently including: +Userspace tools for EROFS filesystem, currently including: mkfs.erofs filesystem formatter erofsfuse FUSE daemon alternative @@ -9,76 +9,60 @@ userspace tools for EROFS filesystem, currently including: fsck.erofs filesystem compatibility & consistency checker as well as extractor -Dependencies & build --------------------- - lz4 1.8.0+ for lz4 enabled [2], lz4 1.9.3+ highly recommended [4][5]. - XZ Utils 5.3.2alpha [6] or later versions for MicroLZMA enabled. - - libfuse 2.6+ for erofsfuse enabled as a plus. - -How to build with lz4-1.9.0 or above -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -To build, you can run the following commands in order: - -:: - - $ ./autogen.sh - $ ./configure - $ make - -mkfs.erofs binary will be generated under mkfs folder. +EROFS filesystem overview +------------------------- -* For lz4 < 1.9.2, there are some stability issues about - LZ4_compress_destSize(). (lz4hc isn't impacted) [3]. +EROFS filesystem stands for Enhanced Read-Only File System. It aims to +form a generic read-only filesystem solution for various read-only use +cases instead of just focusing on storage space saving without +considering any side effects of runtime performance. -** For lz4 = 1.9.2, there is a noticeable regression about - LZ4_decompress_safe_partial() [5], which impacts erofsfuse - functionality for legacy images (without 0PADDING). +Typically EROFS could be considered in the following use scenarios: + - Firmwares in performance-sensitive systems, such as system + partitions of Android smartphones; -How to build with lz4-1.8.0~1.8.3 -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + - Mountable immutable images such as container images for effective + metadata & data access compared with tar, cpio or other local + filesystems (e.g. ext4, XFS, btrfs, etc.) -For these old lz4 versions, lz4hc algorithm cannot be supported -without lz4-static installed due to LZ4_compress_HC_destSize() -unstable api usage, which means lz4 will only be available if -lz4-static isn't found. + - FSDAX-enabled rootfs for secure containers (Linux 5.15+); -On Fedora, lz4-static can be installed by using: + - Live CDs which need a set of files with another high-performance + algorithm to optimize startup time; others files for archival + purposes only are not needed; - yum install lz4-static.x86_64 + - and more. -However, it's still not recommended using those versions directly -since there are serious bugs in these compressors, see [2] [3] [4] -as well. +Note that all EROFS metadata is uncompressed by design, so that you +could take EROFS as a drop-in read-only replacement of ext4, XFS, +btrfs, etc. without any compression-based dependencies and EROFS can +bring more effective filesystem accesses to users with reduced +metadata. -How to build with liblzma -~~~~~~~~~~~~~~~~~~~~~~~~~ +For more details of EROFS filesystem itself, please refer to: +https://www.kernel.org/doc/html/next/filesystems/erofs.html -In order to enable LZMA support, build with the following commands: - $ ./configure --enable-lzma - $ make +For more details on how to build erofs-utils, see `docs/INSTALL.md`. -Additionally, you could specify liblzma build paths with: - --with-liblzma-incdir and --with-liblzma-libdir +For more details about filesystem performance, see +`docs/PERFORMANCE.md`. mkfs.erofs ---------- -two main kinds of EROFS images can be generated: (un)compressed. +Two main kinds of EROFS images can be generated: (un)compressed images. - - For uncompressed images, there will be none of compression - files in these images. However, it can decide whether the tail - block of a file should be inlined or not properly [1]. + - For uncompressed images, there will be none of compresssed files in + these images. However, it can decide whether the tail block of a + file should be inlined or not properly [1]. - - For compressed images, it'll try to use specific algorithms - first for each regular file and see if storage space can be - saved with compression. If not, fallback to an uncompressed - file. + - For compressed images, it'll try to use the given algorithms first + for each regular file and see if storage space can be saved with + compression. If not, fallback to an uncompressed file. -How to generate EROFS images (lz4 for Linux 5.3+, lzma for Linux 5.16+) +How to generate EROFS images (LZ4 for Linux 5.3+, LZMA for Linux 5.16+) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Currently lz4(hc) and lzma are available for compression, e.g. @@ -113,12 +97,55 @@ Note that large pcluster size can cause bad random performance, so please evaluate carefully in advance. Or make your own per-(sub)file compression strategies according to file access patterns if needed. +How to generate EROFS images with multiple algorithms (Linux 5.16+) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +It's possible to generate an EROFS image with files in different +algorithms due to various purposes. For example, LZMA for archival +purposes and LZ4 for runtime purposes. + +In order to use alternative algorithms, just specify two or more +compressing configurations together separated by ':' like below: + -zlzma:lz4hc,12:lzma,9 -C32768 + +Although mkfs still choose the first one by default, you could try to +write a compress-hints file like below: + 4096 1 .*\.so$ + 32768 2 .*\.txt$ + 4096 sbin/.*$ + 16384 0 .* + +and specify with `--compress-hints=` so that ".so" files will use +"lz4hc,12" compression with 4k pclusters, ".txt" files will use +"lzma,9" compression with 32k pclusters, files under "/sbin" will use +the default "lzma" compression with 4k plusters and other files will +use "lzma" compression with 16k pclusters. + +Note that the largest pcluster size should be specified with the "-C" +option (here 32k pcluster size), otherwise all larger pclusters will be +limited. + +How to generate well-compressed EROFS images +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Even if EROFS is not designed for such purposes in the beginning, it +could still produce some smaller images (not always) compared to other +approaches with better performance (see `docs/PERFORMANCE.md`). In +order to build well-compressed EROFS images, try the following options: + -C1048576 (5.13+) + -Eztailpacking (5.16+) + -Efragments / -Eall-fragments ( 6.1+); + -Ededupe ( 6.1+). + +Also EROFS uses lz4hc level 9 by default, whereas some other approaches +use lz4hc level 12 by default. So please explicitly specify +`-zlz4hc,12 ` for comparison purposes. + How to generate legacy EROFS images (Linux 4.19+) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Decompression inplace and compacted indexes have been introduced in -Linux upstream v5.3, which are not forward-compatible with older -kernels. +Linux v5.3, which are not forward-compatible with older kernels. In order to generate _legacy_ EROFS images for old kernels, consider adding "-E legacy-compress" to the command line, e.g. @@ -153,21 +180,6 @@ significant I/O overhead, double caching, etc.) Therefore, NEVER use it if performance is the top concern. -Note that extended attributes and ACLs aren't implemented yet due to -the current Android use case vs limited time. If you are interested, -contribution is, as always, welcome. - -How to build erofsfuse -~~~~~~~~~~~~~~~~~~~~~~ - -It's disabled by default as an experimental feature for now due to -the extra libfuse dependency, to enable and build it manually: - - $ ./configure --enable-fuse - $ make - -erofsfuse binary will be generated under fuse folder. - How to mount an EROFS image with erofsfuse ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -190,36 +202,8 @@ dump.erofs and fsck.erofs dump.erofs and fsck.erofs are used to analyze, check, and extract EROFS filesystems. Note that extended attributes and ACLs are still -unsupported when extracting images with fsck.erofs. - -Container images ----------------- - -EROFS filesystem is well-suitably used for container images with -advanced features like chunk-based files, multi-devices (blobs) -and new fscache backend for lazy pulling and cache management, etc. - -For example, CNCF Dragonfly Nydus image service [7] introduces an -(EROFS-compatible) RAFS v6 image format to overcome flaws of the -current OCIv1 tgz images so that: - - - Images can be downloaded on demand in chunks aka lazy pulling with - new fscache backend (5.19+) or userspace block devices (5.16+); - - - Finer chunk-based content-addressable data deduplication to minimize - storage, transmission and memory footprints; - - - Merged filesystem tree to remove all metadata of intermediate layers - as an option; - - - (e)stargz, zstd::chunked and other formats can be converted and run - on the fly; - - - and more. - -Apart from Dragonfly Nydus, a native user daemon is planned to be added -to erofs-utils to parse EROFS, (e)stargz and zstd::chunked images from -network too as a real part of EROFS filesystem project. +unsupported when extracting images with fsck.erofs. If you are +interested, contribution is, as always, welcome. Contribution @@ -233,58 +217,7 @@ patches or feedback to: Comments -------- -[1] According to the EROFS on-disk format, the tail block of files - could be inlined aggressively with its metadata in order to reduce - the I/O overhead and save the storage space (called tail-packing). - -[2] There was a bug until lz4-1.8.3, which can crash erofs-utils - randomly. Fortunately bugfix by our colleague Qiuyang Sun was - merged in lz4-1.9.0. - - For more details, please refer to - https://github.com/lz4/lz4/commit/660d21272e4c8a0f49db5fc1e6853f08713dff82 - -[3] There were many bugfixes merged into lz4-1.9.2 for - LZ4_compress_destSize(), and I once ran into some crashs due to - those issues. * Again lz4hc is not affected. * - - [LZ4_compress_destSize] Allow 2 more bytes of match length - https://github.com/lz4/lz4/commit/690009e2c2f9e5dcb0d40e7c0c40610ce6006eda - - [LZ4_compress_destSize] Fix rare data corruption bug - https://github.com/lz4/lz4/commit/6bc6f836a18d1f8fd05c8fc2b42f1d800bc25de1 - - [LZ4_compress_destSize] Fix overflow condition - https://github.com/lz4/lz4/commit/13a2d9e34ffc4170720ce417c73e396d0ac1471a - - [LZ4_compress_destSize] Fix off-by-one error in fix - https://github.com/lz4/lz4/commit/7c32101c655d93b61fc212dcd512b87119dd7333 - - [LZ4_compress_destSize] Fix off-by-one error - https://github.com/lz4/lz4/commit/d7cad81093cd805110291f84d64d385557d0ffba - - since upstream lz4 doesn't have stable branch for old versions, it's - preferred to use latest upstream lz4 library (although some regressions - could happen since new features are also introduced to latest upstream - version as well) or backport all stable bugfixes to old stable versions, - e.g. our unofficial lz4 fork: https://github.com/erofs/lz4 - -[4] LZ4HC didn't compress long zeroed buffer properly with - LZ4_compress_HC_destSize() - https://github.com/lz4/lz4/issues/784 - - which has been resolved in - https://github.com/lz4/lz4/commit/e7fe105ac6ed02019d34731d2ba3aceb11b51bb1 - - and already included in lz4-1.9.3, see: - https://github.com/lz4/lz4/releases/tag/v1.9.3 - -[5] LZ4_decompress_safe_partial is broken in 1.9.2 - https://github.com/lz4/lz4/issues/783 - - which is also resolved in lz4-1.9.3. - -[6] https://tukaani.org/xz/xz-5.3.2alpha.tar.xz - -[7] https://nydus.dev - https://github.com/dragonflyoss/image-service +[1] According to the EROFS on-disk format, the tail blocks of files + could be inlined aggressively with their metadata (called + tail-packing) in order to minimize the extra I/Os and the storage + space. diff --git a/docs/INSTALL.md b/docs/INSTALL.md new file mode 100644 index 0000000..2e818da --- /dev/null +++ b/docs/INSTALL.md @@ -0,0 +1,71 @@ +This document describes how to configure and build erofs-utils from +source. + +See the [README](../README) file in the top level directory about +the brief overview of erofs-utils. + +## Dependencies & build + +LZ4 1.9.3+ for LZ4(HC) enabled [^1]. + +[XZ Utils 5.3.2alpha+](https://tukaani.org/xz/xz-5.3.2alpha.tar.gz) for +LZMA enabled, [XZ Utils 5.4+](https://tukaani.org/xz/xz-5.4.1.tar.gz) +highly recommended. + +libfuse 2.6+ for erofsfuse enabled. + +[^1]: It's not recommended to use LZ4 versions under 1.9.3 since +unexpected crashes could make trouble to end users due to broken +LZ4_compress_destSize() (fixed in v1.9.2), +[LZ4_compress_HC_destSize()](https://github.com/lz4/lz4/commit/660d21272e4c8a0f49db5fc1e6853f08713dff82) or +[LZ4_decompress_safe_partial()](https://github.com/lz4/lz4/issues/783). + +## How to build with LZ4 + +To build, the following commands can be used in order: + +``` sh +$ ./autogen.sh +$ ./configure +$ make +``` + +`mkfs.erofs`, `dump.erofs` and `fsck.erofs` binaries will be +generated under the corresponding folders. + +## How to build with liblzma + +In order to enable LZMA support, build with the following commands: + +``` sh +$ ./configure --enable-lzma +$ make +``` + +Additionally, you could specify liblzma target paths with +`--with-liblzma-incdir` and `--with-liblzma-libdir` manually. + +## How to build erofsfuse + +It's disabled by default as an experimental feature for now due +to the extra libfuse dependency, to enable and build it manually: + +``` sh +$ ./configure --enable-fuse +$ make +``` + +`erofsfuse` binary will be generated under `fuse` folder. + +## How to install erofs-utils manually + +Use the following command to install erofs-utils binaries: + +``` sh +# make install +``` + +By default, `make install` will install all the files in +`/usr/local/bin`, `/usr/local/lib` etc. You can specify an +installation prefix other than `/usr/local` using `--prefix`, +for instance `--prefix=$HOME`. diff --git a/docs/PERFORMANCE.md b/docs/PERFORMANCE.md new file mode 100644 index 0000000..fdf4f79 --- /dev/null +++ b/docs/PERFORMANCE.md @@ -0,0 +1,183 @@ +# Test setup + +Processor: x86_64, Intel(R) Xeon(R) Platinum 8369B CPU @ 2.70GHz * 2 VCores + +Storage: Cloud disk, 3000 IOPS upper limit + +OS Kernel: Linux 6.2 + +Software: LZ4 1.9.3, erofs-utils 1.6, squashfs-tools 4.5.1 + +Disclaimer: Test results could be varied from different hardware and/or data patterns. Therefore, the following results are **ONLY for reference**. + +# Benchmark on multiple files + +[Rootfs of Debian docker image](https://github.com/debuerreotype/docker-debian-artifacts/blob/dist-amd64/bullseye/rootfs.tar.xz?raw=true) is used as the dataset, which contains 7000+ files and directories. +Note that that dataset can be replaced regularly, and the SHA1 of the snapshot "rootfs.tar.xz" used here is "aee9b01a530078dbef8f08521bfcabe65b244955". + +## Image size + +| Size | Filesystem | Cluster size | Build options | +|-----------|------------|--------------|----------------------------------------------------------------| +| 124669952 | erofs | uncompressed | -T0 [^1] | +| 124522496 | squashfs | uncompressed | -noD -noI -noX -noF -no-xattrs -all-time 0 -no-duplicates [^2] | +| 73601024 | squashfs | 4096 | -b 4096 -comp lz4 -Xhc -no-xattrs -all-time 0 | +| 73121792 | erofs | 4096 | -zlz4hc,12 [^3] -C4096 -Efragments -T0 | +| 67162112 | squashfs | 16384 | -b 16384 -comp lz4 -Xhc -no-xattrs -all-time 0 | +| 65478656 | erofs | 16384 | -zlz4hc,12 -C16384 -Efragments -T0 | +| 61456384 | squashfs | 65536 | -b 65536 -comp lz4 -Xhc -no-xattrs -all-time 0 | +| 59834368 | erofs | 65536 | -zlz4hc,12 -C65536 -Efragments -T0 | +| 59150336 | squashfs | 131072 | -b 131072 -comp lz4 -Xhc -no-xattrs -all-time 0 | +| 58515456 | erofs | 131072 | -zlz4hc,12 -C131072 -Efragments -T0 | + +[^1]: Forcely reset all timestamps to match squashfs on-disk basic inodes for now. +[^2]: Currently erofs-utils doesn't actively de-duplicate identical files although the on-disk format supports this. +[^3]: Because squashfs uses level 12 for LZ4HC by default. + +## Sequential data access + +hyperfine -p "echo 3 > /proc/sys/vm/drop_caches; sleep 1" "tar cf - . | cat > /dev/null" + +| Filesystem | Cluster size | Time | +|------------|--------------|---------------------------------| +| squashfs | 4096 | 10.257 s ± 0.031 s | +| erofs | uncompressed | 1.111 s ± 0.022 s | +| squashfs | uncompressed | 1.034 s ± 0.020 s | +| squashfs | 131072 | 941.3 ms ± 7.5 ms | +| erofs | 4096 | 848.1 ms ± 17.8 ms | +| erofs | 131072 | 724.2 ms ± 11.0 ms | + +## Sequential metadata access + +hyperfine -p "echo 3 > /proc/sys/vm/drop_caches; sleep 1" "tar cf /dev/null ." + +| Filesystem | Cluster size | Time | +|------------|--------------|---------------------------------| +| erofs | uncompressed | 419.6 ms ± 8.2 ms | +| squashfs | 4096 | 142.5 ms ± 5.4 ms | +| squashfs | uncompressed | 129.2 ms ± 3.9 ms | +| squashfs | 131072 | 125.4 ms ± 4.0 ms | +| erofs | 4096 | 75.5 ms ± 3.5 ms | +| erofs | 131072 | 65.8 ms ± 3.6 ms | + +[ Note that erofs-utils currently doesn't perform quite well for such cases due to metadata arrangement when building. It will be fixed in the later versions. ] + +## Small random data access (~7%) + +find mnt -type f -printf "%p\n" | sort -R | head -n 500 > list.txt +hyperfine -p "echo 3 > /proc/sys/vm/drop_caches; sleep 1" "cat list.txt | xargs cat > /dev/null" + +| Filesystem | Cluster size | Time | +|------------|--------------|---------------------------------| +| squashfs | 4096 | 1.386 s ± 0.032 s | +| squashfs | uncompressed | 1.083 s ± 0.044 s | +| squashfs | 131072 | 1.067 s ± 0.046 s | +| erofs | 4096 | 249.6 ms ± 6.5 ms | +| erofs | uncompressed | 237.8 ms ± 6.3 ms | +| erofs | 131072 | 189.6 ms ± 7.8 ms | + + +## Small random metadata access (~7%) + +find mnt -type f -printf "%p\n" | sort -R | head -n 500 > list.txt +hyperfine -p "echo 3 > /proc/sys/vm/drop_caches; sleep 1" "cat list.txt | xargs stat" + +| Filesystem | Cluster size | Time | +|------------|--------------|---------------------------------| +| squashfs | 4096 | 817.0 ms ± 34.5 ms | +| squashfs | 131072 | 801.0 ms ± 40.1 ms | +| squashfs | uncompressed | 741.3 ms ± 18.2 ms | +| erofs | uncompressed | 197.8 ms ± 4.1 ms | +| erofs | 4096 | 63.1 ms ± 2.0 ms | +| erofs | 131072 | 60.7 ms ± 3.6 ms | + +## Full random data access (~100%) + +find mnt -type f -printf "%p\n" | sort -R > list.txt +hyperfine -p "echo 3 > /proc/sys/vm/drop_caches; sleep 1" "cat list.txt | xargs cat > /dev/null" + +| Filesystem | Cluster size | Time | +|------------|--------------|---------------------------------| +| squashfs | 4096 | 20.668 s ± 0.040 s | +| squashfs | uncompressed | 12.543 s ± 0.041 s | +| squashfs | 131072 | 11.753 s ± 0.412 s | +| erofs | uncompressed | 1.493 s ± 0.023 s | +| erofs | 4096 | 1.223 s ± 0.013 s | +| erofs | 131072 | 598.2 ms ± 6.6 ms | + +## Full random metadata access (~100%) + +find mnt -type f -printf "%p\n" | sort -R > list.txt +hyperfine -p "echo 3 > /proc/sys/vm/drop_caches; sleep 1" "cat list.txt | xargs stat" + +| Filesystem | Cluster size | Time | +|------------|--------------|---------------------------------| +| squashfs | 131072 | 9.212 s ± 0.467 s | +| squashfs | 4096 | 8.905 s ± 0.147 s | +| squashfs | uncompressed | 7.961 s ± 0.045 s | +| erofs | 4096 | 661.2 ms ± 14.9 ms | +| erofs | uncompressed | 125.8 ms ± 6.6 ms | +| erofs | 131072 | 119.6 ms ± 5.5 ms | + + +# FIO benchmark on a single large file + +`silesia.tar` (203M) is used to benchmark, which could be generated from unzipping [silesia.zip](http://mattmahoney.net/dc/silesia.zip) and tar. + +## Image size + +| Size | Filesystem | Cluster size | Build options | +|-----------|------------|--------------|-----------------------------------------------------------| +| 114339840 | squashfs | 4096 | -b 4096 -comp lz4 -Xhc -no-xattrs | +| 104972288 | erofs | 4096 | -zlz4hc,12 -C4096 | +| 98033664 | squashfs | 16384 | -b 4096 -comp lz4 -Xhc -no-xattrs | +| 89571328 | erofs | 16384 | -zlz4hc,12 -C16384 | +| 85143552 | squashfs | 65536 | -b 65536 -comp lz4 -Xhc -no-xattrs | +| 81211392 | squashfs | 131072 | -b 131072 -comp lz4 -Xhc -no-xattrs | +| 80519168 | erofs | 65536 | -zlz4hc,12 -C65536 | +| 78888960 | erofs | 131072 | -zlz4hc,12 -C131072 | + +## Sequential I/Os + +fio -filename=silesia.tar -bs=4k -rw=read -name=job1 + +| Filesystem | Cluster size | Bandwidth | +|------------|--------------|-----------| +| erofs | 65536 | 624 MiB/s | +| erofs | 16384 | 600 MiB/s | +| erofs | 4096 | 569 MiB/s | +| erofs | 131072 | 535 MiB/s | +| squashfs | 131072 | 236 MiB/s | +| squashfs | 65536 | 157 MiB/s | +| squashfs | 16384 | 55.2MiB/s | +| squashfs | 4096 | 12.5MiB/s | + +## Full Random I/Os + +fio -filename=silesia.tar -bs=4k -rw=randread -name=job1 + +| Filesystem | Cluster size | Bandwidth | +|------------|--------------|-----------| +| erofs | 131072 | 242 MiB/s | +| squashfs | 131072 | 232 MiB/s | +| erofs | 65536 | 198 MiB/s | +| squashfs | 65536 | 150 MiB/s | +| erofs | 16384 | 96.4MiB/s | +| squashfs | 16384 | 49.5MiB/s | +| erofs | 4096 | 33.7MiB/s | +| squashfs | 4096 | 6817KiB/s | + +## Small Random I/Os (~5%) + +fio -filename=silesia.tar -bs=4k -rw=randread --io_size=10m -name=job1 + +| Filesystem | Cluster size | Bandwidth | +|------------|--------------|-----------| +| erofs | 131072 | 19.2MiB/s | +| erofs | 65536 | 16.9MiB/s | +| squashfs | 131072 | 15.1MiB/s | +| erofs | 16384 | 14.7MiB/s | +| squashfs | 65536 | 13.8MiB/s | +| erofs | 4096 | 13.0MiB/s | +| squashfs | 16384 | 11.7MiB/s | +| squashfs | 4096 | 4376KiB/s | |