| Age | Commit message (Collapse) | Author | Files | Lines |
|---|
|
Signed-off-by: Gao Xiang <xiang@kernel.org>
|
|
commit 8cbc205185a1 ("erofs-utils: mkfs: fix corrupted directories
with hardlinks") introduced a reference leak although it has no real
impact to users. Fix it now.
Signed-off-by: Gao Xiang <xiang@kernel.org>
Link: https://lore.kernel.org/r/20231019224328.26015-1-xiang@kernel.org
|
|
An inode with hard links may belong to several directories. It's
invalid to update `subdirs_queued` for hard-link inodes since it
only records one of the parent directories.
References: https://github.com/NixOS/nixpkgs/issues/261394
Fixes: 21d84349e79a ("erofs-utils: rearrange on-disk metadata")
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20231017144420.289469-1-hsiangkao@linux.alibaba.com
|
|
`errno` should be set to a non-negative value here.
Link: https://lore.kernel.org/r/CAB+1q0Q3+7s1Lt8uW6DWZ7vfjhEKhG7O7MAQhCuH-C10cr9F4g@mail.gmail.com
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
|
|
AC_RUN_IFELSE expects the action if cross compiling. If not provided
cross compilation fails with error "configure: error: cannot run test
program while cross compiling".
Use 4096 as the buest guess PAGESIZE if cross compiling as it is still
the most common page size.
Reported-in: https://lore.kernel.org/all/0101018aec71b531-0a354b1a-0b70-47a1-8efc-fea8c439304c-000000@us-west-2.amazonses.com/
Fixes: 8ee2e591dfd6 ("erofs-utils: support detecting maximum block size")
Signed-off-by: Sandeep Dhavale <dhavale@google.com>
Link: https://lore.kernel.org/r/20231005224008.817830-1-dhavale@google.com
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
|
|
Signed-off-by: Gao Xiang <xiang@kernel.org>
|
|
Similar to 876bec09e48a ("erofs-utils: lib: fix missing CBLKCNT for
big pcluster dedupe"), the previous CBLKCNT cannot be dropped due to
the extent shortening process.
It may cause data corruption on specific data patterns only if both
big pcluster and dedupe features are enabled.
Link: https://lore.kernel.org/r/20230921032417.82739-1-hsiangkao@linux.alibaba.com
Fixes: f3f9a2ce3137 ("erofs-utils: mkfs: introduce global compressed data deduplication")
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
|
|
Actually, it's false positive and only used for build assertion.
Reported-by: Kelvin Zhang <zhangkelvin@google.com>
Signed-off-by: Gao Xiang <xiang@kernel.org>
Link: https://lore.kernel.org/r/20230920200314.9193-1-hsiangkao@aol.com
|
|
Don't output more than 255 shared xattrs for a single inode due to the
EROFS on-disk format limitation.
Fixes: 116ac0a254fc ("erofs-utils: introduce shared xattr support")
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230920190220.1837650-1-hsiangkao@linux.alibaba.com
|
|
-b block-size
-E ^xattr-name-filter
--gzip
--tar=[fi]
--xattr-prefix=X
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230920164128.1637554-1-hsiangkao@linux.alibaba.com
|
|
Temporarily move the common part into __erofs_fill_inode() for tarerofs.
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230920051223.657008-1-hsiangkao@linux.alibaba.com
|
|
GNU tar volume labels (by using `-V`) will be applied to EROFS.
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230920035141.533474-1-hsiangkao@linux.alibaba.com
|
|
We move `idx` pointer as we iterate through for loop based on `count`. If
we error out from the loop, use the original pointer of allocated memory
when calling free().
Fixes: 39147b48b76d ("erofs-utils: lib: add erofs_rebuild_load_tree() helper")
Signed-off-by: Sandeep Dhavale <dhavale@google.com>
Link: https://lore.kernel.org/r/20230919210220.3657736-1-dhavale@google.com
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
|
|
Introduce iostream to wrap up the input tarball stream for tarerofs.
Besides, add builtin tgz support if zlib is linked to mkfs.
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230919185947.3996843-1-hsiangkao@linux.alibaba.com
|
|
Each tar header keeps the full file path. It's useful to move most
recently used intermediate dirs to list heads.
User time of tarerofs index mode can be reduced by 19%.
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230915082619.3533530-1-hsiangkao@linux.alibaba.com
|
|
Previously, each tar data file will be kept as a temporary file before
landing to the target image since the input stream may be non-seekable.
It's somewhat ineffective. Let's introduce a new diskbuf approach to
manage those buffers. Laterly, each stream can be redirected to blob
files for external reference.
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230920034655.38381-1-xiang@kernel.org
|
|
Since kernel commit 8b465fecc35a ("erofs: support flattened block device
for multi-blob images"), the flatdev feature has been introduced to
support mounting multi-blobs container image as a single block device.
To enable this feature, the mapped_blkaddr of each device slot needs to
be set properly to the offset of the device in the flat address space.
The uuid of the source image is used as the corresponding device tag
in rebuild mode.
Signed-off-by: Jingbo Xu <jefflexu@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230915115333.17599-1-jefflexu@linux.alibaba.com
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
|
|
Add `--ovlfs-strip=[0|1]` option for tarfs and rebuild mode for now
in order to control whether some overlayfs related stuffs (e.g.
whiteout files, OVL_XATTR_OPAQUE and OVL_XATTR_ORIGIN xattrs) are
finally stripped.
This option is disabled by default for mkfs, that is, the overlayfs
related stuffs described above are kept in the image by default.
Specify `--ovlfs-strip=1` explicitly to strip these stuffs.
Signed-off-by: Jingbo Xu <jefflexu@linux.alibaba.com>
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230913120304.15741-9-jefflexu@linux.alibaba.com
|
|
Introduce a new EXPERIMENTAL rebuild mode, which can be used to
generate a meta-only multidev manifest image with an overlayfs-like
merged tree from multiple specific EROFS images either of
tarerofs index mode (--tar=i):
mkfs.erofs --tar=i --aufs layer0.erofs layer0.tar
...
mkfs.erofs --tar=i --aufs layerN.erofs layerN-1.tar
or mkfs.erofs uncompressed mode without inline data:
mkfs.erofs --tar=f -Enoinline_data --aufs layer0.erofs layer0.tar
...
mkfs.erofs --tar=f -Enoinline_data --aufs layerN-1.erofs layerN-1.tar
To merge these layers, just type:
mkfs.erofs merged.erofs layer0.erofs ... layerN-1.erofs
It doesn't support compression and/or flat inline datalayout yet.
Signed-off-by: Jingbo Xu <jefflexu@linux.alibaba.com>
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230913120304.15741-8-jefflexu@linux.alibaba.com
|
|
Add erofs_rebuild_load_tree() helper to load the inode tree from a
given erofs image, and make it merged acting as an overlayfs-like
model.
Since the content of the symlink file needs to be read when loading
tree, let's add a dependency on zlib_LIBS for mkfs.erofs.
Also rename tarerofs_dump_tree() to erofs_rebuild_dump_tree() since
it is also called in the rebuild mode.
Signed-off-by: Jingbo Xu <jefflexu@linux.alibaba.com>
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230913120304.15741-7-jefflexu@linux.alibaba.com
|
|
Rename tarerofs_get_dentry() to erofs_rebuild_get_dentry() and
move it into lib/rebuild.c.
No logic changes.
Signed-off-by: Jingbo Xu <jefflexu@linux.alibaba.com>
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230913120304.15741-6-jefflexu@linux.alibaba.com
|
|
Add erofs_insert_ihash() helper inserting inode into inode hash table.
Also add prototypes of erofs_iget() and erofs_iget_by_nid() in the
header file.
Signed-off-by: Jingbo Xu <jefflexu@linux.alibaba.com>
Reviewed-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230913120304.15741-5-jefflexu@linux.alibaba.com
|
|
Add erofs_read_xattrs_from_disk() helper to reading extended
attributes from disk.
Signed-off-by: Jingbo Xu <jefflexu@linux.alibaba.com>
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230913120304.15741-4-jefflexu@linux.alibaba.com
|
|
... so that it could be called from outside blobchunk.c later.
Signed-off-by: Jingbo Xu <jefflexu@linux.alibaba.com>
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230913120304.15741-3-jefflexu@linux.alibaba.com
|
|
Add list_splice_tail() helper.
Reviewed-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Signed-off-by: Jingbo Xu <jefflexu@linux.alibaba.com>
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230913120304.15741-2-jefflexu@linux.alibaba.com
|
|
Traditionally, truncating to small sizes will trigger some
flush-on-close semantics to avoid the notorious NULL files.
I'm not sure if it's our use case since:
1) we're creating new image files instead of reusing old ones;
2) it kills end-to-end performance in practice;
3) other programs like GNU TAR doesn't work as this too for
such meaningless comparsion;
Let's work around it now.
Link: https://lore.kernel.org/r/20230914093905.1600316-1-hsiangkao@linux.alibaba.com
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
|
|
Previously, EROFS_BLKSIZ was a constant so it doesn't matter to use %
operator. Let's convert the remaining ones now.
Link: https://lore.kernel.org/r/20230914061210.1296457-1-hsiangkao@linux.alibaba.com
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
|
|
Previously, the extended attribute name in `struct xattr_item` was
actually the part with the matched prefix stripped, which makes it
clumsy to strip or delete a specific attribute from one file.
To fix this, make the complete attribute name stored in xattr_item.
One thing worth noting is that the attribute name in xattr_item has a
trailing '\0' for ease of name comparison, while the trailing '\0' will
get stripped when writing to on-disk erofs_xattr_entry.
Signed-off-by: Jingbo Xu <jefflexu@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230913031804.84819-1-jefflexu@linux.alibaba.com
|
|
The value in variable 'e' is checked without initializing and can
wrongly signify end of tar 2 empty blocks.
Signed-off-by: Sandeep Dhavale <dhavale@google.com>
Link: https://lore.kernel.org/r/20230913221104.429825-8-dhavale@google.com
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
|
|
The intended assignment is already part of the next for loop
initialization.
Signed-off-by: Sandeep Dhavale <dhavale@google.com>
Link: https://lore.kernel.org/r/20230913221104.429825-7-dhavale@google.com
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
|
|
If call to inflateInit2() fails, release the memory allocated for buff
before returning.
Signed-off-by: Sandeep Dhavale <dhavale@google.com>
Link: https://lore.kernel.org/r/20230913221104.429825-6-dhavale@google.com
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
|
|
If z_erofs_pack_file_from_fd() fails, take the error path to free up the
allocated resources.
Signed-off-by: Sandeep Dhavale <dhavale@google.com>
Link: https://lore.kernel.org/r/20230913221104.429825-5-dhavale@google.com
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
|
|
Need to free allocated buffer_head bh if __erofs_battach() fails.
Signed-off-by: Sandeep Dhavale <dhavale@google.com>
Link: https://lore.kernel.org/r/20230913221104.429825-4-dhavale@google.com
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
|
|
padding is only used in the next if() block. Remove the redundant call.
Signed-off-by: Sandeep Dhavale <dhavale@google.com>
Link: https://lore.kernel.org/r/20230913221104.429825-3-dhavale@google.com
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
|
|
Free the memory allocated for 'entry' if strdup() fails.
Signed-off-by: Sandeep Dhavale <dhavale@google.com>
Link: https://lore.kernel.org/r/20230913221104.429825-2-dhavale@google.com
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
|
|
This basically follows the fix in kernel commit 001b8ccd0650 ("erofs:
fix compact 4B support for 16k block size").
Without this patch fsck on images with 16k block size reports corruption
which is not correct.
Fixes: 76b822726ff8 ("erofs-utils: introduce compacted compression indexes")
Signed-off-by: Sandeep Dhavale <dhavale@google.com>
Link: https://lore.kernel.org/r/20230912221204.52184-1-dhavale@google.com
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
|
|
Users may be interested in some details of the generated image.
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230912115059.1997278-1-hsiangkao@linux.alibaba.com
|
|
erofs_build_shared_xattrs_from_path()
Free the allocated sorted_n[] buffer when some error occurs.
Signed-off-by: Jingbo Xu <jefflexu@linux.alibaba.com>
Reviewed-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230912135134.21307-1-jefflexu@linux.alibaba.com
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
|
|
Currently mkfs will ignore any unknown extended option, which keeps
silent if a mistyped option is given. Erroring out in this case
allows users to catch their input mistakes more easily.
Link: https://lore.kernel.org/r/20230909123650.2184838-1-zhaoyifan@sjtu.edu.cn
Signed-off-by: Yifan Zhao <zhaoyifan@sjtu.edu.cn>
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
|
|
Later, some preset configurations will be added, which could enable
some of these features by default.
Let's add a way to turn each individual off too.
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230911071819.3714867-1-hsiangkao@linux.alibaba.com
|
|
`gcc (Debian 13.2.0-2) 13.2.0` gives the following report:
kite_deflate.c: In function 'kite_deflate_writeblock':
kite_deflate.c:428:57: warning: 'distLevels' may be used uninitialized
[-Wmaybe-uninitialized]
428 | fixed ? 5 :
distLevels[distSlot]);
| ^
kite_deflate.c:393:34: note: 'distLevels' was declared here
393 | const u8 *litLenLevels, *distLevels;
Actually, `distLevels` won't be used in the static-huffman mode.
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230912032701.8288-1-hsiangkao@linux.alibaba.com
|
|
Don't assume xattr_isize is 0 before since it's not true for the
upcoming rebuild mode.
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230909163240.42057-6-hsiangkao@linux.alibaba.com
|
|
Whiteouts are exposed in the overlayfs mount if these are from some
non-merged directory without OVL_XATTR_ORIGIN set on the directory.
To fix this, tag OVL_XATTR_ORIGIN (with empty value) on the parent
directory of whiteouts.
Fixes: 95d315fd7958 ("erofs-utils: introduce tarerofs")
Signed-off-by: Jingbo Xu <jefflexu@linux.alibaba.com>
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230909163240.42057-5-hsiangkao@linux.alibaba.com
|
|
Add erofs_inode_is_whiteout() helper to check if a given inode is
a whiteout.
Signed-off-by: Jingbo Xu <jefflexu@linux.alibaba.com>
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230909163240.42057-4-hsiangkao@linux.alibaba.com
|
|
Move all non-EROFS xattr macros to xattr.c and introduce
erofs_set_opaque_xattr() helper to hide all these details.
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230909163240.42057-3-hsiangkao@linux.alibaba.com
|
|
get_xattritem() will in no way return NULL. Use DBG_BUGONs instead.
Signed-off-by: Jingbo Xu <jefflexu@linux.alibaba.com>
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230909163240.42057-2-hsiangkao@linux.alibaba.com
|
|
They break the MacOS build [1] and actually have no use anymore after
commit 7715b294087e ("erofs-utils: configure: Use 64bit off_t").
[1] https://github.com/erofs/erofsnightly/actions/runs/6134394959/job/16647166641
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230910115744.83819-1-hsiangkao@linux.alibaba.com
|
|
`fuzz_erofsfsck` reports an issue [1] that:
$ fsck/fuzz_erofsfsck t2/erofsfsck_libfuzzer_jUknVp
==430136==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7ffe44a8de02 at pc 0x55932ea3640c bp 0x7ffe44a8b990 sp 0x7ffe44a8b160
WRITE of size 201338902 at 0x7ffe44a8de02 thread T0
#0 0x55932ea3640b in __asan_memset (/root/erofs-utils/fsck/fuzz_erofsfsck+0xf340b) (BuildId: 0bba6c9ddccb99f520b59bca08a3991a456f7cd4)
#1 0x55932ea8a8e2 in z_erofs_read_data /root/erofs-utils/lib/data.c:335:4
#2 0x55932ea8b136 in erofs_pread /root/erofs-utils/lib/data.c:369:10
Here the size should be `length - skip`, otherwise it could cause
the destination buffer overflow.
[1] https://github.com/erofs/erofsnightly/actions/runs/6104429691/job/16566461154
Fixes: 6c20a6afd871 ("erofs-utils: fuse: fix random readlink error")
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230907031953.14356-1-hsiangkao@linux.alibaba.com
|
|
Keep in sync with kernel commit 5ec693ca70dd ("erofs: don't warn
dedupe and fragments features anymore").
Except that erofs.mkfs is still single-threaded as a limitation for
backup use cases.
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230905043840.61616-1-hsiangkao@linux.alibaba.com
|
|
In some crafted erofs images, fsck.erofs may write outside the
destination directory, which may be used to do some dangerous things.
This commit fixes by checking all directory entry names with a '/'
character when fscking. Squashfs also met the same situation [1],
and have already fixed it here [2].
[1] https://github.com/plougher/squashfs-tools/issues/72
[2] https://github.com/plougher/squashfs-tools/commit/79b5a555058eef4e1e7ff220c344d39f8cd09646
Fixes: 412c8f908132 ("erofs-utils: fsck: add --extract=X support to extract to path X")
Reviewed-by: Guo Xuenan <guoxuenan@huawei.com>
Signed-off-by: Guo Xuenan <guoxuenan@huawei.com>
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230905023207.70314-1-hsiangkao@linux.alibaba.com
|
|
Or "Segmentation fault" can happen if errors are passed into
erofs_igrab().
Fixes: 21d84349e79a ("erofs-utils: rearrange on-disk metadata")
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230901155932.105830-1-hsiangkao@linux.alibaba.com
|
|
Currently, calling erofs_bflush() multiple times is broken due to
outdated `last_mapped_block`.
Fixes: 82dee4501c5a ("erofs-utils: mkfs: enable xattr name filter feature by default")
Fixes: 185b0bcdef4b ("erofs-utils: optimize buffer allocation logic")
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Reviewed-by: Jingbo Xu <jefflexu@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230901031634.115999-1-hsiangkao@linux.alibaba.com
|
|
`SCHILY.xattr.attr` indicates a POSIX.1-2001 coded version of the
Linux extended attributes. Let's dump such xattrs to erofs images.
In addition, `LIBARCHIVE.xattr` is also supported, which uses
URL-Encoding instead of the binary form.
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Tested-by: Jingbo Xu <jefflexu@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230901072642.68200-1-hsiangkao@linux.alibaba.com
|
|
Set mkfs default blocksize to current platform pagesize.
This means mkfs with default options will work on current
platform. If we are building image for a platform for a different
blocksize, we can override default with -b option up to
EROFS_MAX_BLOCK_SIZE.
Signed-off-by: Sandeep Dhavale <dhavale@google.com>
Link: https://lore.kernel.org/r/20230830231606.3783734-2-dhavale@google.com
[ Gao Xiang: compare with EROFS_MAX_BLOCK_SIZE for safety. ]
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
|
|
As erofs-utils supports different block sizes upto
EROFS_MAX_BLOCK_SIZE, relax the checks so same tools
can be used to create images for platforms where
page size can be greater than 4096.
Signed-off-by: Sandeep Dhavale <dhavale@google.com>
Link: https://lore.kernel.org/r/20230830231606.3783734-1-dhavale@google.com
[ Gao Xiang: refine the warning message. ]
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
|
|
Turn it on by default since it's a compatible feature. Instead,
it can be disabled explicitly with "-E^xattr-name-filter".
Signed-off-by: Jingbo Xu <jefflexu@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230831102949.119605-1-jefflexu@linux.alibaba.com
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
|
|
Introduce "-Exattr-name-filter" option to enable the xattr name bloom
filter feature.
Also support listing this feature in dump.erofs.
Signed-off-by: Jingbo Xu <jefflexu@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230829145504.93567-4-jefflexu@linux.alibaba.com
[ Gao Xiang: update the commit subject. ]
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
|
|
Let's keep in sync with kernel commit 3f339920175c ("erofs: update
on-disk format for xattr name filter").
Signed-off-by: Jingbo Xu <jefflexu@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230829145504.93567-3-jefflexu@linux.alibaba.com
[ Gao Xiang: truncate the commit message. ]
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
|
|
Add xxh32 library which could be used by following xattr bloom filter
feature.
Signed-off-by: Jingbo Xu <jefflexu@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230829145504.93567-2-jefflexu@linux.alibaba.com
[ Gao Xiang: use `BSD-2-Clause OR GPL-2.0+` for "erofs/xxhash.h". ]
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
|
|
Keep erofs_init_devices() in sync with kernel erofs_scan_devices()[1],
which scans the devtable (if any) automatically if sbi->extra_devices is
not explicitly specified.
Also fix the missing le32_to_cpu() when parsing the device slot. Read
and cache the number of blocks of each device for later use.
[1] https://git.kernel.org/torvalds/c/ba73eadd23d1
Fixes: 0ce853a01123 ("erofs-utils: fuse: add multiple device support")
Signed-off-by: Jingbo Xu <jefflexu@linux.alibaba.com>
Reviewed-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230823071517.12303-3-jefflexu@linux.alibaba.com
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
|
|
Set the correct dirent type for whiteout.
Fixes: 95d315fd7958 ("erofs-utils: introduce tarerofs")
Signed-off-by: Jingbo Xu <jefflexu@linux.alibaba.com>
Reviewed-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230823071517.12303-2-jefflexu@linux.alibaba.com
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
|
|
Otherwise, it could cause double-free if sbi reuses
when fuzzing [1].
[1] https://github.com/erofs/erofsnightly/actions/runs/5921003885/job/16053013007
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Reviewed-by: Jingbo Xu <jefflexu@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230821093929.17146-1-hsiangkao@linux.alibaba.com
|
|
Keep self allocated and maintained devname in erofs_sb_info.
Reviewed-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Signed-off-by: Jingbo Xu <jefflexu@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230816021347.126886-6-jefflexu@linux.alibaba.com
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
|
|
i_ino[0] is a unique inode serial number in the erofs filesystem to
count inodes one by one, while i_ino[1] is a unique number identifying
the source inode in the source directory, which is usually derived from
st->st_ino.
Read on-disk ino and store it in i_ino[0] given the above background.
Fixes: 5e35b75ad499 ("erofs-utils: introduce fuse implementation")
Reviewed-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Signed-off-by: Jingbo Xu <jefflexu@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230816021347.126886-4-jefflexu@linux.alibaba.com
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
|
|
In tarerofs index mode, tarerofs_write_chunk_indexes() is still called
even for files with i_size 0, and inode->u.chunkformat is initialized
accordingly. This will make the previously set inode->u.i_rdev be
overridden as u.chunkformat and u.i_rdev are reused in one union.
To tidy up the code of writing file, extract two helpers for index and
non-index modes. Also fix the missing erofs_iput() when
tarerofs_write_chunkes() fails.
Fixes: 95d315fd7958 ("erofs-utils: introduce tarerofs")
Reviewed-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Signed-off-by: Jingbo Xu <jefflexu@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230816021347.126886-2-jefflexu@linux.alibaba.com
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
|
|
Make erofs_listxattr() and erofs_getxattr() routine support long xattr
name prefixes.
Although the on-disk format allows long xattr name prefixes to be placed
in the meta inode or packed inode, currently mkfs.erofs will place them
in packed inode by default. Thus let's also read long xattr name prefixes
from packed inode by default.
Since we need to read the content of the packed inode from disk when
loading long xattr name prefixes, add dependency on zlib_LIBS for
mkfs.erofs to resolve the compiling dependency.
Signed-off-by: Jingbo Xu <jefflexu@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230817071455.12040-4-jefflexu@linux.alibaba.com
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
|
|
Add erofs_read_metadata() helper reading variable-sized metadata from
inode specified by @nid. Read from meta inode if @nid is 0.
Signed-off-by: Jingbo Xu <jefflexu@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230817071455.12040-3-jefflexu@linux.alibaba.com
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
|
|
Since the introduction of long xattr name prefix, match_prefix() will
search among the long xattr name prefixes first and return the matched
prefix, while erofs_getxattr() expects a base prefix even when the
queried xattr name matches a long prefix.
Thus introduce match_base_prefix() helper to do this.
Signed-off-by: Jingbo Xu <jefflexu@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230817071455.12040-2-jefflexu@linux.alibaba.com
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
|
|
Check the index before accessing array to avoid the potential
out-of-bound access.
Fixes: c47df5aa2d16 ("erofs-utils: fuse: introduce xattr support")
Reviewed-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Signed-off-by: Jingbo Xu <jefflexu@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230816034941.126866-2-jefflexu@linux.alibaba.com
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
|
|
Commit fc30780ebf90 ("erofs-utils: lib: avoid global sbi dependencies
(take 1)") updated the macro erofs_blknr by adding sbi argument.
Fixes: fc30780ebf90 ("erofs-utils: lib: avoid global sbi dependencies (take 1)")
Signed-off-by: Yue Hu <huyue2@coolpad.com>
Reviewed-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230815102405.19486-1-zbestahu@gmail.com
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
|
|
Remove prototypes of those that have been deleted.
Signed-off-by: Jingbo Xu <jefflexu@linux.alibaba.com>
Reviewed-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230810064633.56218-2-jefflexu@linux.alibaba.com
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
|
|
Fix double write of long xattr name prefixes in non-tarerofs mode.
Besides fix the compiling error of tar.h. Include "internal.h" to
introduce prototypes of `struct erofs_inode` and `struct erofs_sb_info`.
Fixes: 95d315fd7958 ("erofs-utils: introduce tarerofs")
Signed-off-by: Jingbo Xu <jefflexu@linux.alibaba.com>
Reviewed-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230810064633.56218-1-jefflexu@linux.alibaba.com
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
|
|
Let's print compression algorithms and sb_extslots as well as update
feature information for dump.erofs.
The proposed superblock dump is shown as below:
Filesystem magic number: 0xE0F5E1E2
Filesystem blocks: 4624
Filesystem inode metadata start block: 0
Filesystem shared xattr metadata start block: 0
Filesystem root nid: 37
Filesystem compr_algs: lz4, lzma
Filesystem sb_extslots: 0
Filesystem inode count: 6131
Filesystem created: Wed Jun 7 17:15:44 2023
Filesystem features: sb_csum mtime 0padding compr_cfgs big_pcluster
Filesystem UUID: not available
Signed-off-by: Guo Xuenan <guoxuenan@huawei.com>
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230808092335.111834-2-hsiangkao@linux.alibaba.com
|
|
dump.erofs needs to print supported algorithms instead of available
compressors.
In addition, clean up erofs_get_compress_algorithm_id() too.
Tested-by: Guo Xuenan <guoxuenan@huawei.com>
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230808092335.111834-1-hsiangkao@linux.alibaba.com
|
|
It's absolutely unsafe to reuse struct erofs_dir_context. Also,
we'd like to refactor erofs_get_pathname() in the future.
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230803050031.130026-1-hsiangkao@linux.alibaba.com
|
|
It is also useful to fill data blocks from tarballs at runtime in
order to implement image lazy pulling.
Let's generate an additional mapfile for such use cases:
mkfs.erofs --tar=0,MAPFILE IMAGE TARBALL
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Reviewed-by: Jingbo Xu <jefflexu@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230801132755.23997-1-hsiangkao@linux.alibaba.com
|
|
Set dir->i_nlink to 2 since "." and ".." are allocated.
Also, tarerofs_init_empty_dir() is removed then.
Signed-off-by: Jingbo Xu <jefflexu@linux.alibaba.com>
Reviewed-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230727045712.45226-2-jefflexu@linux.alibaba.com
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
|
|
Since EROFS chunk size can be configured on a per-file basis,
let's generate indexes with the best chunk size.
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230724170646.22281-3-hsiangkao@linux.alibaba.com
|
|
Mainly get rid of memory allocation on each chunk.
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230724170646.22281-2-hsiangkao@linux.alibaba.com
|
|
File position should be fixed to aligned with the chunk boundary.
Otherwise, incorrect data could be read.
Fixes: 7c49e8b195ad ("erofs-utils: support chunk-based sparse files")
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230724170646.22281-1-hsiangkao@linux.alibaba.com
|
|
chunksize may not always equal to blocksize (e.g. 4KiB). Also it can
lead to unexpected behaviors when the consecutive chunk merging is
implemented later since the chunksize is also on a per-file basis.
Fixes: d9a01943f8c5 ("AOSP: erofs-utils: mkfs: add block list support for chunked files")
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230721050755.82122-1-hsiangkao@linux.alibaba.com
|
|
Later mkfs is going to be able to convert multiple erofs images into
one merged erofs image, in which case there are multiple device
contexts and sbi instances.
In preparation for that, remove global device context and sbi if
possible except the buffer cache implementation, as there's still only
one erofs output image.
The device context is now folded into sbi. Global sbi still exists
but it is only used by binaries directly, e.g. mkfs/dump/fsck.
Signed-off-by: Jingbo Xu <jefflexu@linux.alibaba.com>
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230720073104.116439-1-hsiangkao@linux.alibaba.com
|
|
gcc-7 and earlier versions fail to infer the initial value of a
const global variable from another const global variable. Therefore,
compiling with gcc-7 and below will result in failure. In fact, for
global const variables, using macros is a better choice.
Signed-off-by: Li Yiyan <lyy0627@sjtu.edu.cn>
Link: https://lore.kernel.org/r/20230725085020.904884-1-lyy0627@sjtu.edu.cn
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
|
|
Let's follow kernel commit b780d3fc6107 ("erofs: simplify iloc()") by
passing in inodes directly to clean up all callers. Also rename iloc()
as erofs_iloc().
Signed-off-by: Jingbo Xu <jefflexu@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230719073319.27996-2-jefflexu@linux.alibaba.com
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
|
|
The helper is not quite useful.
Signed-off-by: Jingbo Xu <jefflexu@linux.alibaba.com>
Reviewed-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230718052101.124039-4-jefflexu@linux.alibaba.com
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
|
|
The 'L' entry is present in a header for a series of 1 or more 512-byte
tar blocks that hold just the filename for a file or directory with a
name over 100 chars.
Following that series is another header block, in the traditional form:
A header with type '0' (regular file) or '5' (directory), followed by
the appropriate number of data blocks with the entry data.
In the header for this series, the name will be truncated to the 1st 100
characters of the actual name.
Tested-by: Jingbo Xu <jefflexu@linux.alibaba.com>
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230717073531.43203-1-hsiangkao@linux.alibaba.com
|
|
Let's try to add a new mode "tarerofs" for mkfs.erofs.
It mainly aims at two use cases:
- Convert a tarball (or later tarballs with a merged view) into
a full EROFS image [--tar=f];
- Generate an EROFS manifest image to reuse tar data [--tar=i],
which also enables EROFS 512-byte blocks.
The second use case is mainly prepared for OCI direct mount without
OCI blob unpacking. This also adds another `--aufs` option to
transform aufs special files into overlayfs metadata.
[ Note that `--tar=f` generates lots of temporary files for now which
can impact performance since the original tar stream(s) may be
non-seekable. ]
Signed-off-by: Jingbo Xu <jefflexu@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230714065851.70583-1-jefflexu@linux.alibaba.com
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
|
|
Eric suggests a "binary search + heuristics" way by using the
current libdeflate APIs to generate fixed-sized output DEFLATE streams.
Compared to the previous built-in one, it will generate smaller images
(which is expected since the built-in one is roughly just the original
zlib replacement), yet the total compression time might be amplified a
lot especially if some larger pclusters are used by users compared to
the built-in one.
For example:
$ time mkfs.erofs -zdeflate,9 -C65536 enwik8.z enwik8
real 0m9.559s
user 0m9.453s
sys 0m0.069s
$ time mkfs.erofs -zlibdeflate,9 -C65536 enwik8.libdeflate.9.z enwik8
real 0m50.184s
user 0m50.082s
sys 0m0.074s
$ mkfs/mkfs.erofs -zlibdeflate,6 -C65536 enwik8.libdeflate.6.z enwik8
real 0m23.428s
user 0m23.329s
sys 0m0.067s
37175296 enwik8.libdeflate.6.z
37142528 enwik8.z
36835328 enwik8.libdeflate.9.z
Anyway, let's use the current APIs for users who needs smaller image
sizes for now. Besides, EROFS also supports multiple per-file
algorithms in one image, so it can be used for specific files as well.
Suggested-by: Eric Biggers <ebiggers@kernel.org>
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230710110251.89464-5-hsiangkao@linux.alibaba.com
|
|
This patch adds DEFLATE compression algorithm support to erofs-utils
compression framework.
Note that windowbits (which indicates dictionary size) is recorded in
the on-disk compression configuration. Since some accelerators (e.g.
Intel IAA) don't have enough on-chip memory, compressed data generated
with large windowbits (e.g. > 12 for the IAA accelerator) doesn't seem
to be worked properly on those.
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230710110251.89464-4-hsiangkao@linux.alibaba.com
|
|
This patch adds DEFLATE compression algorithm support to erofsfuse
by using zlib (by default) and libdeflate. libdeflate will be used
instead of zlib if libdeflate is enabled.
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230712235143.10125-1-hsiangkao@linux.alibaba.com
|
|
As Apple documentation written "If you require interoperability with
non-Apple devices, use COMPRESSION_ZLIB. [1]", DEFLATE is a popular
generic-purpose compression algorithm for a quite long time (many
advanced formats like zlib, gzip, zip, png are all based on that),
which is made of LZ77 as well as Huffman coding, fully documented as
RFC1951 [2] and quite easy to understand, implement.
There are several hardware on-market DEFLATE accelerators as well,
such as (s390) DFLTCC, (Intel) IAA/QAT, (HiSilicon) ZIP accelerator,
etc. Therefore, it's useful to support DEFLATE compression in order
to use these for async I/Os and get benefits from these.
Since there is _no fixed-sized output DEFLATE compression appoach_
available in public (fitblk is somewhat ineffective) and the original
zlib is quite slowly developping, let's work out one for our use cases.
Fortunately, it's only less than 1.5kLOC with lazy matching to just
match the full zlib abilities. Besides, near-optimal block splitting
(based on price function) doesn't support since it's no rush to us.
In the future, there might be more built-in optimizers landed to
fulfill our needs even further (especially for other popular algorithms
without native fixed-sized output support). In addition, I'd be quite
happy to see more popular encoders to support native fixed-sized output
compression too.
[1] https://developer.apple.com/documentation/compression/compression_algorithm
[2] https://datatracker.ietf.org/doc/html/rfc1951
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230710110251.89464-2-hsiangkao@linux.alibaba.com
|
|
EROFS Development is currently on the mailing list _only_.
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230713111046.62793-1-hsiangkao@linux.alibaba.com
|
|
Add ERR_CAST macro to prepare for the upcoming tarerofs feature.
Signed-off-by: Jingbo Xu <jefflexu@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230711061240.23662-1-jefflexu@linux.alibaba.com
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
|
|
Keep in sync with kernel commit 24331050a3e6 ("erofs: fix small
compressed files inlining") to avoid corruption due to m_llen > m_plen
for uncompressed pclusters.
Fixes: 41790d24329d ("erofs-utils: validate the extent length for uncompressed pclusters")
Signed-off-by: Yue Hu <huyue2@coolpad.com>
Link: https://lore.kernel.org/r/20230711032508.6892-1-zbestahu@gmail.com
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
|
|
In order to prepare for LZ77 matchfinder. Note that erofs_memcmp2()
is still not quite effective.
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230709182511.96954-2-hsiangkao@linux.alibaba.com
|
|
The motivation is to have standalone (statically linked) erofs binaries
for simple initrd images, that are nevertheless able to (re)create
erofs images with a given UUID.
For this reason a few of libuuid functions have implementations added
directly in erofs-utils.
A header liberofs_uuid.h provides the new functions, which are
always available. A further sideeffect is that code can be simplified
which calls into this functionality.
The uuid_unparse function replacement is always a private
implementation and split into its own file, this further restricts
the (optional) dependency on libuuid only to the erofs-mkfs tool.
Signed-off-by: Norbert Lange <nolange79@gmail.com>
Link: https://lore.kernel.org/r/20230703215803.4476-1-nolange79@gmail.com
[ Gao Xiang: a getrandom() fix since it was added in glibc 2.25. ]
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
|
|
Update the invalid email address.
Signed-off-by: Huang Jianan <jnhuang95@gmail.com>
Reviewed-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230703093531.22832-1-jnhuang95@gmail.com
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
|
|
Currently hard links can't be extracted correctly, let's support it now.
Signed-off-by: Yue Hu <huyue2@coolpad.com>
Fixes: 412c8f908132 ("erofs-utils: fsck: add --extract=X support to extract to path X")
Reviewed-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230628022558.6198-1-zbestahu@gmail.com
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
|
|
If tmpfs is used for /tmp and blob device is not specified, we need to
copy data between two different file systems during mkfs, which is not
supported by the copy_file_range() syscall. In this case, let's give it
a second chance by fallback to __erofs_copy_file_range().
Link: https://lore.kernel.org/r/20230625080819.44502-1-zhaoyifan@sjtu.edu.cn
Fixes: e9afc0408745 ("erofs-utils: introduce copy_file_range")
Fixes: 03cbf7b8f7f7 ("erofs-utils: mkfs: support chunk-based uncompressed files")
Reviewed-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Signed-off-by: Yifan Zhao <zhaoyifan@sjtu.edu.cn>
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
|
|
The on-disk format specifies that share xattr can be addressed by:
xattr offset = xattr_blkaddr * block_size + 4 * shared_xattr_id
That is, the shared_xattr_id is calculated from the xattr offset
(starting from xattr_blkaddr) divided by 4. Make this semantics
explicitly by calculating the divisor from 'sizeof(__le32)'.
It has no logic change.
Signed-off-by: Jingbo Xu <jefflexu@linux.alibaba.com>
Reviewed-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230609090225.91890-1-jefflexu@linux.alibaba.com
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
|
|
Keep in sync with kernel erofs_fs.h of commit 6a318ccd7e08
("erofs: enable long extended attribute name prefixes").
Reviewed-by: Guo Xuenan <guoxuenan@huaweicloud.com>
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230606170144.36902-1-hsiangkao@linux.alibaba.com
|
|
Let's introduce a fuzzer for fsck.erofs by using libFuzzer [1]:
- Clang 6.0+ installed;
- Build with "CC=clang ./configure --enable-fuzzing";
- Set stack size by using `ulimit -s` properly;
- fsck/fuzz_erofsfsck -timeout=40 -max_total_time=600 CORPUS_DIR.
[1] https://www.llvm.org/docs/LibFuzzer.html
[ Gao Xiang: a dedicated test job is also set up at
https://github.com/erofs/erofsnightly/actions/workflows/fuzzers.yml ]
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230605170551.273399-1-xiang@kernel.org
|
|
de_namelen 0 is invalid for now.
Fixes: 564adb0a852b ("erofs-utils: lib: add API to iterate dirs in EROFS")
Reviewed-by: Guo Xuenan <guoxuenan@huaweicloud.com>
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230606075125.75125-1-hsiangkao@linux.alibaba.com
|
|
Currently, we use fsckcfg.extract_path to record the path of file
to be extracted, when the name is too long, it will exceed the
fsckcfg.extract_path[PATH_MAX] array and segmentation fault may
occur.
Test and reproduce with the following script:
``` bash
FSCK=`which fsck.erofs`
MKFS=`which mkfs.erofs`
IN_DIR=./src
$MKFS x.img ${IN_DIR}
get_dst_dir()
{
local len=$1
local perlen=$2
local dst_dir=$(printf 'a%.0s' $(seq 1 $((perlen - 1))))
local n=$((len / ${perlen}))
local lastlen=$((len - perlen * n))
local lastdir=$(printf 'a%.0s' $(seq 1 $lastlen))
local outdir=""
for x in `seq 1 $n`
do
outdir=${outdir}/${dst_dir}
done
[[ -n $lastdir ]] && outdir=${outdir}/${lastdir}
echo ${outdir}
}
for n in `seq 4000 1 5000`
do
dst_dir=$(get_dst_dir $n 255)
echo ${#dst_dir}
OUT_DIR="./${dst_dir}"
rm -rf $(dirname $OUT_DIR) > /dev/null 2>&1
mkdir -p $OUT_DIR
$FSCK --extract=${OUT_DIR} x.img > /dev/null 2>&1
done
```
Fixes: f44043561491 ("erofs-utils: introduce fsck.erofs")
Fixes: b11f84f593f9 ("erofs-utils: fsck: convert to use erofs_iterate_dir()")
Fixes: 412c8f908132 ("erofs-utils: fsck: add --extract=X support to extract to path X")
Signed-off-by: Guo Xuenan <guoxuenan@huawei.com>
Reviewed-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230605125815.2835732-1-guoxuenan@huawei.com
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
|
|
Otherwise, read_count could be overflow and causes unexpected
behaviors.
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230603133130.34364-2-hsiangkao@linux.alibaba.com
|
|
The maximum supported block size is EROFS_MAX_BLOCK_SIZE.
[ Gao Xiang: shifted values can also overflow and need to be fixed. ]
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230603133130.34364-1-hsiangkao@linux.alibaba.com
|
|
Since dedupe feature is also using the same feature bit as fragments.
Fixes: 017f5b402d14 ("erofs-utils: fsck: add a check to packed inode")
Signed-off-by: Yue Hu <huyue2@coolpad.com>
Link: https://lore.kernel.org/r/8e87974be6533d03cff7bf6af222869e7ddba015.1685700307.git.huyue2@coolpad.com
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
|
|
Since dedupe feature is also using the same feature bit as fragments.
Meanwhile, add missing dedupe feature to feature_lists[].
Fixes: a6336feefe37 ("erofs-utils: dump: support fragments")
Signed-off-by: Yue Hu <huyue2@coolpad.com>
Link: https://lore.kernel.org/r/d755ab1c4eaa634f8822b6ce663c1d1a66aae09c.1685700307.git.huyue2@coolpad.com
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
|
|
The variable 'ctx->pclustersize' could be larger than max pclustersize.
Signed-off-by: Noboru Asai <asai@sijam.com>
Reviewed-by: Yue Hu <huyue2@coolpad.com>
Link: https://lore.kernel.org/r/20230602052039.615632-1-asai@sijam.com
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
|
|
Since some crafted EROFS filesystem images could have insane deep
hierarchy (or may form directory loops) which triggers the
PATH_MAX-sized path buffer OR stack overflow.
Actually some crafted images cannot be deemed as real corrupted
images but over-PATH_MAX paths are not something that we'd like to
support for now.
CVE: CVE-2023-33551
Closes: https://nvd.nist.gov/vuln/detail/CVE-2023-33551
Reported-by: Chaoming Yang <lometsj@live.com>
Fixes: f44043561491 ("erofs-utils: introduce fsck.erofs")
Fixes: b11f84f593f9 ("erofs-utils: fsck: convert to use erofs_iterate_dir()")
Fixes: 412c8f908132 ("erofs-utils: fsck: add --extract=X support to extract to path X")
Signeo-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230602055256.18061-1-hsiangkao@linux.alibaba.com
|
|
Since some crafted EROFS filesystem images could have insane large
extents, which causes unexpected bahaviors when extracting data.
Fix it by extracting large extents with a buffer of a reasonable
maximum size limit and reading multiple times instead.
Note that only `--extract` option is impacted.
CVE: CVE-2023-33552
Closes: https://nvd.nist.gov/vuln/detail/CVE-2023-33552
Reported-by: Chaoming Yang <lometsj@live.com>
Fixes: 412c8f908132 ("erofs-utils: fsck: add --extract=X support to extract to path X")
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230602030519.117071-1-hsiangkao@linux.alibaba.com
|
|
Use compact 4B format for large lcluster sizes if possible or fall
back to the non-compact format.
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230601113416.71774-1-hsiangkao@linux.alibaba.com
|
|
Previously PAGE_SIZE was actually unset for most cases so that it was
always hard-coded 4096.
In order to set EROFS_MAX_BLOCK_SIZE correctly, let's detect the real
page size when configuring and get rid of PAGE_SIZE.
Cc: Kelvin Zhang <zhangkelvin@google.com>
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230601145359.103536-1-hsiangkao@linux.alibaba.com
|
|
In crafted erofs image, extract files may lead to fsck.erofs
memory access out of bounds.
Actually, there is already interception in the code, but which only
take effect in debug mode, change it to avoid that.
Signed-off-by: Guo Xuenan <guoxuenan@huawei.com>
Reviewed-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Fixes: 730979c061d5 ("erofs-utils: fuse: add compressed file support")
Link: https://lore.kernel.org/r/20230531072612.2643983-3-guoxuenan@huawei.com
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
|
|
This improves support for non 4K block sizes
Signed-off-by: Kelvin Zhang <zhangkelvin@google.com>
Reviewed-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230530202413.2734743-1-zhangkelvin@google.com
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
|
|
Add the missing cpu_to_le[16|32]() conversion when initializing
erofs_super_block.
Fixes: 0a94653c56b2 ("erofs-utils: introduce mkfs support")
Fixes: 116ac0a254fc ("erofs-utils: introduce shared xattr support")
Fixes: a70f35adc1b0 ("erofs-utils: introduce ondisk compression cfgs")
Signed-off-by: Jingbo Xu <jefflexu@linux.alibaba.com>
Reviewed-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230522024848.89861-1-jefflexu@linux.alibaba.com
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
|
|
Actually it's outdated code compared with kernel
commit b86269f43892 ("erofs: support parsing big pcluster compact indexes")
This will cause fsck.erofs works endlessly on some crafted images.
Reported-by: Chaoming Yang <lometsj@live.com>
Fixes: 418fb683fd96 ("erofs-utils: fuse: support compact indexes for bigpcluster")
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230511141520.32835-1-xiang@kernel.org
|
|
Otherwise, an unexpected segfault will happen since
EROFS_FEATURE_INCOMPAT_FRAGMENTS and
EROFS_FEATURE_INCOMPAT_DEDUPE share the same bit.
Fixes: 18fbf7d12e4f ("erofs-utils: build xattrs upon extra long name prefixes")
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Signed-off-by: Jingbo Xu <jefflexu@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230506035306.2269-1-jefflexu@linux.alibaba.com
|
|
- record device_id in erofs_blobchunk for later tarerofs;
- fix up endianness conversion;
- blob (hash) entries should be released in the blobchunk submodule.
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230421073926.85369-1-hsiangkao@linux.alibaba.com
|
|
Extra long xattr name prefixes are also considered when generating
xattr entries.
Note that the given user-specified extra long xattr name prefixes
are preferred to the pre-defined xattr name prefixes.
Signed-off-by: Jingbo Xu <jefflexu@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230407140902.97275-3-jefflexu@linux.alibaba.com
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
|
|
Besides the predefined xattr name prefixes, introduces long xattr name
prefixes, which work similarly as the predefined name prefixes, except
that they are user specified.
It is especially useful for use cases together with overlayfs like
Composefs model, which introduces diverse xattr values with only a few
common xattr names (trusted.overlay.redirect, trusted.overlay.digest,
and maybe more in the future). That makes the existing predefined
prefixes ineffective in both image size and runtime performance.
When a user specified long xattr name prefix is used, only the trailing
part of the xattr name apart from the long xattr name prefix will be
stored in erofs_xattr_entry.e_name. e_name is empty if the xattr name
matches exactly as the long xattr name prefix. All long xattr prefixes
are stored in the packed or meta inode, which depends if fragments
feature is enabled or not.
For each long xattr name prefix, the on-disk format is kept as the same
as the unique metadata format: ALIGN({__le16 len, data}, 4), where len
represents the total size of struct erofs_xattr_long_prefix, followed
by data of struct erofs_xattr_long_prefix itself.
Each erofs_xattr_long_prefix keeps predefined prefixes (base_index)
and the remaining prefix string without the trailing '\0'.
Two fields are introduced to the on-disk superblock, where
xattr_prefix_count represents the total number of the long xattr name
prefixes recorded, and xattr_prefix_start represents the start offset of
recorded name prefixes in the packed/meta inode divided by 4.
When referring to a long xattr name prefix, the highest bit (bit 7) of
erofs_xattr_entry.e_name_index is set, while the lower bits (bit 0-6)
as a whole represents the index of the referred long name prefix among
all long xattr name prefixes.
Signed-off-by: Jingbo Xu <jefflexu@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230407140902.97275-2-jefflexu@linux.alibaba.com
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
|
|
Later packed_inode will be used in other scenarios other than
compressed fragments. Let's separate packedfile APIs for future use.
Signed-off-by: Jingbo Xu <jefflexu@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230407140902.97275-1-jefflexu@linux.alibaba.com
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
|
|
Skip xattrs with unidentified "system." prefix to avoid ENODATA error.
Such as building AOSP on NFSv4 servers.
Signed-off-by: Weizhao Ouyang <o451686892@gmail.com>
Reviewed-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230420092739.75464-1-o451686892@gmail.com
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
|
|
Some things that gave me pause or were weirdly formatted.
Signed-off-by: Ahelenia Ziemiańska <nabijaczleweli@nabijaczleweli.xyz>
Reviewed-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/c556cbafa7124d5ab5d20979be068ba36a125ed6.1681041325.git.nabijaczleweli@nabijaczleweli.xyz
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
|
|
Some things that gave me pause or were weirdly formatted.
Signed-off-by: Ahelenia Ziemiańska <nabijaczleweli@nabijaczleweli.xyz>
Reviewed-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/nrjslxj7x6fufnvg4qavwm6zy5gues42wbjf23fgxpihsxyrrc@ddkzmz2usmyl
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
|
|
Some things that gave me pause or were weirdly formatted.
Signed-off-by: Ahelenia Ziemiańska <nabijaczleweli@nabijaczleweli.xyz>
Reviewed-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/8490d5f3cbfdd11ee2690b3e642f7cd70ac9f582.1681041325.git.nabijaczleweli@nabijaczleweli.xyz
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
|
|
-z is specified as "z:".
Signed-off-by: Ahelenia Ziemiańska <nabijaczleweli@nabijaczleweli.xyz>
Reviewed-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/df945a64dbab234faec152a2fca75747f724d0c2.1681041325.git.nabijaczleweli@nabijaczleweli.xyz
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
|
|
In file included from rb_tree.c:34:
./rb_tree.h:96:17: warning: a function declaration without a prototype
is deprecated in all versions of C and is treated as a zero-parameter
prototype in C2x, conflicting with a subsequent definition
[-Wdeprecated-non-prototype]
struct rb_iter *rb_iter_init ();
^
rb_tree.c:422:1: note: conflicting prototype is here
rb_iter_init (struct rb_iter *self) {
^
Signed-off-by: Ahelenia Ziemiańska <nabijaczleweli@nabijaczleweli.xyz>
Reviewed-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/d80484200b3ba60127ff3b92e0c7660a2e8726bf.1681041325.git.nabijaczleweli@nabijaczleweli.xyz
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
|
|
Let's introduce next_shared_xattr instead to chain shared xattrs,
and it will also be used for ranged shared xattrs.
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230407133805.60975-3-hsiangkao@linux.alibaba.com
|
|
Let's calculate shared_xattrs_size lazily instead.
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230407133805.60975-2-hsiangkao@linux.alibaba.com
|
|
`nbh->off - bh->off` in erofs_bh_flush_generic_write() is
problematic due to erofs_bdrop(bh, false).
Let's avoid generic erofs_buf_write_bhops instead.
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230407140856.100602-1-hsiangkao@linux.alibaba.com
|
|
The on-disk erofs_xattr_entry.e_name_len is declared as u8, which
implies the maximum length of the xattr name.
Let's also declare xattr_prefix.prefix_len as u8 to keep sync with the
on-disk structure.
Signed-off-by: Jingbo Xu <jefflexu@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230404080224.77577-2-jefflexu@linux.alibaba.com
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
|
|
If inode->nid is less than 0, it should be unassigned as well.
Reported-by: Jingbo Xu <jefflexu@linux.alibaba.com>
Fixes: 21d84349e79a ("erofs-utils: rearrange on-disk metadata")
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230403083819.47453-1-hsiangkao@linux.alibaba.com
|
|
Since images could be very small for smaller block sizes.
Fixes: a4fb8ea13ac7 ("erofs-utils: support arbitrary block sizes")
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230328035456.89831-2-hsiangkao@linux.alibaba.com
|
|
Currently directory blocks will be allocated after inode metadata
space is reserved, but miss to fix tail blocks.
Fixes: 21d84349e79a ("erofs-utils: rearrange on-disk metadata")
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230328035456.89831-1-hsiangkao@linux.alibaba.com
|
|
In the past, errors will be returned when reading post-EOD data.
Let's align this with the generic EOD behavior (zero-filling) instead.
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230327025737.62488-1-hsiangkao@linux.alibaba.com
|
|
It'd better to split directory data out although directory data is
still not lazy written since it tends to put directory data next to
the corresponding inode metadata.
Laterly, aligned directory data could be written in a batch way so
that inodes can be more compact.
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230324082749.1915-1-hsiangkao@linux.alibaba.com
|
|
i_mode should be used instead for some old random fs.
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230319151417.20432-1-hsiangkao@linux.alibaba.com
|
|
- Use BFS instead of DFS;
- Regular data is separated from metadata and dir data.
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230317085045.16263-1-hsiangkao@linux.alibaba.com
|
|
- Add a new command option for mkfs.erofs;
- fuse/dump/fsck supports arbitrary block sizes (uncompressed files).
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230314062121.115020-4-hsiangkao@linux.alibaba.com
|
|
Now block sizes can be less than PAGE_SIZE.
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230314074251.80425-1-hsiangkao@linux.alibaba.com
|
|
Will be used for subpage blocksize support.
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230314062121.115020-2-hsiangkao@linux.alibaba.com
|
|
Laterly, erofs-utils will support sub-page block sizes and
an arbitrary block size can be given at any position of the
command line.
Therefore, chunk/pcluster sizes needs to be validated in the end.
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230314062121.115020-1-hsiangkao@linux.alibaba.com
|
|
Trivial updates.
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230315082523.10314-1-hsiangkao@linux.alibaba.com
|
|
Signed-off-by: Gao Xiang <xiang@kernel.org>
|
|
Users can try this with the detailed instructions.
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230311041245.27006-1-hsiangkao@linux.alibaba.com
|
|
Add some words about userspace fragment cache as a TODO.
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230310091601.97930-2-hsiangkao@linux.alibaba.com
|
|
100 ~ 109 are now valid for LZMA extreme compression.
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230310091601.97930-1-hsiangkao@linux.alibaba.com
|
|
EOPNOTSUPP could be reported by llistxattr() and mkfs.erofs could
fail out due to the following error:
<E> erofs: llistxattr to get the size of names for xxxxx failed
<E> erofs: Could not format the device : [Error 95] Operation not supported
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230309112630.74230-3-hsiangkao@linux.alibaba.com
|
|
Refine README as well as add docs/INSTALL.md and docs/PERFORMANCE.md.
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230309112630.74230-2-hsiangkao@linux.alibaba.com
|
|
Match in words instead of byte granularity.
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230309112630.74230-1-hsiangkao@linux.alibaba.com
|
|
In addition to just compare the in-memory data.
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230304195812.120063-5-hsiangkao@linux.alibaba.com
|
|
Directly read/write if mmap fails.
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230304195812.120063-4-hsiangkao@linux.alibaba.com
|
|
To avoid unnecessary memory overhead.
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230304195812.120063-3-hsiangkao@linux.alibaba.com
|
|
LibTomCrypt was released into public domain so that we don't need
to worry about license issues against Apache-2.0 later.
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230304195812.120063-2-hsiangkao@linux.alibaba.com
|
|
delta[1] should not exceed (1 << lclusterbits) - 1, usually 4095
blocks, which means a single compressed extent which isn't less
than 16MiB could trigger this issue (it very rare happens).
Fixes: 76b822726ff8 ("erofs-utils: introduce compacted compression indexes")
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230304195812.120063-1-hsiangkao@linux.alibaba.com
|
|
Keep in sync with the kernel commit c505feba4c0d ("erofs: validate the
extent length for uncompressed pclusters"), so that we can catch the
issue as well in fuse.
Signed-off-by: Yue Hu <huyue2@coolpad.com>
Link: https://lore.kernel.org/r/20230303080743.25713-1-zbestahu@gmail.com
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
|
|
It's almost the same as `-Efragments` option, except that will
explicitly pack the whole data into the special inode instead.
Tested-by: Yue Hu <huyue2@coolpad.com>
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230228185459.117762-3-hsiangkao@linux.alibaba.com
|
|
The decoded lengths of uncompressed pclusters should be
strictly no more than encoded lengths.
Fixes: 9fa9b017f773 ("erofs-utils: mkfs: support fragments")
Reviewed-by: Yue Hu <huyue2@coolpad.com>
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230228185459.117762-2-hsiangkao@linux.alibaba.com
|
|
Also refine the longest detection.
Fixes: 990c7e383795 ("erofs-utils: mkfs: support fragment deduplication")
Reviewed-by: Yue Hu <huyue2@coolpad.com>
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230228185459.117762-1-hsiangkao@linux.alibaba.com
|
|
The ztailpacking feature has been merged for a year, it has been mostly
stable now. Let's drop such warning now.
Signed-off-by: Yue Hu <huyue2@coolpad.com>
Link: https://lore.kernel.org/r/20230227095311.13033-1-zbestahu@gmail.com
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
|
|
Complete the manpage for global compressed data deduplication feature.
Signed-off-by: Yue Hu <huyue2@coolpad.com>
Link: https://lore.kernel.org/r/20230225142707.28408-1-zbestahu@163.com
[ Gao Xiang: refine manpage description. ]
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
|
|
Complete the manpage for fragments feature.
Signed-off-by: Yue Hu <huyue2@coolpad.com>
Link: https://lore.kernel.org/r/20230225125133.26165-1-zbestahu@163.com
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
|
|
It's used for multiple algorithms selftest.
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230222191148.112677-3-hsiangkao@linux.alibaba.com
|
|
Some binaries in an image may be just for archival purposes only
so that the runtime performance is not the top priority at all.
Therefore, it'd better to use another algorithm with higher
compression ratios and (even) pack the whole file into the packed
inode entirely (will enable this way later.)
In order to use alternative algorithms, just specify two or more
compressing configurations together separated by ':' like below:
-zlzma:lz4hc,12:lzma,9 -C32768
Although mkfs still choose the first one by default, you could try
to write a compress-hints file like below:
4096 1 .*\.so$
32768 2 .*\.txt$
4096 sbin/.*$
16384 0 .*
So that ".so" files will use "lz4hc,12" compression with 4k pclusters,
".txt" files will use "lzma,9" compression with 32k pclusters, files
under "/sbin" will use the default "lzma" compression with 4k plusters
and other files will use "lzma" compression with 16k pclusters.
Note that the largest pcluster size should be specified with the "-C"
option, otherwise all larger pclusters will be limited.
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230222191148.112677-2-hsiangkao@linux.alibaba.com
|
|
Identify lines starting with '#' as a comment.
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230222191148.112677-1-hsiangkao@linux.alibaba.com
|
|
commit c47df5aa2d16 ("erofs-utils: fuse: introduce xattr support")
introduced two new references.
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230222190814.102585-1-hsiangkao@linux.alibaba.com
|
|
It's somewhat strange to skip the `lost+found` directory.
Fixes: a17497f0844a ("erofs-utils: introduce inode operations")
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230209033524.38234-1-hsiangkao@linux.alibaba.com
|
|
Scan holes for chunk-based inodes if either --chunksize=# or
-Ededupe (without compression) is specified so that sparse files
can be made, which has already been supported since Linux 5.15.
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230131051454.47719-1-hsiangkao@linux.alibaba.com
|
|
Otherwise it will report:
<I> erofs: total metadata: 982 blocks
<E> erofs: Could not format the device : [Error 5] Input/output error
Fixes: 03cbf7b8f7f7 ("erofs-utils: mkfs: support chunk-based uncompressed files")
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230131094808.67525-1-hsiangkao@linux.alibaba.com
|
|
Reported by the latest nightly build:
https://github.com/erofs/erofsnightly/actions/runs/4050082667/jobs/6969376259
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230131084805.2494-1-hsiangkao@linux.alibaba.com
|
|
Add a check to packed inode for fsck.erofs.
Signed-off-by: Yue Hu <huyue2@coolpad.com>
Link: https://lore.kernel.org/r/ce29198d32f284c19589877c4c9e4e1588ad77b3.1673401718.git.huyue2@coolpad.com
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
|
|
Diretly call {z_}erofs_read_one_data() to avoid duplicated code.
Accordingly, fragment and partial-referenced plusters are also supported
after this change.
Signed-off-by: Yue Hu <huyue2@coolpad.com>
Link: https://lore.kernel.org/r/115e61fc9c2d34cab6d3dd78383ac57c94a491fc.1673401718.git.huyue2@coolpad.com
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
|
|
Export parts of erofs_pread() to avoid duplicated code in
erofs_verify_inode_data(). Let's make two helpers for this.
Signed-off-by: Yue Hu <huyue2@coolpad.com>
Link: https://lore.kernel.org/r/ff560da9c798b2ca1f1a663a000501486d865487.1673401718.git.huyue2@coolpad.com
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
|
|
Add compressed fragments support for dump feature.
Signed-off-by: Yue Hu <huyue2@coolpad.com>
Link: https://lore.kernel.org/r/20230105021343.23419-1-zbestahu@gmail.com
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
|
|
The statistic update of occupied_size is the same as original size.
Also, correct naming to update_file_size_statistics().
Signed-off-by: Yue Hu <huyue2@coolpad.com>
Link: https://lore.kernel.org/r/0e4797ef5f22ac3c2134aa4b005c489c233d2eec.1671443064.git.huyue2@coolpad.com
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
|
|
inode.extent_isize is only meaningful during mkfs for now.
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20230104073019.90530-1-hsiangkao@linux.alibaba.com
|
|
Packed inode may be uncompressed as well due to no space.
Signed-off-by: Yue Hu <huyue2@coolpad.com>
Link: https://lore.kernel.org/r/20221224093851.10095-1-zbestahu@gmail.com
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
|
|
Packed inode is composed of fragments which have already been
deduplicated before.
Signed-off-by: Yue Hu <huyue2@coolpad.com>
Link: https://lore.kernel.org/r/20221209035337.26998-1-zbestahu@gmail.com
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
|
|
The return value of ftell() is a long int, use ftello{64} instead. Also,
need to return at once if it fails. Meanwhile, use lseek64 for large
file position as well to avoid this error if we have it.
Signed-off-by: Yue Hu <huyue2@coolpad.com>
Link: https://lore.kernel.org/r/20221213091746.16683-1-zbestahu@gmail.com
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
|
|
Previously, there's no fragment deduplication when this feature is
introduced. Let's support it now.
Fragments are deduplicated before compression, so that duplicated
parts will not be written into the packed inode again.
Dataset: linux 5.10 + 5.10.50 + 5.10.100
Compression algorithm: lz4hc,12
Additional options: -T0 --force-uid=1000 --force-gid=1000
(in order to force 32-byte inodes to match squashfs)
4k pcluster + fragment + dedupe 397168640
8k pcluster + fragment + dedupe 364224512
16k pcluster + fragment + dedupe 341921792
32k pcluster + fragment + dedupe 328298496
64k pcluster + fragment + dedupe 324694016
128k pcluster + fragment + dedupe 323674112
256k pcluster + fragment + dedupe 322011136
squashfs-tools 4.5.1 test results (which uses level 12 by default
for lz4hc):
16k block 428785664
32k block 382894080
64k block 350179328
128k block 327073792
128k block + noI 334327808
256k block 315441152
256k block + noI 322707456
1m block 307425280
1m block + noI 314712064
If the compressed block size is large, the image size of squashfs
will benefit from its simple _unseekable_ data block indexes:
location = block_start
for i = 0; i < index; i++
location += block_sizes[i] & 0x00FFFFFF
and its fragment lookup table. EROFS can improve such cases later.
Suggested-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Signed-off-by: Yue Hu <huyue2@coolpad.com>
Link: https://lore.kernel.org/r/20221208084859.24190-1-zbestahu@gmail.com
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
|
|
Support uncompressed data layout with on-disk interlaced offset in
compression mode for fsck.erofs.
Signed-off-by: Yue Hu <huyue2@coolpad.com>
Link: https://lore.kernel.org/r/20221221084842.28196-1-zbestahu@gmail.com
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
|
|
erofs depends on the consistent use of a 64bit offset
type, force downstreams to use transparent LFS (_FILE_OFFSET_BITS=64),
so that it becomes impossible for them to use 32bit interfaces.
Signed-off-by: Khem Raj <raj.khem@gmail.com>
Link: https://lore.kernel.org/r/20221215085842.130804-3-raj.khem@gmail.com
[ Gao Xiang: subject title fixup. ]
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
|
|
Signed-off-by: Khem Raj <raj.khem@gmail.com>
Link: https://lore.kernel.org/r/20221215085842.130804-2-raj.khem@gmail.com
[ Gao Xiang: subject title fixup and code rebase. ]
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
|
|
Pass -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 via CFLAGS
this enabled large file support on 32bit architectures
Signed-off-by: Khem Raj <raj.khem@gmail.com>
Link: https://lore.kernel.org/r/20221215085842.130804-1-raj.khem@gmail.com
[ Gao Xiang: subject title fixup. ]
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
|
|
Need to check if we got an error. Also, make erofs_write_file() static.
Signed-off-by: Yue Hu <huyue2@coolpad.com>
Link: https://lore.kernel.org/r/20221213092317.17236-1-zbestahu@gmail.com
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
|
|
CBLKCNT needs to be stored for big pcluster dedupe. Otherwise,
the decompression could fail due to incomplete compressed data.
Reported-by: Yue Hu <huyue2@coolpad.com>
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Tested-by: Yue Hu <huyue2@coolpad.com>
Link: https://lore.kernel.org/r/20221123023034.123095-1-hsiangkao@linux.alibaba.com
|
|
Remove duplicate ".txt" file type distribution.
Signed-off-by: Weizhao Ouyang <o451686892@gmail.com>
Link: https://lore.kernel.org/r/20221120040648.706871-1-o451686892@gmail.com
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
|
|
Keep in sync with the kernel commit 0d53d2e882f9 ("erofs: avoid the
potentially wrong m_plen for big pcluster").
Signed-off-by: Yue Hu <huyue2@coolpad.com>
Reviewed-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20221031022653.14981-1-zbestahu@gmail.com
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
|
|
When using `fsck.erofs` to extract some image have a very large file
inside, for example 2G, my fsck.erofs report some thing like this:
<E> erofs_io: Failed to read data from device - erofs.image:[4096,
2147483648].
<E> erofs: failed to read data of m_pa 4096, m_plen 2147483648 @ nid 40: -17
<E> erofs: Failed to extract filesystem
You can use this script to reproduce this issue.
mkdir tmp extract
dd if=/dev/urandom of=tmp/big_file bs=1M count=2048
mkfs.erofs erofs.image tmp
fsck.erofs erofs.image --extract=extract
I found that dev_open will failed if we can not get all data we want
with one pread call.
I write this little patch try to fix this issue.
Signed-off-by: Chen Linxuan <chenlinxuan@uniontech.com>
Reviewed-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20221021064332.357316-1-chenlinxuan@uniontech.com
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
|
|
After this patch, "-E dedupe" can now support chunk-based data
deduplication with block-sized chunks if compression is off.
Link: https://lore.kernel.org/r/20221017031736.90542-1-hsiangkao@linux.alibaba.com
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
|
|
We should do nothing in dedupe inserting when it's not configured.
Signed-off-by: Yue Hu <huyue2@coolpad.com>
Reviewed-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20221013040011.31944-1-zbestahu@gmail.com
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
|
|
The on-disk erofs_super_block has the volume_name field. On the other
hand, mkfs.erofs doesn't support setting volume label.
This patch adds volume-label setting support to mkfs.erofs.
Option keyword is similar to mke2fs.
Usage:
mkfs.erofs -L volume-label image-fn dir
Signed-off-by: Naoto Yamaguchi <naoto.yamaguchi@aisin.co.jp>
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/YzxzRTg5oUeOCMr+@debian
|
|
This patch introduces global compressed data deduplication to
reuse potential prefixes for each pcluster.
It also uses rolling hashing and tries to shorten the previous
compressed extent in order to explore more possibilities for
data deduplication.
Co-developed-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Signed-off-by: Ziyang Zhang <ZiyangZhang@linux.alibaba.com>
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20220926152511.94832-9-hsiangkao@linux.alibaba.com
|
|
Due to deduplication for compressed data, pclusters can be partially
referenced with their prefixes.
Decompression algorithms should know that in advance, otherwise they
will fail out unexpectedly.
Keep in sync with the latest kernel
commit 5c2a64252c5d ("erofs: introduce partial-referenced pclusters")
Signed-off-by: Ziyang Zhang <ZiyangZhang@linux.alibaba.com>
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20220926152511.94832-8-hsiangkao@linux.alibaba.com
|
|
Introduce a simple rb-tree implementation in order to store the
hash map for deduplication.
Signed-off-by: Ziyang Zhang <ZiyangZhang@linux.alibaba.com>
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20220926152511.94832-7-hsiangkao@linux.alibaba.com
|
|
This approach can merge tail pclusters or the whole files into a
special inode in order to achieve greater compression ratios. Also,
an option of pcluster size is provided for different compression
requirements.
Enable interlaced uncompressed data layout for compressed files
at the same time as well.
Signed-off-by: Yue Hu <huyue2@coolpad.com>
[ Gao Xiang: duplicated fragment data can still exist in the packed
inode and fragment deduplication needs be to implemented later. ]
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20220926152511.94832-6-hsiangkao@linux.alibaba.com
|
|
Interlaced uncompressed data can minimize unnecessary data processing
by using in-place I/O.
However it cannot be used together with deduplication.
Signed-off-by: Yue Hu <huyue2@coolpad.com>
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20220926152511.94832-5-hsiangkao@linux.alibaba.com
|
|
In order to introduce deduplicatation for compressed pclusters.
A lookahead extent is recorded so that the following deduplication
process can adjust the previous extent on demand.
Also, in the future, it can be used for parallel compression
as well.
Signed-off-by: Ziyang Zhang <ZiyangZhang@linux.alibaba.com>
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20220926152511.94832-4-hsiangkao@linux.alibaba.com
|
|
Add fragments support for erofsfuse.
Also, this keeps in sync with the latest kernel
commit b15b2e307c3a ("erofs: support on-disk compressed fragments data")
Signed-off-by: Yue Hu <huyue2@coolpad.com>
Link: https://lore.kernel.org/r/20220930130842.6728-1-zbestahu@163.com
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
|
