5. FUSE Caches

5.1. Introduction

This document summarises the different types of caches used in FUSE. For each cache type, it documents the rules to insert data into it. It also documents the rules for validating and invalidating data in the cache.

5.3. Attributes caching

Inode attributes may be obtained from user-space by different FUSE operations. For example, FUSE_LOOKUP, FUSE_GETATTR, and also several other operations that create file system objects (e.g. FUSE_MKDIR). These attributes obtained from user-space are cached by the kernel. They have, however, a timeout associated and once it expires, they are invalidated. The next time the attributes are needed, a request (FUSE_GETATTR) will be sent to the FUSE server.

The FUSE_GETATTR request can be sent to user-space in three different scenarios:

  1. if the attributes for the inode aren’t yet available in the kernel;

  2. if they are not valid any more (timed-out, or have been invalidated), or

  3. if there is an explicit request for forcing the request to be sent (for example, by using the AT_STATX_FORCE_SYNC flag in statx).

Regarding the attributes invalidation, they may happen in several occasions. For example, upon a user-space request for invalidation, through FUSE_NOTIFY_INVAL_INODE, FUSE_NOTIFY_INVAL_ENTRY, or FUSE_NOTIFY_DELETE requests.

FUSE uses fine-grained invalidation masks rather than invalidating all attributes at once. The principle is that each operation only invalidates the specific attributes that the operation could have changed on the server. The masks used are:

  • STATX_ATIME - after reads and readlink, since the server may update access time

  • STATX_CTIME - after xattr changes (including ACL set/remove) and rename

  • STATX_BLOCKS - after a successful flush with writeback cache, since the server’s block count may differ from the local one

  • FUSE_STATX_MODIFY (STATX_MTIME | STATX_CTIME | STATX_BLOCKS) - after writeback completion (without writeback cache), since the server may have updated modification metadata

  • FUSE_STATX_MODSIZE (FUSE_STATX_MODIFY | STATX_SIZE) - after writes, truncate-on-open, and fallocate, since the server’s size and modification metadata may have changed

  • FUSE_STATX_MODDIR (FUSE_STATX_MODSIZE | STATX_NLINK) - after directory modifications (create, unlink, mkdir, rmdir, rename), since the server may have updated the directory’s size, timestamps, and link count

  • STATX_BASIC_STATS - as a full invalidation, used for server-initiated invalidation (FUSE_NOTIFY_INVAL_INODE), interrupted setattr, and interrupted link

The full set of invalidation points can be found by searching for fuse_invalidate_attr_mask() in the FUSE source.

5.4. ACL caching

FUSE has allowed the usage of POSIX Access Control Lists (ACLs) for a long time, as they can be set and accessed simply as extended attributes. However, it was only with the introduction of the FUSE_POSIX_ACL flag that ACLs started to be fully supported. Without this flag being set during the FUSE_INIT negotiation, ACLs can still be set, but the VFS won’t use them for performing permission checks - that would be the user-space server’s responsibility.

Without setting FUSE_POSIX_ACL, ACLs will not be cached by the kernel. In this case, new inodes i_acl and i_default_acl fields will be set to ACL_DONT_CACHE.

If the FUSE_POSIX_ACL flag is set, when an inode ACL is accessed VFS will first check if it’s already cached. If it is not, FUSE ->get_acl() operation (fuse_get_acl()) is called, which will eventually send a user-space request. Future accesses to this inode ACL will use the cached data.

Setting an ACL in an inode will also result in sending a request to the FUSE server for setting it. But this operation won’t immediately cache the ACL -- it will only be cached after it is accessed again and requested from user-space.

ACLs will be removed from the cache in the following situations:

  • When setting an ACL in an inode (and the FUSE_POSIX_ACL flag is set), previously cached ACLs for this inode will be invalidated.

  • When invalidating an inode through the FUSE_NOTIFY_INVAL_INODE operation.

  • After setting an inode attribute (i.e. operation FUSE_SETATTR is sent to user-space), the user-space server may have also updated the ACLs. Thus, any cached ACLs for this inode are also invalidated.

  • Whenever attributes are refreshed from the server. For example, when revalidating a dentry (->d_revalidate()), or when updating a dentry while processing a FUSE_READDIRPLUS.

  • In general, when there is the need to send a FUSE_STATX or FUSE_GETATTR to user-space (e.g. when attributes expired).

5.5. readdir caching

When opening a directory a FUSE_OPENDIR will be sent to the FUSE server, and server will be responsible for setting the open flags related with caching, namely FOPEN_KEEP_CACHE and FOPEN_CACHE_DIR.

FOPEN_CACHE_DIR determines if readdir results of this open will be cached and if readdir cache will be used to return readdir results during the current open. If FOPEN_KEEP_CACHE is set, any readdir cache from previous opens is preserved when the directory is opened. Otherwise, the old readdir cache is invalidated on open.

The readdir cache will also expire and reset if the inode’s mtime or iversion don’t match the cached values, or if the FUSE connection epoch doesn’t match the cache epoch.

5.6. dentry caching

FUSE keeps track of all its file systems dentries in a set of rbtrees. These trees will keep the dentries sorted by their expiry time, so that it is easy to quickly invalidate expired dentries.

This set of rbtrees is protected through hashed locks to reduce contention while doing trees traversal.

When FUSE is initialised, an array of struct dentry_bucket is initialised with a predefined (FUSE_HASH_SIZE) number of elements. Each element of this array (dentry_hash) holds an rbtree (initially empty) of dentries. Every time a new dentry is created it will be added to one of the elements of this array’s rbtree, where it will be sorted by its expiry time (timeout). The selection of the array element where a dentry is inserted is done through the hash_ptr() hash function.

A dentry expiry time will be set to a valid value (provided by the FUSE server) in several occasions:

  • When a new file system object is created (e.g. FUSE_CREATE, FUSE_MKDIR, etc).

  • When a lookup is sucessfully performed.

  • When a dentry is sucessfully revalidated (->d_revalidate()).

A dentry expiry time can be set to 0, indicating that it is stale. This means that a FUSE_LOOKUP will be sent to user-space then next time it needs to be looked up. It can be set to 0 in several occasions:

  • When the user-space FUSE server explicitly requests a dentry to be invalidated (FUSE_NOTIFY_INVAL_ENTRY).

  • When a file system object is being deleted (e.g. FUSE_UNLINK) or renamed.

  • When a filesystem object is being created and fails with -EEXIST.

  • When a lookup fails with -ENOENT.

Cleaning up expired dentries can happen in different ways. One scenario is when the VFS requests a revalidation. If the timeout value is 0 (expired), a new lookup is sent to user-space. If this lookup results in -ENOENT, the dentry is invalidated. If the lookup is successful, the dentry is re-validated and the expiry time will be updated. (There’s a special case where the lookup is successful but the nodeid returned is different from the expected one; in this case a FUSE_FORGET needs to be sent to user-space and the dentry is invalidated.)

Another way of cleaning-up expired dentries is through a mechanism that can be enabled through the inval_wq FUSE module parameter. Setting this parameter to a value >= 5 will create a workqueue that will periodically walk through all the dentries rbtrees in the dentry_hash array and invalidate those dentries that have already expired. Once all the rbtrees have been checked, the workqueue reschedules itself to run again after inval_wq seconds. By default, this workqueue is disabled (inval_wq is set to 0).

There is yet another mechanism that allows to clean all the cached dentries for a mounted file system, which is by incrementing the ‘epoch’ (FUSE_NOTIFY_INC_EPOCH). Each FUSE connection will have its epoch value set when it is created, and every new dentry have its ->d_time set to the current connection epoch value. By incrementing the connection epoch value, all dentries that go through ->d_revalidate() will be automatically invalidated because their ->d_time is be smaller than the connection epoch. If inval_wq is set, the workqueue to invalidate the dentries will be triggered immediately after the epoch is incremented.