Page Pool API¶
The page_pool allocator is optimized for recycling page or page fragment used by skb packet and xdp frame.
Basic use involves replacing and alloc_pages()
calls with page_pool_alloc(),
which allocate memory with or without page splitting depending on the
requested memory size.
If the driver knows that it always requires full pages or its allocations are always smaller than half a page, it can use one of the more specific API calls:
1. page_pool_alloc_pages(): allocate memory without page splitting when driver knows that the memory it need is always bigger than half of the page allocated from page pool. There is no cache line dirtying for 'struct page' when a page is recycled back to the page pool.
2. page_pool_alloc_frag(): allocate memory with page splitting when driver knows that the memory it need is always smaller than or equal to half of the page allocated from page pool. Page splitting enables memory saving and thus avoids TLB/cache miss for data access, but there also is some cost to implement page splitting, mainly some cache line dirtying/bouncing for 'struct page' and atomic operation for page->pp_frag_count.
The API keeps track of in-flight pages, in order to let API users know when
it is safe to free a page_pool object, the API users must call
page_pool_put_page()
or page_pool_free_va()
to free the page_pool object, or
attach the page_pool object to a page_pool-aware object like skbs marked with
skb_mark_for_recycle().
page_pool_put_page()
may be called multi times on the same page if a page is
split into multi fragments. For the last fragment, it will either recycle the
page, or in case of page->_refcount > 1, it will release the DMA mapping and
in-flight state accounting.
dma_sync_single_range_for_device() is only called for the last fragment when
page_pool is created with PP_FLAG_DMA_SYNC_DEV flag, so it depends on the
last freed fragment to do the sync_for_device operation for all fragments in
the same page when a page is split, the API user must setup pool->p.max_len
and pool->p.offset correctly and ensure that page_pool_put_page()
is called
with dma_sync_size being -1 for fragment API.
Architecture overview¶
+------------------+
| Driver |
+------------------+
^
|
|
|
v
+--------------------------------------------+
| request memory |
+--------------------------------------------+
^ ^
| |
| Pool empty | Pool has entries
| |
v v
+-----------------------+ +------------------------+
| alloc (and map) pages | | get page from cache |
+-----------------------+ +------------------------+
^ ^
| |
| cache available | No entries, refill
| | from ptr-ring
| |
v v
+-----------------+ +------------------+
| Fast cache | | ptr-ring cache |
+-----------------+ +------------------+
API interface¶
The number of pools created must match the number of hardware queues unless hardware restrictions make that impossible. This would otherwise beat the purpose of page pool, which is allocate pages fast from cache without locking. This lockless guarantee naturally comes from running under a NAPI softirq. The protection doesn't strictly have to be NAPI, any guarantee that allocating a page will cause no race conditions is enough.
-
struct page_pool *page_pool_create(const struct page_pool_params *params)¶
create a page pool.
Parameters
const struct page_pool_params *params
parameters, see
struct page_pool_params
-
struct page_pool_params¶
page pool parameters
Definition:
struct page_pool_params {
unsigned int flags;
unsigned int order;
unsigned int pool_size;
int nid;
struct device *dev;
struct napi_struct *napi;
enum dma_data_direction dma_dir;
unsigned int max_len;
unsigned int offset;
};
Members
flags
PP_FLAG_DMA_MAP, PP_FLAG_DMA_SYNC_DEV
order
2^order pages on allocation
pool_size
size of the ptr_ring
nid
NUMA node id to allocate from pages from
dev
device, for DMA pre-mapping purposes
napi
NAPI which is the sole consumer of pages, otherwise NULL
dma_dir
DMA mapping direction
max_len
max DMA sync memory size for PP_FLAG_DMA_SYNC_DEV
offset
DMA sync address offset for PP_FLAG_DMA_SYNC_DEV
-
struct page *page_pool_dev_alloc_pages(struct page_pool *pool)¶
allocate a page.
Parameters
struct page_pool *pool
pool from which to allocate
Description
Get a page from the page allocator or page_pool caches.
-
struct page *page_pool_dev_alloc_frag(struct page_pool *pool, unsigned int *offset, unsigned int size)¶
allocate a page fragment.
Parameters
struct page_pool *pool
pool from which to allocate
unsigned int *offset
offset to the allocated page
unsigned int size
requested size
Description
Get a page fragment from the page allocator or page_pool caches.
Return
Return allocated page fragment, otherwise return NULL.
-
struct page *page_pool_dev_alloc(struct page_pool *pool, unsigned int *offset, unsigned int *size)¶
allocate a page or a page fragment.
Parameters
struct page_pool *pool
pool from which to allocate
unsigned int *offset
offset to the allocated page
unsigned int *size
in as the requested size, out as the allocated size
Description
Get a page or a page fragment from the page allocator or page_pool caches depending on the requested size in order to allocate memory with least memory utilization and performance penalty.
Return
Return allocated page or page fragment, otherwise return NULL.
-
void *page_pool_dev_alloc_va(struct page_pool *pool, unsigned int *size)¶
allocate a page or a page fragment and return its va.
Parameters
struct page_pool *pool
pool from which to allocate
unsigned int *size
in as the requested size, out as the allocated size
Description
This is just a thin wrapper around the page_pool_alloc() API, and it returns va of the allocated page or page fragment.
Return
Return the va for the allocated page or page fragment, otherwise return NULL.
-
enum dma_data_direction page_pool_get_dma_dir(struct page_pool *pool)¶
Retrieve the stored DMA direction.
Parameters
struct page_pool *pool
pool from which page was allocated
Description
Get the stored dma direction. A driver might decide to store this locally and avoid the extra cache line from page_pool to determine the direction.
-
void page_pool_put_page(struct page_pool *pool, struct page *page, unsigned int dma_sync_size, bool allow_direct)¶
release a reference to a page pool page
Parameters
struct page_pool *pool
pool from which page was allocated
struct page *page
page to release a reference on
unsigned int dma_sync_size
how much of the page may have been touched by the device
bool allow_direct
released by the consumer, allow lockless caching
Description
The outcome of this depends on the page refcnt. If the driver bumps the refcnt > 1 this will unmap the page. If the page refcnt is 1 the allocator owns the page and will try to recycle it in one of the pool caches. If PP_FLAG_DMA_SYNC_DEV is set, the page will be synced for_device using dma_sync_single_range_for_device().
-
void page_pool_put_full_page(struct page_pool *pool, struct page *page, bool allow_direct)¶
release a reference on a page pool page
Parameters
struct page_pool *pool
pool from which page was allocated
struct page *page
page to release a reference on
bool allow_direct
released by the consumer, allow lockless caching
Description
Similar to page_pool_put_page()
, but will DMA sync the entire memory area
as configured in page_pool_params.max_len
.
-
void page_pool_recycle_direct(struct page_pool *pool, struct page *page)¶
release a reference on a page pool page
Parameters
struct page_pool *pool
pool from which page was allocated
struct page *page
page to release a reference on
Description
Similar to page_pool_put_full_page()
but caller must guarantee safe context
(e.g NAPI), since it will recycle the page directly into the pool fast cache.
-
void page_pool_free_va(struct page_pool *pool, void *va, bool allow_direct)¶
free a va into the page_pool
Parameters
struct page_pool *pool
pool from which va was allocated
void *va
va to be freed
bool allow_direct
freed by the consumer, allow lockless caching
Description
Free a va allocated from page_pool_allo_va().
Parameters
struct page *page
page allocated from a page pool
Description
Fetch the DMA address of the page. The page pool to which the page belongs must had been created with PP_FLAG_DMA_MAP.
-
bool page_pool_get_stats(struct page_pool *pool, struct page_pool_stats *stats)¶
fetch page pool stats
Parameters
struct page_pool *pool
pool from which page was allocated
struct page_pool_stats *stats
struct page_pool_stats
to fill in
Description
Retrieve statistics about the page_pool. This API is only available
if the kernel has been configured with CONFIG_PAGE_POOL_STATS=y
.
A pointer to a caller allocated struct page_pool_stats
structure
is passed to this API which is filled in. The caller can then report
those stats to the user (perhaps via ethtool, debugfs, etc.).
-
void page_pool_put_page_bulk(struct page_pool *pool, void **data, int count)¶
release references on multiple pages
Parameters
struct page_pool *pool
pool from which pages were allocated
void **data
array holding page pointers
int count
number of pages in data
Description
Tries to refill a number of pages into the ptr_ring cache holding ptr_ring
producer lock. If the ptr_ring is full, page_pool_put_page_bulk()
will release leftover pages to the page allocator.
page_pool_put_page_bulk()
is suitable to be run inside the driver NAPI tx
completion loop for the XDP_REDIRECT use case.
Please note the caller must not use data area after running
page_pool_put_page_bulk()
, as this function overwrites it.
DMA sync¶
Driver is always responsible for syncing the pages for the CPU.
Drivers may choose to take care of syncing for the device as well
or set the PP_FLAG_DMA_SYNC_DEV
flag to request that pages
allocated from the page pool are already synced for the device.
If PP_FLAG_DMA_SYNC_DEV
is set, the driver must inform the core what portion
of the buffer has to be synced. This allows the core to avoid syncing the entire
page when the drivers knows that the device only accessed a portion of the page.
Most drivers will reserve headroom in front of the frame. This part
of the buffer is not touched by the device, so to avoid syncing
it drivers can set the offset
field in struct page_pool_params
appropriately.
For pages recycled on the XDP xmit and skb paths the page pool will
use the max_len
member of struct page_pool_params
to decide how
much of the page needs to be synced (starting at offset
).
When directly freeing pages in the driver (page_pool_put_page()
)
the dma_sync_size
argument specifies how much of the buffer needs
to be synced.
If in doubt set offset
to 0, max_len
to PAGE_SIZE
and
pass -1 as dma_sync_size
. That combination of arguments is always
correct.
Note that the syncing parameters are for the entire page.
This is important to remember when using fragments (PP_FLAG_PAGE_FRAG
),
where allocated buffers may be smaller than a full page.
Unless the driver author really understands page pool internals
it's recommended to always use offset = 0
, max_len = PAGE_SIZE
with fragmented page pools.
Stats API and structures¶
If the kernel is configured with CONFIG_PAGE_POOL_STATS=y
, the API
page_pool_get_stats()
and structures described below are available.
It takes a pointer to a struct page_pool
and a pointer to a struct
page_pool_stats
allocated by the caller.
The API will fill in the provided struct page_pool_stats
with
statistics about the page_pool.
-
struct page_pool_alloc_stats¶
allocation statistics
Definition:
struct page_pool_alloc_stats {
u64 fast;
u64 slow;
u64 slow_high_order;
u64 empty;
u64 refill;
u64 waive;
};
Members
fast
successful fast path allocations
slow
slow path order-0 allocations
slow_high_order
slow path high order allocations
empty
ptr ring is empty, so a slow path allocation was forced
refill
an allocation which triggered a refill of the cache
waive
pages obtained from the ptr ring that cannot be added to the cache due to a NUMA mismatch
-
struct page_pool_recycle_stats¶
recycling (freeing) statistics
Definition:
struct page_pool_recycle_stats {
u64 cached;
u64 cache_full;
u64 ring;
u64 ring_full;
u64 released_refcnt;
};
Members
cached
recycling placed page in the page pool cache
cache_full
page pool cache was full
ring
page placed into the ptr ring
ring_full
page released from page pool because the ptr ring was full
released_refcnt
page released (and not recycled) because refcnt > 1
-
struct page_pool_stats¶
combined page pool use statistics
Definition:
struct page_pool_stats {
struct page_pool_alloc_stats alloc_stats;
struct page_pool_recycle_stats recycle_stats;
};
Members
alloc_stats
recycle_stats
Description
Wrapper struct for combining page pool stats with different storage requirements.
Coding examples¶
Registration¶
/* Page pool registration */
struct page_pool_params pp_params = { 0 };
struct xdp_rxq_info xdp_rxq;
int err;
pp_params.order = 0;
/* internal DMA mapping in page_pool */
pp_params.flags = PP_FLAG_DMA_MAP;
pp_params.pool_size = DESC_NUM;
pp_params.nid = NUMA_NO_NODE;
pp_params.dev = priv->dev;
pp_params.napi = napi; /* only if locking is tied to NAPI */
pp_params.dma_dir = xdp_prog ? DMA_BIDIRECTIONAL : DMA_FROM_DEVICE;
page_pool = page_pool_create(&pp_params);
err = xdp_rxq_info_reg(&xdp_rxq, ndev, 0);
if (err)
goto err_out;
err = xdp_rxq_info_reg_mem_model(&xdp_rxq, MEM_TYPE_PAGE_POOL, page_pool);
if (err)
goto err_out;
NAPI poller¶
/* NAPI Rx poller */
enum dma_data_direction dma_dir;
dma_dir = page_pool_get_dma_dir(dring->page_pool);
while (done < budget) {
if (some error)
page_pool_recycle_direct(page_pool, page);
if (packet_is_xdp) {
if XDP_DROP:
page_pool_recycle_direct(page_pool, page);
} else (packet_is_skb) {
skb_mark_for_recycle(skb);
new_page = page_pool_dev_alloc_pages(page_pool);
}
}
Stats¶
#ifdef CONFIG_PAGE_POOL_STATS
/* retrieve stats */
struct page_pool_stats stats = { 0 };
if (page_pool_get_stats(page_pool, &stats)) {
/* perhaps the driver reports statistics with ethool */
ethtool_print_allocation_stats(&stats.alloc_stats);
ethtool_print_recycle_stats(&stats.recycle_stats);
}
#endif
Driver unload¶
/* Driver unload */
page_pool_put_full_page(page_pool, page, false);
xdp_rxq_info_unreg(&xdp_rxq);