aboutsummaryrefslogtreecommitdiffstats
AgeCommit message (Collapse)AuthorFilesLines
13 daysfix: handle EINTR correctly in get_cpu_mask_from_sysfsHEADmasterMathieu Desnoyers1-1/+1
If the read() in get_cpu_mask_from_sysfs() fails with EINTR, the code is supposed to retry, but the while loop condition has (bytes_read > 0), which is false when read() fails with EINTR. The result is that the code exits the loop, having only read part of the string. Use (bytes_read != 0) in the while loop condition instead, since the (bytes_read < 0) case is already handled in the loop. Original fix in liburcu from Benjamin Marzinski <bmarzins@redhat.com>: commit 9922f33e2986 ("fix: handle EINTR correctly in get_cpu_mask_from_sysfs") Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: I134662fe83613ff914433d3b30a63a0134ffa124
2024-04-12Revert "mempool: Track allocated items per range"Mathieu Desnoyers1-12/+1
This reverts commit ffea0dea60059c855cb0b4e1f784352012ae0b37. Counting allocated items per range has no inherent use and adds a counter to the allocation/free fast path: we could not easily unmap ranges when the allocated items count reaches 0 anyway because the pool free list spawns across items from all ranges. Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
2024-04-12mempool: malloc: handle empty range listMathieu Desnoyers1-12/+18
Although it is not currently possible to have an empty range list because there is a range created on pool creation, handle empty range list in __rseq_percpu_malloc for future-proofing. Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: Ie646b16114eb1812916dad2b08f9d87884925748
2024-04-12mempool: use list.h APIMathieu Desnoyers5-23/+27
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: I2cabb4b8b6ff70cd111961335c0306b0ec76e827
2024-04-12mempool: Track allocated items per rangeMathieu Desnoyers1-1/+12
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: I48c05318c39b3f3bcf9c14a5aa3181e29754db97
2024-03-22mempool: default to per-cpu mempool typeMathieu Desnoyers2-5/+5
The primary use-case for mempool is per-cpu data. Default to this rather than global. Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: I63d994de80e8ed16e0f5666a846d7e872a58cdca
2024-03-20mempool: Rename pool policyMathieu Desnoyers4-68/+101
Rename pool policy from: RSEQ_MEMPOOL_POPULATE_PRIVATE_NONE -> RSEQ_MEMPOOL_POPULATE_COW_INIT RSEQ_MEMPOOL_POPULATE_PRIVATE_ALL -> RSEQ_MEMPOOL_POPULATE_COW_ZERO to make it clear to the user that the COW_ZERO pool policy benefits from leaving the memory zero-allocated. Also, now that zmalloc checks the prior content for zero to prevent useless COW of the per-cpu ranges, the term "populate all" does not fit well anymore. Copy-on-write from the zero page is clearer. Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: If97984225d971a2f48ba241b2a9570e81079bd03
2024-03-19Add Olivier Dion to list of mempool authorsMathieu Desnoyers2-0/+2
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: I472019d46812e574bc6bef3271635c786263c4a3
2024-03-19mempool: Introduce COW_ZERO poison valueMathieu Desnoyers1-3/+8
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: Id7fdfa0ceb8631e1b4bfe594d9e68c4d8663c6ef
2024-03-19mempool: Document destroy after fork for populate noneMathieu Desnoyers1-1/+3
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: I826990c9ee58dbda80f3271e361328b474edbf08
2024-03-19mempool: Tests mempool destroy in child processMathieu Desnoyers1-0/+42
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: I8615436deaf089ba0f560145d604251307b307b9
2024-03-19mempool: Allow mempool destroy in children processesMathieu Desnoyers1-19/+83
Allow destroy of mempool objects from a child process by validating whether the mappings are accessible or not. Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: Ice9dd8cd1fad1e656dab0b457fc5469ecfae1130
2024-03-19tests/mempool: Allocate pool in child for robust testsOlivier Dion1-30/+56
Depending on the populate policy, it is not possible to use a pool across a fork. Since robust tests rely on SIGABRT to be triggered in a child process, the pool must be allocated in the child instead of the parent for the RSEQ_MEMPOOL_POPULATE_PRIVATE_NONE policy. Change-Id: I78bc19d12b4f8bc0c7e2a2f05fab870ed7524e13 Signed-off-by: Olivier Dion <odion@efficios.com> Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
2024-03-19mempool: Fix error pool->rangeMathieu Desnoyers1-1/+1
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: I563b33d55e565a8e1af1558f7ade56e977fd98bd
2024-03-19mempool: Protect whole range of populate none across forkMathieu Desnoyers2-7/+9
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: I6928128ceffed57fd4ecfdaee50a3085e585387a
2024-03-19mempool: Fix memfd leaks on errorMathieu Desnoyers1-2/+7
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: Id22b56ffc10f57021a8cb1828ef5b032436d63ea
2024-03-19mempool: Use MADV_DONTFORK on init valuesMathieu Desnoyers1-0/+6
The init values shared mapping should not be shared across fork. Use madvise MADV_DONTFORK on the memory range to ensure those pages are not shared with children processes across fork. Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: I4b3541d21401227ca568ef6e8105d088746341cb
2024-03-19mempool: Add custom name to memfd_createOlivier Dion1-3/+12
Change-Id: I12459129e2c4aa7e35ab557fbc6060bd63de816b Signed-off-by: Olivier Dion <odion@efficios.com> Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
2024-03-19mempool: Create memfd only for range creationMathieu Desnoyers1-104/+27
Don't share memfd across ranges, we can close its file descriptor immediately after mapping it. Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: Ia0c63f74e17b16039dff0b9367a19c411396605d
2024-03-18mempool: Rename DEFAULT_POISON_VALUE to DEFAULT_PRIVATE_POISON_VALUEMathieu Desnoyers1-3/+3
I expect the eventual default SHARED poison value to be 0x0 to eliminate useless allocation over zero pages for unused cpus. Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: I6faad228bce381485ed60cd92b3f155187a68b4e
2024-03-18mempool: Introduce "private" populate policyMathieu Desnoyers4-28/+31
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: I3a4907ee65a58e7dbb3d03bf5096175620892915
2024-03-18mempool: Remove mmap/munmap callbacksMathieu Desnoyers2-73/+11
Exposes too much internals of mempool to the application. Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: I49bd3613b473882eb97e71bce2cf31fcb052dc5a
2024-03-18mempool: replace memcmpbyte by rseq_cmp_itemMathieu Desnoyers1-25/+14
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: Ic45d9a6a64e86772c45815127d3d0a601e344278
2024-03-18mempool: use bzeroMathieu Desnoyers1-2/+2
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: I1a1d13e856dd73576cc0c2fea46fb4dcd3eef963
2024-03-18mempool: do not overwrite same per-cpu valuesMathieu Desnoyers1-16/+56
If the per-cpu items already have the correct values, do not write to the pages. This eliminates useless COW in populate-all pools over the zero page. Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: I4e47db82ccf15c3cca30a8df48d367cbc05f3887
2024-03-18mempool cow test: use default strideMathieu Desnoyers1-4/+3
Fix ppc64 which has a 64kB page size. Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: I2debd1c50f1ecc0486a557b69af3fda70bae2dcd
2024-03-17mempool: Test COW vs malloc_init raceMathieu Desnoyers4-0/+221
Test that the entire malloc init value is visible in CPU mappings. If the COW page copy race vs init happens while init is in the middle of storing to the newly allocated area, iteration on all CPUs comparing the visible content to the init value is responsible for detecting and mitigating uninitialized or partially initialized init value from the point of view of a CPU. Validate that this scheme has the intended effect wrt a concurrent COW caused by storing to a nearby per-cpu area on the same page. Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: Iebde10eb61e80cc0d8eb0ebc89925f4a2ad9de9c
2024-03-16mempool tests: Adapt to free list new locationMathieu Desnoyers1-27/+44
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: If582864dcc7aceea99f866ad3ca0f6391d6ca5c5
2024-03-16Move robust pool free list to own memory areaMathieu Desnoyers1-20/+36
Consumes more memory in the robust pool, but removes blind spots for validation of poison due to aliasing with free list. Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: Ia8187cf3049aad8e54daaced4005fbe2e841ae7e
2024-03-16mempool: Fix off_t cast warning on 32-bitMathieu Desnoyers1-4/+10
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: I93978e7f0f583ae4a9bf62d3f707258c385bb0a8
2024-03-16mempool test: test populate none policyMathieu Desnoyers1-18/+28
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: I2eb34565ba84ae9479eb235743fbba07fe277165
2024-03-16mempool: Implement populate none policyMathieu Desnoyers3-32/+325
Implement lazy-populate policy (populate none) relying on kernel COW of init values pages. Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: Ie4467f1806208271a5462605445abc2f0c1c1d38
2024-03-15mempool: Fix leak on pool destroyMathieu Desnoyers1-1/+1
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: I3534084f1055b0db73ac9ec5005a6c88d8056d78
2024-03-13mempool: Introduce mmap_addr and mmap_len range fieldsMathieu Desnoyers1-2/+8
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: Ia91571ba9f3535402f456074973875503e688690
2024-03-10mempool_test: test rseq_mempool_percpu_malloc_initMathieu Desnoyers1-0/+20
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: Ia266d69a4007340ffe1d98950619ce90582c2e40
2024-03-10mempool: implement malloc_initMathieu Desnoyers2-15/+117
Add a *_malloc_init to the API to allow allocating memory initialized to a given initial value. This replaces application patterns where a loop on all possible CPUs is needed after allocation to initialize memory to a given nonzero value, and will eventually allow doing the initialization lazily (the first time the cpu touches memory within the pool). Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: I3acb1505d7b4f46ceb6a8128d2ebda346b8e359c
2024-03-09mempool_test: test poison corruption detectionMathieu Desnoyers1-8/+45
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: Ia284caa599f2c9c7c142efe6508779d36d828e17
2024-03-09mempool: Use default poison value when robust is setMathieu Desnoyers1-2/+12
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: I84a75bd14f6798261fd630aaf076cc3981bad0a1
2024-03-09mempool: check poison on destroyMathieu Desnoyers1-3/+28
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: I40fe7d4179e42a392f3528cc9697552fe7c4ad09
2024-03-09mempool: Detect poison corruption on allocMathieu Desnoyers2-8/+42
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: I1d4181f089aa08d1c14466751661bba8a3727cf5
2024-03-09mempool test: add one extra object after freeMathieu Desnoyers1-1/+10
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: Ie4b5f47ac4e2108cecb079c63767259ee0493a48
2024-03-09mempool: introduce poison attributeMathieu Desnoyers3-0/+59
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: I056c0d8c4f0d5b1d2c401592a08b22ff6f239d11
2024-03-09mempool test: Only use a single range for large strideMathieu Desnoyers1-1/+1
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: I62baaa751d46abf0341e4e1c3dffecd1b2402986
2024-03-09mempool test: test max_nr_ranges > 1Mathieu Desnoyers1-7/+18
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: I7d3b48690c5e61789f0bc5f50db27898499c335a
2024-03-09mempool: Introduce rseq_mempool_attr_set_max_nr_rangesMathieu Desnoyers3-4/+28
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: I5417673d16ce7cfa5e8f2b9c120978f8633664bc
2024-03-09mempool: Introduce support for ranges linked listMathieu Desnoyers1-28/+54
Each pool can allocate additional ranges when the initial range is full. There is currently a limit of 1 range (to be made configurable in the future). Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: Ibc59f140a8dc1ac9152b0a4ae82d32db76a2c42e
2024-03-09mempool: remove unused FIRST_POOLMathieu Desnoyers1-6/+0
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: I545c7f2ff7f4b760c92bf0689b5f4edbe1894329
2024-03-09mempool: cpu=-1 for init_func argument (global pool)Mathieu Desnoyers2-9/+25
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: I956006f59186252d3dcc767fa21e5928f68d5cb8
2024-03-09mempool: init_func can return an errorMathieu Desnoyers2-7/+12
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: I212fd5a6ec6ae7580a28068de4ee88d7955da9d0
2024-03-09mempool: aligned_mmap_anonymous: support non-power-of-2 nr_cpusMathieu Desnoyers1-1/+1
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: I7231ee9ac6e822b9d43dae596763d84b0abfcbe1
2024-03-09rseq: Introduce rseq_get_max_nr_cpusMathieu Desnoyers3-2/+16
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: Ie7d24470994b3a6ec805bfec7111ff3492b46997
2024-03-09param_test: Use mempool max_nr_cpus rather than CPU_SETSIZEMathieu Desnoyers1-35/+45
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: I6401e5584ee97ea8958898e377e1ec52476814c6
2024-03-09mempool: Introduce rseq_mempool_get_max_nr_cpusMathieu Desnoyers2-0/+18
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: Ibd5a7abf40e56d8e57c7b51a5dd9495c51bd9624
2024-03-09mempool: Introduce rseq_mempool_range_init_numa helperMathieu Desnoyers2-55/+51
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: Icc0ac582fbd21a71f01212d23b86a9ecd5679465
2024-03-09mempool: Introduce init_func attributeMathieu Desnoyers2-1/+42
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: I1e36bae465b0f459b4b6a4f8182cf524e0e6ade5
2024-03-09mempool: reenable rseq_mempool_range_init_numaMathieu Desnoyers1-9/+11
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: I8c7d3ccdf0279cca44e8ad33f8904e34b38f8a79
2024-03-08smp.c: fix includes and macro namespaceMathieu Desnoyers1-5/+5
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: I5ef3c45240120f7f415c3bb38a280e3a7ceb410a
2024-03-08Mempool: default as globalMathieu Desnoyers2-7/+22
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: I1a2c44188404e1703a9996a83495086d2b0e3ca3
2024-03-08mempool: Receive mempool type (percpu/global) as attributeMathieu Desnoyers7-66/+499
Also introduce a smp.c/h helper to get the number of possible cpus, copied from libside (MIT). Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: I4dfc8fa56ae29e5586eb029377abad62c93de6a9
2024-03-08mempool test: iterate from page sizeMathieu Desnoyers2-5/+9
Fix test on ppc64, which has a 64kB page size. Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: I6e309855c190b056d886aa3c187a9dc1929d7391
2024-03-08mempool: update comments in header fileMathieu Desnoyers1-11/+11
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: I35a017716d5f49e2eb827cd7ed4f56855d0f8e1c
2024-03-08mempool: reduce default RSEQ_PERCPU_STRIDE to 64kB on 64-bitMathieu Desnoyers1-5/+1
aarch64 has issues with mmap() of 16MB * 1000 cpus (ENOMEM). Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: If22aab4f9afacaf9bdf0318c8876486cdda2ec25
2024-03-08membarrier rseq fence test: adapt to RSEQ_PERCPU_STRIDEMathieu Desnoyers1-2/+1
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: Ifaa59012cf19695eabe9977cae73a281e34afaa7
2024-03-08Add missing rseq_after_asm_goto in rseq_load_add_load_load_add_store__ptrMathieu Desnoyers1-0/+3
Change-Id: If2a966e39302d5163203769855ed0d5f5e6012a7 Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
2024-03-08Revert "x86-64: Implement/use load_cbne_load_add_load_add_store"Mathieu Desnoyers4-39/+26
Now that the percpu mempool allocator has a stride known in advance, it is possible to use the offset approach in the critical section to index by cpu number. This reverts commit d87b0cd222a1c11751be7f8b655fc0dc45b7dc12. Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: I8b394650f1dbe409eb9ed3db88afcc43cd230133
2024-03-08mempool: namespacing, add global alloc/freeMathieu Desnoyers4-54/+130
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: I6b3a6f9948f8400ae691738d7caa15164e634b74
2024-03-08mempool: Introduce optional stride parameterMathieu Desnoyers4-20/+31
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: Ic7119bb33f86cf72c73c2d50dbe55e4769ce0b22
2024-03-08mempool: mass renaming of APIMathieu Desnoyers4-151/+151
- rseq_percpu_pool -> rseq_mempool - rseq_pool_attr -> rseq_mempool_attr - rseq_percpu_pool_set -> rseq_mempool_set Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: Ic34a4163854e91838b2f0727d26469d55c69a0e0
2024-03-08tests/mempool_test: Add robust testingOlivier Dion1-0/+106
Change-Id: I0e7c9dd7ad03070f0524edb4252cb11988418108 Signed-off-by: Olivier Dion <odion@efficios.com> Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
2024-03-08mempool: check for NULL pool on destroyMathieu Desnoyers1-5/+2
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: I1a095ac04c4545095ed53699402a30d2d1ace8a7
2024-03-08mempool: update header commentsMathieu Desnoyers1-2/+12
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: I7648cd668ddb4a18dd6d04d021198833bfc078b5
2024-03-08mempool: Remove unused definesMathieu Desnoyers1-17/+1
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: I35f3ce87da25c47097bb76ab9c3aeadee46f5e32
2024-03-08mempool: Remove pool array and lockMathieu Desnoyers1-37/+5
Those are not needed anymore with the new mask-based scheme. Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: Id0d401d21b4ac3b8b7c7e50fecf6bd92513bd162
2024-03-08mempool: Use masks to get pool range from pointer and stride for percpu offsetMathieu Desnoyers3-34/+35
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: I45bb460be66bc4843ead316d7b870958cd1217c2
2024-03-08Memory map ranges aligned on the strideMathieu Desnoyers1-8/+93
This will allow finding the range header with a mask applied on the pointer in free(). Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: I25821ebb4a7333fe53658932f58c6d9cad1bd64a
2024-03-08percpu alloc: Introduce stride parameterMathieu Desnoyers5-45/+73
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: I6469725da2610560872d2fe678128455d7ff22f1
2024-03-08rseq-mempool: Call set_alloc_slot() on all pathsOlivier Dion1-1/+2
set_alloc_slot() was not called when getting free slot from free-list. Change-Id: I21d633ced6592039fe39e0ae121128958048f80e Signed-off-by: Olivier Dion <odion@efficios.com> Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
2024-03-08mempool: implement one rangeMathieu Desnoyers1-57/+132
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: I189ee936cd11229b5770ed95c8ac32b67fede20d
2024-03-08Update mempool testMathieu Desnoyers1-0/+1
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: I57fe799bcdb4df1d6d6fb6e605e17cf581f13197
2024-03-08Add mempool test to make checkMathieu Desnoyers2-1/+5
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: Ia998f1157083df566e8e8e9c331983bab256880f
2024-03-08mempool test: safe iteration for freeMathieu Desnoyers1-2/+2
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: Id5fe8f73dff0c914646a76af45738a2e80a8e2b2
2024-03-08Introduce mempool testMathieu Desnoyers5-0/+313
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
2024-03-08Rename percpu-alloc files to mempoolMathieu Desnoyers6-12/+12
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: I0a1ba042649cf3b2224a90a05e095733e9a597fc
2024-03-08Update gitignoreMathieu Desnoyers1-0/+1
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: I55d5e3571e7115cec5d09fac52ce51fa3601c12d
2024-03-07percpu alloc: Remove RSEQ_POOL_ROBUST attributeMathieu Desnoyers1-15/+0
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: If90ce5767d0d2516674efda08fb0d694b4f8d1a2
2024-03-07percpu-alloc: Add move_pages batchingOlivier Dion2-8/+40
Change-Id: I5e9f40d73f45f2abeded1d1f62056a478fd8455a Signed-off-by: Olivier Dion <odion@efficios.com> Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
2024-03-07percpu pool: Move robust flag to atttributeMathieu Desnoyers3-25/+45
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: Ie83fff22b71dc9be39d90e4be97d094b90542715
2024-03-07percpu pool: mmap set attr: return error valueMathieu Desnoyers2-2/+9
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: Id2debf769299ee1d6624097b38e79f916b8c0f2b
2024-03-07percpu pool: Introduce generic attributesMathieu Desnoyers2-46/+49
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: I7641218c7f69b73fc998eddf2839312fcf5726e9
2024-03-07percpu alloc: add pool nameMathieu Desnoyers3-24/+45
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: I886c840f394cb1de0b5ab71b7417b99675ea1468
2024-03-07Fix typos in fprintfMathieu Desnoyers1-2/+2
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: Ic82493c649a2389391808f079d9ed58c81e61018
2024-03-07percpu-alloc: Add free-list check on pool destroyOlivier Dion1-1/+54
Change-Id: I68bab4673aabde92dc0d8321de2d5561c5e7a11c Signed-off-by: Olivier Dion <odion@efficios.com> Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
2024-03-07rseq percpu: print caller ip on double-free/alloc corruption and leakMathieu Desnoyers1-10/+14
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: I5b1bc7a2a5711e6c9de4bc82df92a49df5654f8b
2024-03-07Fix: add missing \n in fprintfMathieu Desnoyers1-2/+2
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: Ibbb61e3b7f66f8d9b7ed125e4123581b2c1b93d9
2024-03-07Implement rseq_hweight_ulong with __builtin_popcountlMathieu Desnoyers1-5/+1
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: I483d53c7c5402e034b8d468e0ed2ab048184f069
2024-03-07Fix destroy_alloc_bitmap sizeMathieu Desnoyers1-2/+1
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: Ic53da70a6fbcbbdd87c885d01d80a5ba65befb5a
2024-03-07percpu alloc: Improve robust poolsMathieu Desnoyers2-56/+95
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: I4966965cca5143640347a4b21fef3e2c353c8366
2024-03-07percu-allocator: Implement robust pool validationOlivier Dion2-3/+104
When set, the RSEQ_POOL_ROBUST flag will add a bitmap to the pool which is used to track double-free and allocation leaks in the pool. Signed-off-by: Olivier Dion <odion@efficios.com> Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: I531782ee57d33c9704b9cca44af1a9464f60cb90
2024-03-05percpu: remove rseq_percpu_pool_ptr_offsetMathieu Desnoyers2-18/+0
The rseq_percpu_pool_ptr_offset API is now unused, and it leaks implementation details about per-cpu pools which may prevent implementing auto-extensible pools efficiently in the future. Remove it. Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: Ib62c4af34c04f9c4e5227dab2f487356478b32fa
2024-03-05x86-64: Implement/use load_cbne_load_add_load_add_storeMathieu Desnoyers4-26/+39
Change the current load_add_load_load_add_store for a load_cbne_load_add_load_add_store, which allows validating that the percpu pointer did not change since it was loaded from C. This allows precomputing the address of the percpu memory area in C and provide it as a second pointer argument. The comparison approach is prefered to the offset-from-pointer approach because it does not leak implementation details of the percpu allocator. Add missing rseq_after_asm_goto() in the static inline function. Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: I3fe9d57d13f7507d5af95ef37391ad36fe2221fe
2024-03-05Remove riscv has_load_add_load_load_add_store implementationMathieu Desnoyers2-56/+0
The implementation is currently buggy: RSEQ_ASM_OP_R_DEREF_ADDV() should increment the address _target_ as a final store, not just the address within the temporary register. Remove it for now. Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: I0f2a854cec4ab5e58ea6cbb74d40ca29b256d27e
2024-03-05param test: membarrier: validate total number of incrementsMathieu Desnoyers1-2/+23
While reviewing the riscv implementation for this test, I noticed that the final store is only incrementing the content of its register, without ever storing it to memory. This passes testing because the increment is effectively a no-op, and the test never validates that any increment happen in the first place. Introduce a validation of the increment total to eliminate those false-positives. Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: Ie9ee1c7902509da75dd85babd07bc180ad0b9ae9
2024-03-05percpu allocator: Add flags argument for future extensionsMathieu Desnoyers3-8/+18
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: I4c507a99351c450ef2c6396af5391f019f8e3d91
2024-03-04percpu alloc: introduce rseq mmap attributesMathieu Desnoyers3-50/+126
Introduce rseq mmap attribute argument to pool creation, allowing users to override the memory allocator used to allocate/free the pool memory. Move NUMA page move to a separate rseq_percpu_pool_init_numa() API. Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: Ie08c7250fce5f567c7a4aee469e010c5dcf39bc6
2024-03-04Cleanup: Remove whitelineMathieu Desnoyers1-1/+0
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: I537b82707f1f9a95005da672b98196ab44875e2a
2024-03-04Rename RSEQ_ARCH_HAS_OFFSET_DEREF_ADDV to ↵Mathieu Desnoyers4-4/+4
rseq_arch_has_load_add_load_load_add_store Fix param test to use rseq_arch_has_load_add_load_load_add_store rather than rseq_arch_has_load_cbne_load_add_store. Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: I373c91dc9178dd69ed159d568090e8c732cf4485
2024-03-04Revert "Introduce rseq_load_cbne_load_add_store pseudocode"Mathieu Desnoyers1-15/+0
This reverts commit 3eff770448484493fa15ea90db643f86c124db6c. Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
2024-03-04Revert "x86: Introduce rseq_load_cbne_load_add_store"Mathieu Desnoyers1-120/+0
This reverts commit 18f21b27f01b9147cd95c538ece07f95c37150a2. Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: Ieffcd50965fe6dcf2c7d067f5fddb4e9f62e82f3
2024-03-04Revert "rseq.h: Introduce rseq_load_cbne_load_add_store__ptr"Mathieu Desnoyers1-19/+0
This is not needed anymore. This reverts commit 040147907ff31cdc279568ffcb74197650430e4b. Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
2024-03-04param tests: membarrier: pre-decode pointer offsetMathieu Desnoyers1-5/+7
Loading "cpulist->head" outside of the rseq critical section is unintended: loading cpulist->head loads memory from the per-cpu list outside of the rseq critical section, which happens to work because the memory is not actually freed in this specific test case, but would break if memory would be reclaimed after the rseq fence. So go back to the originally used rseq_load_add_load_load_add_store__ptr(), and use the new rseq_percpu_pool_ptr_offset() to pre-calculate the offset. Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: I5e621c724e143e38a6dfabd26bbf682a6fcc5182
2024-03-04percpu alloc: introduce rseq_percpu_pool_ptr_offsetMathieu Desnoyers2-0/+18
Introduce rseq_percpu_pool_ptr_offset to pre-decode the offset from all __rseq_percpu pointers for a given pool. This is useful to prepare offsets that would need to be calculated within rseq critical sections. Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: I42d2669ed5ed81d74aae551b721c9cb04b25fe45
2024-03-04Document percpu allocMathieu Desnoyers2-4/+204
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: I8fa4b68e8a68c57da8ab726d0f68ee12aad2f15e
2024-03-04rseq percpu: Remove __rseq_percpu annotation from returned pointerMathieu Desnoyers1-1/+1
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: Iea04ce173cc529612b1024393794a304b0e7e219
2024-03-03x86-32: ip-relative: return to post-call addressMathieu Desnoyers1-4/+4
This simplifies the assembler returns to the expected (speculated) address. Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: I68c59f72ac32011d37cc9399414ff328f7b8fdd0
2024-03-02param test: x86-32: use ip-relative addressing for RSEQ_INJECT_ASMMathieu Desnoyers1-7/+8
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: Id8349df846f5b0491b348b603f0423b726c55b8c
2024-03-02x86-32: Save ip reference for ip-relative addressingMathieu Desnoyers2-20/+83
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: Ie95e1bf409f0ca0c271f529b694839905d4d0d91
2024-03-02x86-32: Use ip-relative addressing to store rseq_csMathieu Desnoyers1-6/+16
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: I294a4844bf8022be480059a028b9a17a5a2dcc4b
2024-03-02x86-32: work-around register pressure limitationsMathieu Desnoyers1-58/+48
Use explicit register clobbers and load input values from "m" input operands to reduce register pressure. Note that old gcc does not support output operands for asm goto, so we cannot simply re-use an input register as output. Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: I745d27f416ef303b1049e8c12b122e5b4a271f63
2024-03-02x86-32: Document linker warningMathieu Desnoyers2-0/+14
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: I35e1db5eda78ea4b6349039ed34d739b33b81d6d
2024-03-02arm: work-around register pressure limitationsMathieu Desnoyers1-61/+50
The recent move to percpu allocator breaks the 32-bit ARM build due to register pressure. Fix this by using explicit register clobbers and loading input values from "m" input operands. Note that old gcc does not support output operands for asm goto, so we cannot simply re-use an input register as output. Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: I1d546e427c7f6b62035433815806c8ec275a55f8
2024-03-01param tests: percpu memcpu buffer: use percpu allocMathieu Desnoyers1-32/+56
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: I86c975aa8578469f685dcdb4951f4af50f06b3fe
2024-03-01param tests: percpu buffer: use percpu allocMathieu Desnoyers1-31/+56
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: I49f29134679931eed4e06140a72cc56592cc12b8
2024-03-01param tests: percpu list: use percpu allocMathieu Desnoyers1-43/+91
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: Ie1415e33187902e76dd3d492cd8ed2fe9b980565
2024-03-01rseq.h: Introduce rseq_load_cbne_load_add_store__ptrMathieu Desnoyers1-0/+19
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: I5011d4115d5dc0064b0d3fc359c037dcda02508c
2024-03-01x86: Introduce rseq_load_cbne_load_add_storeMathieu Desnoyers1-0/+111
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: Id5d8ebfe056ef60ba38a7ab0bf1e368ca324ab25
2024-03-01Introduce rseq_load_cbne_load_add_store pseudocodeMathieu Desnoyers1-0/+15
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: I2c5099af7958591521f6e934da128b54ca3535c3
2024-03-01Rename rseq_load_add_load_add_store to rseq_load_add_load_load_add_storeMathieu Desnoyers5-8/+8
Match the pseudo-code. Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: I48d60a9688cf76936fa1ba7a3b1d6b7412a77e69
2024-03-01param test: counter increment: use percpu allocMathieu Desnoyers1-12/+27
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: Ia98154794a80ce917a742961c95d3aef28f7aa1e
2024-03-01param test: use __rseq_percpu pointer taggingMathieu Desnoyers1-6/+6
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: I2735cd5d08e5f7ed085411934cc5e74938a7ca46
2024-03-01Add __rseq_percpu pointer taggingMathieu Desnoyers2-18/+28
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: I7a10ac1ad1e97aaaad947e29f377a103e380286d
2024-03-01Adapt spinlock param test to percpu allocMathieu Desnoyers1-19/+37
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: I7602983d2aaf9f9a59209c9b3b1cce6fed4ca74d
2024-03-01Skip first poolMathieu Desnoyers1-1/+7
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: If07b9bbe91fa24b805d7943d4c0cb448cc89c210
2024-03-01percpu alloc: add missing extern CMathieu Desnoyers1-0/+8
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: I3039487f8e9a130b2fbcb8e6771df449d6021087
2024-03-01Add missing include in percpu-alloc.hMathieu Desnoyers1-0/+1
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: Ic3d124544c8a651d61f69a872ed03bdfcb638232
2024-03-01Set errno on malloc errorMathieu Desnoyers1-0/+1
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: Ia03a24e63e10907525042f59b1c062ca4f8fe8fc
2024-03-01percpu pool set malloc: start search at relevant alloc orderMathieu Desnoyers1-0/+3
Start search for a pool which can accomodate the @len allocation at an allocation order where it can fit. Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: Ib887759416120cad78da6a3758d5a8e5781380ad
2024-03-01rseq-percpu: Improve documentationMathieu Desnoyers1-8/+13
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: Ib9e850d38c7c8bac2a7b94df767275f8f659cb9c
2024-03-01Introduce rseq-alloc-utils.hMathieu Desnoyers3-106/+115
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: I33970b558ad0d59629737d1a6c0d8531fbcf93e9
2024-03-01rseq percpu: Use high bits for pool indexMathieu Desnoyers1-14/+22
This leaves the low bits available for pointer tagging. Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: I7cebf88a1db4d6cdfd5d287626c67888e80ffbd2
2024-03-01rseq percpu alloc: Implement numa supportMathieu Desnoyers4-39/+123
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: I732b632f476ffef362a1ab486bcf425e4ded6644
2024-03-01Implement per-cpu memory allocatorMathieu Desnoyers4-1/+485
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: I54f43ccec743b7c5311221fac22a0b690a863377
2024-02-29Introduce __RSEQ_ASM_DEFINE_CS_FIELDSMathieu Desnoyers3-10/+17
The new macro __RSEQ_ASM_DEFINE_CS_FIELDS can be used by __RSEQ_ASM_DEFINE_TABLE as well as __RSEQ_ASM_DEFINE_ABORT on arm and mips, thus removing redundant code. Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: Ie03f5d51d20027a3eb3b9eba636b0f5cc890c333
2024-02-29Introduce common generic header fileMathieu Desnoyers9-452/+131
Move: - RSEQ_ASM_DEFINE_TABLE, - __RSEQ_ASM_DEFINE_TABLE, - RSEQ_ASM_DEFINE_EXIT_POINT. to a generic common header file now that those are exact copy across all architectures. Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: I1f27e8889b13d263467a1bab92b272cc80aa0743
2024-02-29x86: Introduce RSEQ_ASM_U32Mathieu Desnoyers1-1/+4
This will allow moving __RSEQ_ASM_DEFINE_TABLE and RSEQ_ASM_DEFINE_EXIT_POINT to a generic common header file. Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: I966b67f00bf39bd0b26f8efd6c4f40cba4caa1cf
2024-02-29riscv: Introduce RSEQ_ASM_U64_PTR and RSEQ_ASM_U32Mathieu Desnoyers1-7/+12
These will allow moving __RSEQ_ASM_DEFINE_TABLE and RSEQ_ASM_DEFINE_EXIT_POINT to a generic common header file. Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: I9cac7076b9ef9bd7dd4a818b9ad7aae1df45ff5a
2024-02-29s390: Introduce RSEQ_ASM_U32Mathieu Desnoyers1-2/+5
This will allow moving __RSEQ_ASM_DEFINE_TABLE and RSEQ_ASM_DEFINE_EXIT_POINT to a generic common header file. Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: I0af09a3fa3970efbe523db04154f1969241369a1
2024-02-29ppc: Introduce RSEQ_ASM_U32Mathieu Desnoyers1-2/+5
This will allow moving __RSEQ_ASM_DEFINE_TABLE and RSEQ_ASM_DEFINE_EXIT_POINT to a generic common header file. Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: Ic476d6dfca6f714cf892f84560884157abbc0360
2024-02-29mips: Introduce RSEQ_ASM_U32Mathieu Desnoyers1-3/+7
This will allow moving __RSEQ_ASM_DEFINE_TABLE and RSEQ_ASM_DEFINE_EXIT_POINT to a generic common header file. Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: I5a68ce2782bc36d5fc8c030ca7716e41e925cd54
2024-02-29arm: Introduce RSEQ_ASM_U32Mathieu Desnoyers1-3/+7
This will allow moving __RSEQ_ASM_DEFINE_TABLE and RSEQ_ASM_DEFINE_EXIT_POINT to a generic common header file. Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: I32a0eff75998afb1f8257c3312187ab2938d59a3
2024-02-29aarch64: Introduce RSEQ_ASM_U32Mathieu Desnoyers1-1/+3
This will allow moving __RSEQ_ASM_DEFINE_TABLE and RSEQ_ASM_DEFINE_EXIT_POINT to a generic common header file. Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: I4496c41d37094983b724fc8bb101688f75fefbbd
2024-02-29thread-pointer: suffix include guard with _HMathieu Desnoyers1-2/+2
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: Id35f1cb103e1d68bde10785e44c5c16104f000d4
2024-02-29arm: Fix big endian, introduce RSEQ_ASM_U64_PTRMathieu Desnoyers1-4/+15
Fix support for Big endian ARM by introducing RSEQ_ASM_U64_PTR. This will allow moving __RSEQ_ASM_DEFINE_TABLE and RSEQ_ASM_DEFINE_EXIT_POINT to a generic common header file. Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: I6271f217b0e1463968bd7d520ddd0fb61678bb84
2024-02-29arm: Cleanup header codying styleMathieu Desnoyers1-4/+4
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: Iab1aee9103eef2179fcdc8c2171afda3d311379b
2024-02-29aarch64: Introduce RSEQ_ASM_U64_PTRMathieu Desnoyers1-5/+8
This will allow moving __RSEQ_ASM_DEFINE_TABLE and RSEQ_ASM_DEFINE_EXIT_POINT to a generic common header file. Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: Ic5470bb851be1b79bbba387a4db2ce0911bbe23d
2024-02-29aarch64: Cleanup header coding styleMathieu Desnoyers1-4/+4
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: I7986cf871be81fc8bb94d0d44d9f61ee41ba0fc1
2024-02-28s390: Introduce code deduplication macrosMathieu Desnoyers1-86/+53
Introduce RSEQ_ASM_U64_PTR() which expresses the type and instance of the defined variable, allowing deduplication of the following macros: - __RSEQ_ASM_DEFINE_TABLE, - RSEQ_ASM_DEFINE_EXIT_POINT. Separate load/store/op helper macros, RSEQ_ASM_STORE_RSEQ_CS, and RSEQ_ASM_U64_PTR into their respective ifdef regions, as the resulting documentation is clearer (and not duplicated). Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: I37c481c9eb726b49dfdf00539c42d68bb500b2c7
2024-02-28ppc: Introduce code deduplication macrosMathieu Desnoyers1-106/+69
Introduce RSEQ_ASM_U64_PTR() which expresses the type and instance of the defined variable, allowing deduplication of the following macros: - __RSEQ_ASM_DEFINE_TABLE, - RSEQ_ASM_DEFINE_EXIT_POINT. Separate load/store/op helper macros, RSEQ_ASM_STORE_RSEQ_CS, and RSEQ_ASM_U64_PTR into their respective ifdef regions, as the resulting documentation is clearer (and not duplicated). Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: I7accb4925e1e294f2d65267641fd4c8dccb4837f
2024-02-28x86: Introduce code deduplication macrosMathieu Desnoyers1-84/+45
Introduce RSEQ_ASM_U64_PTR() which expresses the type and instance of the defined variable, allowing deduplication of the following macros: - __RSEQ_ASM_DEFINE_TABLE, - RSEQ_ASM_DEFINE_EXIT_POINT. Separate RSEQ_ASM_TP_SEGMENT, RSEQ_ASM_STORE_RSEQ_CS, and RSEQ_ASM_U64_PTR into their respective ifdef regions, as the resulting documentation is clearer (and not duplicated). Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: I13e90120f9f50bb898f99f545c3eb205fe9d8437
2024-02-28mips: Reorganise code deduplication macrosMathieu Desnoyers1-19/+31
Introduce RSEQ_ASM_U64_PTR() which expresses the type and instance of the defined variable. Separate the RSEQ_ASM_LONG* macros from the 64-bit pointer types, as those two are unrelated. Having them in the same vincinity is misleading. Test on RSEQ_BITS_PER_LONG rather than _MIPS_SZLONG to make it consistent across architectures. Use both __BYTE_ORDER and __BIG_ENDIAN to detect byte order. Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: I1a65210e9ab0c00a5625c9feba7971081eb34c35
2024-02-28Re-organise public headersMichael Jeanson29-252/+441
The structure is inspired by the liburcu project. - Remove the double namespacing 'rseq/rseq-' - Add a structured architecture support layout - Add missing guards on arch headers - Split the main rseq.h header into utils.h and inject.h - Add the always_inline attribute to all static inline functions Change-Id: I6a244e5f364d28d5d8c3d426d6039c4d8aa15a36 Signed-off-by: Michael Jeanson <mjeanson@efficios.com> Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
2024-02-28Fix supported autotools versions in README.mdMichael Jeanson1-2/+2
Change-Id: I07257e51544546a8c39325dd05a0a0b99b47a5e2 Signed-off-by: Michael Jeanson <mjeanson@efficios.com> Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
2024-02-27RSEQ_ASM_DEFINE_TABLE: use parentheses around parameters on all archsMathieu Desnoyers7-8/+8
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: I09ad49a2eb132768d9fca4547d602240bc3d84a4
2024-02-27Comment s390 macros implementationMathieu Desnoyers1-19/+119
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: Ifaeeed3e7721b9c0f758abef63c286a4d22e5268
2024-02-27s390: namespace public header macros under RSEQ_ASM_Mathieu Desnoyers2-63/+63
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: I777a7b3feb9361c9dbc5327f643fc2d09aaa238e
2024-02-27Comment riscv macros implementationMathieu Desnoyers1-10/+107
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: I4299061a38443bff9ff0701b73752153d2c3df54
2024-02-27riscv: Add missing "inc" parameter to RSEQ_ASM_OP_R_DEREF_ADDVMathieu Desnoyers2-2/+2
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: Iff9fb8f46acef741a22065a6c0b976794c4606dd
2024-02-27riscv: namespace public header macros under RSEQ_ASM_Mathieu Desnoyers1-19/+19
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: If8750ceaee9ca4d30309c974e675336ea4290d19
2024-02-27mips: Move RSEQ_ASM_DEFINE_ABORT table_label parameter to 4th argumentMathieu Desnoyers2-17/+17
Makes the first 3 arguments same as other architectures. Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: Icf41ded4ce5860cf8c12a23dff1fe434210844b0
2024-02-27arm: Move RSEQ_ASM_DEFINE_ABORT table_label parameter to 4th argumentMathieu Desnoyers2-17/+17
Makes the first 3 arguments same as other architectures. Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: I855d645ec5e315ef19836adebfd2f9350a3763c5
2024-02-27Comment mips macros implementationMathieu Desnoyers1-12/+101
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: I6a1646a8cc18cf249b7323afc9ceb0600f001649
2024-02-27Comment ppc macros implementationMathieu Desnoyers1-25/+129
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: I0b51030c416d875de296ba7f4490bd59ba810f72
2024-02-27Comment arm64 macros implementationMathieu Desnoyers1-10/+93
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: Id918fa702bfbee014766f949329d1068a0404340
2024-02-27Comment arm macros implementationMathieu Desnoyers1-8/+93
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: Ibb74479b2219cf2da92a17034785f94cb7dba932
2024-02-27Comment x86 asm macros implementationMathieu Desnoyers2-43/+129
Combine common code between x86-32/x86-64. Comment asm macro implementation for both architectures. Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: If6bb763edd4821173babfbc3ac1557f4ae576a86
2024-02-27mips: Namespace type helpers under RSEQ_ASM_*Mathieu Desnoyers2-70/+70
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: If509780615849c0bf69b36a4363a6ff146b22c50
2024-02-27ppc: Rename RSEQ_ASM_OP_R_MEMCPY to RSEQ_ASM_OP_R_BYTEWISE_MEMCPYMathieu Desnoyers2-2/+2
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: I5ef55c2d23a0cd5b62c0276698a9ccf8e839fc26
2024-02-27riscv: Rename RSEQ_ASM_OP_R_BAD_MEMCPY to RSEQ_ASM_OP_R_BYTEWISE_MEMCPYMathieu Desnoyers2-2/+2
This is more descriptive. Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: I3f397cc63f60c2d806b1665de9b6bae4106f1db1
2024-02-27arm64: Rename RSEQ_ASM_OP_R_BAD_MEMCPY to RSEQ_ASM_OP_R_BYTEWISE_MEMCPYMathieu Desnoyers2-2/+2
This is more descriptive. Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: Ic5b33693ae39729859ea734721ffd323a7151c68
2024-02-27ppc: Rename RSEQ_ helpers to RSEQ_ASM_Mathieu Desnoyers1-29/+29
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: I0baec337248866688b2534164a29ab525c18265f
2024-02-27Rename RSEQ_TEMPLATE macrosMathieu Desnoyers16-153/+153
For added clarity, rename: RSEQ_TEMPLATE_CPU_ID -> RSEQ_TEMPLATE_INDEX_CPU_ID RSEQ_TEMPLATE_MM_CID -> RSEQ_TEMPLATE_INDEX_MM_CID RSEQ_TEMPLATE_CPU_ID_NONE -> RSEQ_TEMPLATE_INDEX_NONE Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: Ie3c80d15755b295543ae41b86e8879a61d7a8863
2024-02-27x86: Use RSEQ_ASM_ prefix for RSEQ_CPU_ID_OFFSET, RSEQ_CS_OFFSET, ↵Mathieu Desnoyers3-18/+18
RSEQ_MM_CID_OFFSET This makes this clearly internal and not part of the public API. Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: I695af5866ac8bb8f1b62c705df85f96edaa40fa0
2024-02-27x86-32: Remove support for old IDT WinChip and PPROMathieu Desnoyers1-26/+4
The rseq system call first appeared in Linux v4.18, which did not support the old IDT WinChip Intel clones and PentiumPro CPUs anymore. Therefore, remove CPU barrier instructions from the smp_rmb() and smp_wmb() macros given that librseq will realistically never be used on such hardware. Expect all x86-32 hardware to be TSO. Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: I07a03006e2eb8ec0928c63f16df94ad42d440b10
2024-02-27Fix: seccomp tests require shared libsMichael Jeanson2-1/+6
Disable the seccomp dependent tests when the project is built without shared library support. Change-Id: If17caf056c612a22b649281f25775f55546074f1 Signed-off-by: Michael Jeanson <mjeanson@efficios.com> Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
2024-02-27x86-64: change jnz/jz for jne/je after cmp/testMathieu Desnoyers1-22/+22
Those are the exact same Jcc instructions (they both depend on the ZF), but it's clearer to use jne/je after cmp/test rather than jnz/jz. Note that the jnz instruction is kept when it is used to check the result of a dec instruction. Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: Ibc2e5902b3047cb291d11d82b7e9dfa530a46891
2024-02-27Rename RSEQ_ASM_DEFINE_CMPFAIL to RSEQ_ASM_DEFINE_TEARDOWNMathieu Desnoyers8-25/+25
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: Ic386511c72e99b0818c6ff29fd2996d110296e59
2024-02-27Rename branch labels from cmpfail to ne/eqMathieu Desnoyers7-168/+168
This is done to match the pseudo-code. Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: I56e2bd0c838e16ef2de459da7d15493dd191df4a
2024-02-27Update rseq critical section identifiers to match pseudo-codeMathieu Desnoyers10-116/+88
Apply the following mapping: rseq_cmpeqv_storev -> rseq_load_cbne_store__ptr rseq_addv -> rseq_load_add_store__ptr rseq_cmpnev_storeoffp_load -> rseq_load_cbeq_store_add_load_store__ptr rseq_offset_deref_addv -> rseq_load_add_load_add_store__ptr rseq_cmpeqv_cmpeqv_storev -> rseq_load_cbne_load_cbne_store__ptr rseq_cmpeqv_trystorev_storev -> rseq_load_cbne_store_store__ptr rseq_cmpeqv_trymemcpy_storev -> rseq_load_cbne_memcpy_store__ptr Summary of the identifier convention: - Begin with an "rseq_" prefix, - Followed by the pseudo-code, - Followed by __ and the type (or eventually types) on which the API applies (similar to the approach taken for C++ mangling). Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: I59f251bcf177779ee5f8eb18c629dcd8bf1d633a
2024-02-27Add rseq critical section pseudocode documentationMathieu Desnoyers10-0/+185
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: Ic7bd7df3566401562a41ffc8112fe95d740c0016
2024-02-23tests: add unregistered and no syscall testsMichael Jeanson12-4/+259
Add tests to validate the state of the public symbols when no registration has occured and when the rseq syscall is unavailable. The no syscall test uses seccomp to deny access to rseq, libseccomp is added as an optionnal build dependency. Change-Id: I0bf6249fa13bd39af80c21a9d2892cd132644e69 Signed-off-by: Michael Jeanson <mjeanson@efficios.com> Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
2024-02-23fix: always set the rseq offset and flagsMichael Jeanson1-6/+6
Even when the rseq syscall is unavailable, set the __rseq_offset to the correct value as application or library code might try to check the registration status by reading the cpu_id. Change-Id: I8f579af236e17a45dfb71c3310bc41a9c17652fd Signed-off-by: Michael Jeanson <mjeanson@efficios.com> Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
2024-02-22Add 'teardown' parameter to RSEQ_ASM_DEFINE_ABORT on all archMichael Jeanson6-22/+25
For consistency, add the 'teardown' parameter to the RSEQ_ASM_DEFINE_ABORT macro on architectures where it was missing. Change-Id: I2c4bc46fc0f4a4e4ba0f704950753bf6485794d3 Signed-off-by: Michael Jeanson <mjeanson@efficios.com> Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
2024-02-22Add comments to rseq_init()Michael Jeanson1-0/+39
Change-Id: Ie81eb51d0892e7f84db5b57476c8b14c3a6f5940 Signed-off-by: Michael Jeanson <mjeanson@efficios.com> Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
2024-02-22doc: fix typo in rseq(2) man pageMichael Jeanson1-2/+1
Change-Id: I52573c813418fd6d4961e3780cfb7c054f5519f2 Signed-off-by: Michael Jeanson <mjeanson@efficios.com> Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
2024-02-22Update autotools archive macrosMichael Jeanson3-49/+102
Change-Id: I2e200e6c5e9b14e2cfeeda6319f00a4d1795f6f2 Signed-off-by: Michael Jeanson <mjeanson@efficios.com> Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
2024-02-22Clarify logic of RSEQ_ASM*_CMP* macrosMathieu Desnoyers14-154/+154
The RSEQ_ASM*_CMP* macros come from the x86 implementation which has "RSEQ_ASM_CMP_CPU_ID()" and static inline functions such as "rseq_cmpeqv_storev()". The meaning of the "cmp" here is that the critical section does _not_ abort (does not branch) if the comparison matches. But I understand how the ASM helpers that were contributed for other architectures such as "RSEQ_ASM_OP_CMPEQ()", with the same semantic of "do not branch to abort if the comparison matches" can be misleading for someone used to reading assembler on pretty much any architecture, where the conditional branch is expected to be taken if the condition matches. So what I have here in librseq is backwards. Fortunately, librseq is still just a master branch (no releases yet), and the copy in the Linux kernel selftests is internal to that selftest, so there are no stable API expectations at this stage. So I don't think the semantic of e.g. "rseq_cmpeqv_storev()" is misleading: it proceeds to do the store if the comparison matches. However, the ASM macros would benefit from a logic flip. Introduce this in a way that will allow users of the API to catch the change at compile-time. This commit applies the following remapping of the macros for added clarity: RSEQ_ASM_OP_CMPNE becomes RSEQ_ASM_OP_CBEQ (compare and branch if equal) RSEQ_ASM_OP_CMPEQ becomes RSEQ_ASM_OP_CBNE (compare and branch if not equal) RSEQ_ASM_CMP_CPU_ID becomes RSEQ_ASM_CBNE_CPU_ID (compare and branch if cpu id is not equal) Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: I13bff7e22569b11565a23aca87cf571431d57106
2024-02-22Document alignment of rseq_abi for allocated sizeMathieu Desnoyers1-0/+6
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: Ie78f475a2d6f8756b166db45d96367104871c2a8
2024-01-10Fix: do not skip !allowed_cpus for mm_cidMathieu Desnoyers2-8/+28
Indexing with mm_cid is incompatible with skipping disallowed cpumask. Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: I138882eef8a5a447e82dddeca9df73047987545f
2024-01-10Add arm64 headers to list of headers to be installedMathias Stearn1-0/+2
Currently these aren't being installed leading to compile failures on arm64 when trying to include rseq.h after a `make install`. Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: Id8bb911ad4157e710ee2731436c92756f73cc1be
2023-10-25riscv: Fix __always_inline causes duplicate inlineMathieu Desnoyers1-7/+7
In file included from /usr/include/features.h:503, from /usr/include/bits/libc-header-start.h:33, from /usr/include/stdint.h:26, from /usr/lib64/gcc/riscv64-suse-linux/13/include/stdint.h:9, from ../../../tests/unit/arch-mo.c:5, from ../../../tests/unit/arch-mo-cxx.cpp:5: ../../../include/rseq/rseq-riscv-bits.h:9:15: error: duplicate 'inline' 9 | static inline __always_inline | ^~~~~~~~~~~~~~~ Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: Iabf16e221e6a84ac31163d58c7e6edf2e87c5d18
2023-06-12Add missing copyright headersMichael Jeanson4-0/+10
Change-Id: Icb6343e541608ec00b0c86c6f6c7231abffce25e Signed-off-by: Michael Jeanson <mjeanson@efficios.com> Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
generated by cgit 1.2.3-korg (git 2.39.0) at 2024-05-16 01:12:59 +0000