mm: gup: FOLL_UNSHARE and COR faultmapcount_unshare

The primary objective of this patchset is to research a zero cons,
only pros alternative to commit
178 (17839856fd588f4ab6b789f482ed3ffd7c403e1f) and the new approach in
commit 098 (09854ba94c6aad7886996bfbee2530b3d8a7f4f4). Overall this
patchset resolves all issues described in:

	https://lkml.kernel.org/r/3ae33b08-d9ef-f846-56fb-645e3b9b4c66@redhat.com

17839856fd588f4ab6b789f482ed3ffd7c403e1f was a secure complete fix for
CVE-2020-29374 however it caused various regressions: ptrace on pmem
crashes, uffd-wp instability, reduced KSM density.

This patchset fixes CVE-2020-29374 as well as 178 did, but without
introducing those regression.

The difference between 178 and FOLL_UNSHARE/COR is that the unsharing
of the page in GUP is done without setting FOLL_WRITE.

With this solution GUP detects when the page has to be "unshared" and
in such case it sets FOLL_UNSHARE before taking any readonly GUP
pin. FOLL_UNSHARE then invokes the COR fault to execute the
"unsharing".

This ultimately allows to deliver full MM coherency to all GUP pins,
short or long term, no matter which memory is being pinned and without
requiring MMU Notifier. MMU Notifiers are still required for the GUP
pins not to disable all the advanced VM features (i.e. swapping, NUMA
balancing, KSM, memory hotunplug, compaction, CMA, etc..), but they're
not required anymore just to deliver full MM coherency.

To review the patchset it's best to start from the end of it, so from
the below matrix:

+ * The reason short term pins don't lose coherency even if they don't
+ * unshare those three kind of readonly page types, is that they will
+ * refresh the physaddr from the pgtable before each DMA (or even
+ * if they issue many DMA from the same GUP they are ok that the
+ * snapshot of the data payload is taken at GUP time and that the
+ * coherency with the CPU may be temorarily lost until the next GUP
+ * invocation). So the MM coherency won't break for them even if a COW
+ * happens after the short term pin has been taken by GUP.
+ *
+ * The COR fault will then disambiguate a long term pin requested with
+ * FOLL_MM_SYNC from a short term pin using the
+ * FAULT_FLAG_UNSHARE_MM_SYNC flag.
+ *
+ * The below table assumes !FOLL_WRITE (readonly GUP pins) and answers
+ * when FOLL_UNSHARE is activated during GUP to generate exclusive
+ * anonymous memory mapped readonly through the COR fault.
+ *
+ * page/mapping type	 | short term   | FOLL_MM_SYNC
+ * ------------------------------------------------------
+ * PageAnon && !PageKsm  | mapcount > 1 | mapcount > 1
+ * PageAnon && PageKsm   | never	| always
+ * zeropage		 | never	| always
+ * MAP_PRIVATE !PageAnon | never	| always
+ * MAP_SHARED  !PageAnon | never	| never

FOLL_MM_SYNC in the above code comment can be interpreted as
FOLL_LONGTERM for simplicity (it's supposed to be set along with it).

By the time the COW fault triggers, it's not possible anymore to
disambiguate what an extra refcount on the page actually means. It
could be a GUP pin or a temporary refcount (like a speculative
pagecache lookup) or even a swapcache refcount. The GUP pin
information is lost by the time GUP returns, so it should be acted
upon within GUP itself, in a way that will prevent the COW fault to
ever encounter a COW page that was GUP pinned by another process.

This patchset can be split in two parts. The objective of the first
part until the patch introducing FOLL_MM_SYNC (not included) is:

- fix CVE-2020-29374: the GUP security issue that allows the fork()
  child to read the memory of the parent.

- fix the ABI break for sub-PAGE_SIZE short term GUP pins using
  FOLL_GET, as O_DIRECT read(), showing userland MM
  corruption (despite the read syscall returning success) in
  combination with swap or clear_refs.

- fix FOLL_LONGTERM/RDMA+swap or FOLL_LONGTERM+clear_refs silent
  memory corruption that may happen if pin-fast elevates the
  page_count after page_maybe_dma_pinned() already returned a false
  negative.

- fix the ABI break for readonly long term GUP pins that may lose
  coherency with the MM (which can result in silent data loss on the
  receive end) if the pinned pages are wrprotected in the pagetable.

  Example: readonly FOLL_LONGTERM pins used to deliver dirty data
  written by the CPU and tracked with clear_refs.

  Another example: mprotect(PROT_READ) followed by
  mprotect(PROT_READ|PROT_WRITE) on pages that have been previously
  pinned with a readonly FOLL_LONGTERM.

- clear_refs and uffd-wp both optimized not to require the
  mmap_write_lock.

- pmbench swapcache 24.9% performance boost against the
  09854ba94c6aad7886996bfbee2530b3d8a7f4f4~ baseline (not only a 23.8%
  improvement provided by f4c4a3f48480730214c4f02ffa480f6bf5b0718f if
  applied on top of 09854ba94c6aad7886996bfbee2530b3d8a7f4f4). It
  avoids not only generating orphaned swapcache, but also the spurious
  allocation and copies that generated it in the first place.

Continuing from the patch introducing FOLL_MM_SYNC (included):

- allows to mark writable all exclusive COW anon pages during mprotect
  to avoid spurious COW faults, even if the pages are GUP pinned.

- Reuse the previously introduced FOLL_UNSHARE COR logic to deliver
  FOLL_LONGTERM MM coherency by the specs to readonly long term
  pins (FOLL_MM_SYNC supersedes FOLL_WRITE|FOLL_FORCE).

- Other THP mapcount and GUP related optimizations.

-----------------------------------------------------------------------

Patches originated in this patchset successfully merged upstream:

292648ac5cf1 ("mm: gup: allow FOLL_PIN to scale in SMP")
a458b76a4171 ("mm: gup: pack has_pinned in MMF_HAS_PINNED")

which provided these benefits to upstream:

- >4000% SMP scalability performance improvement to
  pin_user_pages_fast() with 1 thread per CPU and 2 NUMA nodes with
  64 cores each.

Changelog:

785f2d5f5a1a8b239ba89c3a1555aed9085644e9

- sync with main-5.15.y

488c4509c7d892db6b5c7170095bc6bf651beb2b

- Cleanup FOLL_UNSHARE definition from code and commit headers and
  other no-op cleanups. It's a further sync-up with the cleanups from
  the v1 "mm: COW fixes part 1: fix the COW security issue for THP and
  hugetlb" submit.

1b972f2a40824f35709dd8b730aedfd0eaff4ff8

- Tentative fix for the false positive BUILD_BUG_ON build error on some
  arches reported by the kernel test robot.

e9a49c91f01d445e3be0b527a05eb097b81b8741

- The mprotect optimization that was proposed upstream to skip
  spurious COW faults had a bug in not checking the swapcount which
  could result in erroneously skipping the COW fault with swap
  enabled. This implementation inherited the same bug that the
  original upstream posted patch had. The bug has been found by source
  review and it has been fixed: in this implementation the swapcount
  is now taken into account as required for safety.

3346c14187913954a0e50d0069b231b55568a6af

- optimized wp_page_unshare() with can_read_pin_swap_page(), in
  addition this change is a dependency for the PageKsm FOLL_MM_SYNC
  rework.

- reworked from scratch PageKsm FOLL_MM_SYNC using
  can_read_pin_swap_page(). Enforcing that no FOLL_LONGTERM read pin
  can be ever taken on any PageKsm feels simpler in comparison to
  enforcing no PageAnon can be converted to PageKsm if there's any
  outstanding pin and that no wrprotected PageAnon can be replaced by
  an equal PageKsm if the PageAnon had any outstanding FOLL_LONGTERM
  pins. Both guarantees are required for FOLL_MM_SYNC to deliver
  full synchronicity to FOLL_LONGTERM pins on VM_MERGEABLE vmas too.

6865ab35a3b9720b1820c8016f20c589a0293e0e

- gup_must_unshare() optimized with can_read_pin_swap_page().

- added the page lock in the hugetlbfs gup_must_unshare() path to
  protect against page migration. It'd be ideal if page migration could
  be improved to count how many migration entries it installed and then
  drop the mapcount accordingly only after the refcount freezing.

- Improved FOLL_MM_SYNC for PageKsm: KSM code should cooperate with
  GUP and make sure to never de-dup pages with GUP pins. GUP already does
  its part in unsharing PageKsm pages with the COR fault before taking
  readonly FOLL_LONGTERM pins (with FOLL_MM_SYNC implicitly set).

d0de2b3fef4a4e94207020c560158b2ccc97baa0

- More noop cleanups.

- Added a missing update_mmu_tlb() which is also a noop for all arches
  except mips.

5db5863e2c773f347ded84c98d6f28f692cd2def

- A solution based on the FOLL_UNSHARE+COR solution that originated in
  this tree has been proposed upstream and the review showed the
  gup_must_unshare() didn't properly take into account the swapcount.

  The lack of swapcount calculation reported upstream is a minor
  implementation issue and requires no change in design to fix. In
  fact it has been fixed in less than 48 hours as demonstrated by this
  quick hotfix update.

  It's worth pointing out that the lack of swapcount calculation in
  the previous version caused zero regressions compared to upstream
  v5.7 and in fact the previous version was preferable than v5.7.

  As opposed upstream still randomly corrupts memory if swap is
  enabled with O_DIRECT + swap if using 64k PAGE SIZE on aarch64 and a
  4k db blocksize, with io_uring and all FOLL_LONGTERM and causes
  various horizontal regressions (for example all swapcache is COWed
  unconditionally even if it's exclusive).

  At the time of this writing, this is the only known solution that
  resolves all known security issues and that introduces zero user ABI
  regression compared to v5.7 and that retains the full power of the
  MM.

  In fact this goes beyond what v5.7 could do: with FOLL_MM_SYNC for
  the first time this solution provides full POSIX semantics to all
  FOLL_LONGTERM and short term pins by leveraging the COR (Copy On
  Read) fault.

bbb329566757030dec3960ffbc8465a8092a17fd

- Peter Xu discovered that the THP path of __page_mapcount was reading
  the first tail page instead of the right tailpage in a doublemap.
  This has been corrected.

- David Hildenbrand reported that __page_mapcount and gup_must_unshare
  shared some code paths between THP and hugetlbfs, but the mapcount
  seqcount wasn't initialized in hugetlbfs which could result in a
  softlockup. This has been corrected and the hugetlbfs paths in
  __page_mapcount and gup_must_unshare don't share the same code paths
  anymore.

- Merged a permutation from David Hildenbrand that simplifies
  __split_huge_pmd_locked() and reduces the
  page_trans_huge_mapcount_lock() hold time as well.

- Merged FOLL_NOUNSHARE from David Hildenbrand "deactivate" the COR
  fault in follow_page(). follow_page() is special because the kernel
  is the "user" and the kernel intends to work on the real thing, not
  on the post-COR copy. Obtaining a (post-COR) copy of the page is
  functionally harmless from the userland point of view, but it'd
  defeat various kernel MM optimizations.

b2bf3165cf3269a2b6b4e52a14f275b212f0bbe9

- rebase on top of THP folio.

89ab75761d378e4f3c165ff8c125f7bf9f433ab4

- added the COR fault and the FAULT_FLAG_UNSHARE support to hugetlbfs.

3e14641933648eecdb49c3f949d09097e733413b

- Improved the 3771dc26618494d2fca1f8489cc1581a63a51ce8 commit header.

236f0d6d3eb19f81d53352dcb7f73d5cbb1ac1eb

- cleanup gup_must_unshare(): added is_fast_only_in_irq() to document
  and deduplicate the irq_count() check.

1bc34a65718685b8caa980185de49dc04377eacd

- Added feb889fb40fafc6933339cf1cca8f770126819fb to the list of
  reverts since it's unnecessary after reverting
  09854ba94c6aad7886996bfbee2530b3d8a7f4f4.

- Documented more details on the SMP race against pin-fast of
  feb889fb40fafc6933339cf1cca8f770126819fb and
  9348b73c2e1bfea74ccd4a44fb4ccc7276ab9623 at the end of the commit
  header of 5f3f91f23e41359338a41991fe19e4735d7e56e4 ("mm: COW:
  restore full accuracy in page reuse").

Signed-off-by: Andrea Arcangeli <aarcange@redhat.com>

0 files changed, 0 insertions, 0 deletions