diff options
| author | Junio C Hamano <gitster@pobox.com> | 2022-06-03 15:26:23 -0700 |
|---|---|---|
| committer | Junio C Hamano <gitster@pobox.com> | 2022-06-03 15:26:23 -0700 |
| commit | c926f62d061511274a5bfe2b3653a1f729b97a04 (patch) | |
| tree | 11b6cecfee8dcaeeb11a6fba485f60001e103778 | |
| parent | 5e833e00b1d044e2b5830a3fb639718bcc4c1811 (diff) | |
| download | git-htmldocs-c926f62d061511274a5bfe2b3653a1f729b97a04.tar.gz | |
Autogenerated HTML docs for v2.36.1-299-gab336
32 files changed, 1875 insertions, 194 deletions
diff --git a/RelNotes/2.37.0.txt b/RelNotes/2.37.0.txt index a619b5f79..345f1b9e6 100644 --- a/RelNotes/2.37.0.txt +++ b/RelNotes/2.37.0.txt @@ -38,6 +38,20 @@ UI, Workflows & Features testing; the reimplementation is now exposed to general public by default. + * Deprecate non-cone mode of the sparse-checkout feature. + + * Introduce a filesystem-dependent mechanism to optimize the way the + bits for many loose object files are ensured to hit the disk + platter. + + * The "do not remove the directory the user started Git in" logic, + when Git cannot tell where that directory is, is disabled. Earlier + we refused to run in such a case. + + * A mechanism to pack unreachable objects into a "cruft pack", + instead of ejecting them into loose form to be reclaimed later, has + been introduced. + Performance, Internal Implementation, Development Support etc. @@ -54,6 +68,16 @@ Performance, Internal Implementation, Development Support etc. comparison between a pointer and NULL, and applies the clean-up to the maintenance track. + * Preliminary code refactoring around transport and bundle code. + + * "sparse-checkout" learns to work better with the sparse-index + feature. + + * A workflow change for translators are being proposed. git.pot is + no longer version controlled and it is local responsibility of + translaters to generate it. + + Fixes since v2.36 ----------------- @@ -196,6 +220,18 @@ Fixes since v2.36 * Some real problems noticed by gcc 12 have been fixed, while false positives have been worked around. + * Update the version of FreeBSD image used in Cirrus CI. + (merge c58bebd4c6 pb/use-freebsd-12.3-in-cirrus-ci later to maint). + + * The multi-pack-index code did not protect the packfile it is going + to depend on from getting removed while in use, which has been + corrected. + (merge 4090511e40 tb/midx-race-in-pack-objects later to maint). + + * Teach "git repack --geometric" work better with "--keep-pack" and + avoid corrupting the repository when packsize limit is used. + (merge 66731ff921 tb/geom-repack-with-keep-and-max later to maint). + * Other code cleanup, docfix, build fix, etc. (merge e6b2582da3 cm/reftable-0-length-memset later to maint). (merge 0b75e5bf22 ab/misc-cleanup later to maint). diff --git a/git-config.html b/git-config.html index 992a69f39..c1a39a66b 100644 --- a/git-config.html +++ b/git-config.html @@ -3178,6 +3178,17 @@ core.fsyncMethod durable in the event of a system crash. This is the default mode on macOS.
</p>
</li>
+<li>
+<p>
+<code>batch</code> enables a mode that uses writeout-only flushes to stage multiple
+ updates in the disk writeback cache and then does a single full fsync of
+ a dummy file to trigger the disk cache flush at the end of the operation.
+</p>
+<div class="paragraph"><p>Currently <code>batch</code> mode only applies to loose-object files. Other repository
+data is made durable as if <code>fsync</code> was specified. This mode is expected to
+be as safe as <code>fsync</code> on macOS for repos stored on HFS+ or APFS filesystems
+and on Windows for repos stored on NTFS or ReFS filesystems.</p></div>
+</li>
</ul></div>
</dd>
<dt class="hdlist1">
@@ -3299,8 +3310,10 @@ core.sparseCheckoutCone <dd>
<p>
Enables the "cone mode" of the sparse checkout feature. When the
- sparse-checkout file contains a limited set of patterns, then this
- mode provides significant performance advantages. See
+ sparse-checkout file contains a limited set of patterns, this
+ mode provides significant performance advantages. The "non
+ cone mode" can be requested to allow specifying a more flexible
+ patterns by setting this variable to <em>false</em>. See
<a href="git-sparse-checkout.html">git-sparse-checkout(1)</a> for more information.
</p>
</dd>
@@ -5863,17 +5876,29 @@ gc.packRefs </p>
</dd>
<dt class="hdlist1">
+gc.cruftPacks
+</dt>
+<dd>
+<p>
+ Store unreachable objects in a cruft pack (see
+ <a href="git-repack.html">git-repack(1)</a>) instead of as loose objects. The default
+ is <code>false</code>.
+</p>
+</dd>
+<dt class="hdlist1">
gc.pruneExpire
</dt>
<dd>
<p>
- When <em>git gc</em> is run, it will call <em>prune --expire 2.weeks.ago</em>.
- Override the grace period with this config variable. The value
- "now" may be used to disable this grace period and always prune
- unreachable objects immediately, or "never" may be used to
- suppress pruning. This feature helps prevent corruption when
- <em>git gc</em> runs concurrently with another process writing to the
- repository; see the "NOTES" section of <a href="git-gc.html">git-gc(1)</a>.
+ When <em>git gc</em> is run, it will call <em>prune --expire 2.weeks.ago</em>
+ (and <em>repack --cruft --cruft-expiration 2.weeks.ago</em> if using
+ cruft packs via <code>gc.cruftPacks</code> or <code>--cruft</code>). Override the
+ grace period with this config variable. The value "now" may be
+ used to disable this grace period and always prune unreachable
+ objects immediately, or "never" may be used to suppress pruning.
+ This feature helps prevent corruption when <em>git gc</em> runs
+ concurrently with another process writing to the repository; see
+ the "NOTES" section of <a href="git-gc.html">git-gc(1)</a>.
</p>
</dd>
<dt class="hdlist1">
@@ -9702,6 +9727,26 @@ repack.updateServerInfo </p>
</dd>
<dt class="hdlist1">
+repack.cruftWindow
+</dt>
+<dt class="hdlist1">
+repack.cruftWindowMemory
+</dt>
+<dt class="hdlist1">
+repack.cruftDepth
+</dt>
+<dt class="hdlist1">
+repack.cruftThreads
+</dt>
+<dd>
+<p>
+ Parameters used by <a href="git-pack-objects.html">git-pack-objects(1)</a> when generating
+ a cruft pack and the respective parameters are not given over
+ the command line. See similarly named <code>pack.*</code> configuration
+ variables for defaults and meaning.
+</p>
+</dd>
+<dt class="hdlist1">
rerere.autoUpdate
</dt>
<dd>
diff --git a/git-gc.html b/git-gc.html index 843933db6..bfd91cae6 100644 --- a/git-gc.html +++ b/git-gc.html @@ -806,6 +806,16 @@ other housekeeping tasks (e.g. rerere, working trees, reflog…) will be performed as well.</p></div>
</dd>
<dt class="hdlist1">
+--cruft
+</dt>
+<dd>
+<p>
+ When expiring unreachable objects, pack them separately into a
+ cruft pack instead of storing the loose objects as loose
+ objects.
+</p>
+</dd>
+<dt class="hdlist1">
--prune=<date>
</dt>
<dd>
@@ -1009,17 +1019,29 @@ gc.packRefs </p>
</dd>
<dt class="hdlist1">
+gc.cruftPacks
+</dt>
+<dd>
+<p>
+ Store unreachable objects in a cruft pack (see
+ <a href="git-repack.html">git-repack(1)</a>) instead of as loose objects. The default
+ is <code>false</code>.
+</p>
+</dd>
+<dt class="hdlist1">
gc.pruneExpire
</dt>
<dd>
<p>
- When <em>git gc</em> is run, it will call <em>prune --expire 2.weeks.ago</em>.
- Override the grace period with this config variable. The value
- "now" may be used to disable this grace period and always prune
- unreachable objects immediately, or "never" may be used to
- suppress pruning. This feature helps prevent corruption when
- <em>git gc</em> runs concurrently with another process writing to the
- repository; see the "NOTES" section of <a href="git-gc.html">git-gc(1)</a>.
+ When <em>git gc</em> is run, it will call <em>prune --expire 2.weeks.ago</em>
+ (and <em>repack --cruft --cruft-expiration 2.weeks.ago</em> if using
+ cruft packs via <code>gc.cruftPacks</code> or <code>--cruft</code>). Override the
+ grace period with this config variable. The value "now" may be
+ used to disable this grace period and always prune unreachable
+ objects immediately, or "never" may be used to suppress pruning.
+ This feature helps prevent corruption when <em>git gc</em> runs
+ concurrently with another process writing to the repository; see
+ the "NOTES" section of <a href="git-gc.html">git-gc(1)</a>.
</p>
</dd>
<dt class="hdlist1">
@@ -1165,7 +1187,7 @@ seems to be low in practice).</p></div> <div id="footer">
<div id="footer-text">
Last updated
- 2021-02-26 23:35:19 PST
+ 2022-06-03 15:24:31 PDT
</div>
</div>
</body>
diff --git a/git-gc.txt b/git-gc.txt index 853967dea..ba4e67700 100644 --- a/git-gc.txt +++ b/git-gc.txt @@ -54,6 +54,11 @@ other housekeeping tasks (e.g. rerere, working trees, reflog...) will be performed as well. +--cruft:: + When expiring unreachable objects, pack them separately into a + cruft pack instead of storing the loose objects as loose + objects. + --prune=<date>:: Prune loose objects older than date (default is 2 weeks ago, overridable by the config variable `gc.pruneExpire`). diff --git a/git-pack-objects.html b/git-pack-objects.html index 776a12013..f592c279e 100644 --- a/git-pack-objects.html +++ b/git-pack-objects.html @@ -753,6 +753,7 @@ git-pack-objects(1) Manual Page [--no-reuse-delta] [--delta-base-offset] [--non-empty]
[--local] [--incremental] [--window=<n>] [--depth=<n>]
[--revs [--unpacked | --all]] [--keep-pack=<pack-name>]
+ [--cruft] [--cruft-expiration=<time>]
[--stdout [--filter=<filter-spec>] | <base-name>]
[--shallow] [--keep-true-parents] [--[no-]sparse] < <object-list></pre>
<div class="attribution">
@@ -871,6 +872,43 @@ base-name <code>--all</code>), with the exception of <code>--unpacked</code>, which is compatible.</p></div>
</dd>
<dt class="hdlist1">
+--cruft
+</dt>
+<dd>
+<p>
+ Packs unreachable objects into a separate "cruft" pack, denoted
+ by the existence of a <code>.mtimes</code> file. Typically used by <code>git
+ repack --cruft</code>. Callers provide a list of pack names and
+ indicate which packs will remain in the repository, along with
+ which packs will be deleted (indicated by the <code>-</code> prefix). The
+ contents of the cruft pack are all objects not contained in the
+ surviving packs which have not exceeded the grace period (see
+ <code>--cruft-expiration</code> below), or which have exceeded the grace
+ period, but are reachable from an other object which hasn’t.
+</p>
+<div class="paragraph"><p>When the input lists a pack containing all reachable objects (and lists
+all other packs as pending deletion), the corresponding cruft pack will
+contain all unreachable objects (with mtime newer than the
+<code>--cruft-expiration</code>) along with any unreachable objects whose mtime is
+older than the <code>--cruft-expiration</code>, but are reachable from an
+unreachable object whose mtime is newer than the <code>--cruft-expiration</code>).</p></div>
+<div class="paragraph"><p>Incompatible with <code>--unpack-unreachable</code>, <code>--keep-unreachable</code>,
+<code>--pack-loose-unreachable</code>, <code>--stdin-packs</code>, as well as any other
+options which imply <code>--revs</code>. Also incompatible with <code>--max-pack-size</code>;
+when this option is set, the maximum pack size is not inferred from
+<code>pack.packSizeLimit</code>.</p></div>
+</dd>
+<dt class="hdlist1">
+--cruft-expiration=<approxidate>
+</dt>
+<dd>
+<p>
+ If specified, objects are eliminated from the cruft pack if they
+ have an mtime older than <code><approxidate></code>. If unspecified (and
+ given <code>--cruft</code>), then no objects are eliminated.
+</p>
+</dd>
+<dt class="hdlist1">
--window=<n>
</dt>
<dt class="hdlist1">
@@ -1352,7 +1390,7 @@ attribute <code>delta</code> set to false.</p></div> <div id="footer">
<div id="footer-text">
Last updated
- 2021-12-10 14:52:02 PST
+ 2022-06-03 15:24:31 PDT
</div>
</div>
</body>
diff --git a/git-pack-objects.txt b/git-pack-objects.txt index f8344e1e5..a9995a932 100644 --- a/git-pack-objects.txt +++ b/git-pack-objects.txt @@ -13,6 +13,7 @@ SYNOPSIS [--no-reuse-delta] [--delta-base-offset] [--non-empty] [--local] [--incremental] [--window=<n>] [--depth=<n>] [--revs [--unpacked | --all]] [--keep-pack=<pack-name>] + [--cruft] [--cruft-expiration=<time>] [--stdout [--filter=<filter-spec>] | <base-name>] [--shallow] [--keep-true-parents] [--[no-]sparse] < <object-list> @@ -95,6 +96,35 @@ base-name:: Incompatible with `--revs`, or options that imply `--revs` (such as `--all`), with the exception of `--unpacked`, which is compatible. +--cruft:: + Packs unreachable objects into a separate "cruft" pack, denoted + by the existence of a `.mtimes` file. Typically used by `git + repack --cruft`. Callers provide a list of pack names and + indicate which packs will remain in the repository, along with + which packs will be deleted (indicated by the `-` prefix). The + contents of the cruft pack are all objects not contained in the + surviving packs which have not exceeded the grace period (see + `--cruft-expiration` below), or which have exceeded the grace + period, but are reachable from an other object which hasn't. ++ +When the input lists a pack containing all reachable objects (and lists +all other packs as pending deletion), the corresponding cruft pack will +contain all unreachable objects (with mtime newer than the +`--cruft-expiration`) along with any unreachable objects whose mtime is +older than the `--cruft-expiration`, but are reachable from an +unreachable object whose mtime is newer than the `--cruft-expiration`). ++ +Incompatible with `--unpack-unreachable`, `--keep-unreachable`, +`--pack-loose-unreachable`, `--stdin-packs`, as well as any other +options which imply `--revs`. Also incompatible with `--max-pack-size`; +when this option is set, the maximum pack size is not inferred from +`pack.packSizeLimit`. + +--cruft-expiration=<approxidate>:: + If specified, objects are eliminated from the cruft pack if they + have an mtime older than `<approxidate>`. If unspecified (and + given `--cruft`), then no objects are eliminated. + --window=<n>:: --depth=<n>:: These two options affect how the objects contained in diff --git a/git-read-tree.html b/git-read-tree.html index 53bbfcfc7..0f6d590bb 100644 --- a/git-read-tree.html +++ b/git-read-tree.html @@ -1230,10 +1230,13 @@ have finished your work-in-progress), attempt the merge again.</p></div> <div class="sect1">
<h2 id="_sparse_checkout">SPARSE CHECKOUT</h2>
<div class="sectionbody">
-<div class="paragraph"><p>Note: The <code>update-index</code> and <code>read-tree</code> primitives for supporting the
-skip-worktree bit predated the introduction of
-<a href="git-sparse-checkout.html">git-sparse-checkout(1)</a>. Users are encouraged to use
-<code>sparse-checkout</code> in preference to these low-level primitives.</p></div>
+<div class="paragraph"><p>Note: The skip-worktree capabilities in <a href="git-update-index.html">git-update-index(1)</a>
+and <code>read-tree</code> predated the introduction of
+<a href="git-sparse-checkout.html">git-sparse-checkout(1)</a>. Users are encouraged to use the
+<code>sparse-checkout</code> command in preference to these plumbing commands for
+sparse-checkout/skip-worktree related needs. However, the information
+below might be useful to users trying to understand the pattern style
+used in non-cone mode of the <code>sparse-checkout</code> command.</p></div>
<div class="paragraph"><p>"Sparse checkout" allows populating the working directory sparsely.
It uses the skip-worktree bit (see <a href="git-update-index.html">git-update-index(1)</a>) to
tell Git whether a file in the working directory is worth looking at.</p></div>
@@ -1291,7 +1294,7 @@ support.</p></div> <div id="footer">
<div id="footer-text">
Last updated
- 2022-03-14 14:16:56 PDT
+ 2022-06-03 15:24:31 PDT
</div>
</div>
</body>
diff --git a/git-read-tree.txt b/git-read-tree.txt index a5356a230..b9bfdc0a3 100644 --- a/git-read-tree.txt +++ b/git-read-tree.txt @@ -375,10 +375,13 @@ have finished your work-in-progress), attempt the merge again. SPARSE CHECKOUT --------------- -Note: The `update-index` and `read-tree` primitives for supporting the -skip-worktree bit predated the introduction of -linkgit:git-sparse-checkout[1]. Users are encouraged to use -`sparse-checkout` in preference to these low-level primitives. +Note: The skip-worktree capabilities in linkgit:git-update-index[1] +and `read-tree` predated the introduction of +linkgit:git-sparse-checkout[1]. Users are encouraged to use the +`sparse-checkout` command in preference to these plumbing commands for +sparse-checkout/skip-worktree related needs. However, the information +below might be useful to users trying to understand the pattern style +used in non-cone mode of the `sparse-checkout` command. "Sparse checkout" allows populating the working directory sparsely. It uses the skip-worktree bit (see linkgit:git-update-index[1]) to diff --git a/git-repack.html b/git-repack.html index 4d66f0343..f77c7bfd4 100644 --- a/git-repack.html +++ b/git-repack.html @@ -820,6 +820,27 @@ to the new separate pack will be written.</p></div> </p>
</dd>
<dt class="hdlist1">
+--cruft
+</dt>
+<dd>
+<p>
+ Same as <code>-a</code>, unless <code>-d</code> is used. Then any unreachable objects
+ are packed into a separate cruft pack. Unreachable objects can
+ be pruned using the normal expiry rules with the next <code>git gc</code>
+ invocation (see <a href="git-gc.html">git-gc(1)</a>). Incompatible with <code>-k</code>.
+</p>
+</dd>
+<dt class="hdlist1">
+--cruft-expiration=<approxidate>
+</dt>
+<dd>
+<p>
+ Expire unreachable objects older than <code><approxidate></code>
+ immediately instead of waiting for the next <code>git gc</code> invocation.
+ Only useful with <code>--cruft -d</code>.
+</p>
+</dd>
+<dt class="hdlist1">
-l
</dt>
<dd>
@@ -1091,7 +1112,7 @@ attribute <code>delta</code> set to false.</p></div> <div id="footer">
<div id="footer-text">
Last updated
- 2022-01-05 14:13:29 PST
+ 2022-06-03 15:24:31 PDT
</div>
</div>
</body>
diff --git a/git-repack.txt b/git-repack.txt index ee30edc17..0bf13893d 100644 --- a/git-repack.txt +++ b/git-repack.txt @@ -63,6 +63,17 @@ to the new separate pack will be written. Also run 'git prune-packed' to remove redundant loose object files. +--cruft:: + Same as `-a`, unless `-d` is used. Then any unreachable objects + are packed into a separate cruft pack. Unreachable objects can + be pruned using the normal expiry rules with the next `git gc` + invocation (see linkgit:git-gc[1]). Incompatible with `-k`. + +--cruft-expiration=<approxidate>:: + Expire unreachable objects older than `<approxidate>` + immediately instead of waiting for the next `git gc` invocation. + Only useful with `--cruft -d`. + -l:: Pass the `--local` option to 'git pack-objects'. See linkgit:git-pack-objects[1]. diff --git a/git-sparse-checkout.html b/git-sparse-checkout.html index 089fbe6d7..799c017c4 100644 --- a/git-sparse-checkout.html +++ b/git-sparse-checkout.html @@ -757,14 +757,14 @@ git-sparse-checkout(1) Manual Page <div class="sect1">
<h2 id="_description">DESCRIPTION</h2>
<div class="sectionbody">
-<div class="paragraph"><p>This command is used to create sparse checkouts, which means that it
-changes the working tree from having all tracked files present, to only
-have a subset of them. It can also switch which subset of files are
-present, or undo and go back to having all tracked files present in the
-working copy.</p></div>
+<div class="paragraph"><p>This command is used to create sparse checkouts, which change the
+working tree from having all tracked files present to only having a
+subset of those files. It can also switch which subset of files are
+present, or undo and go back to having all tracked files present in
+the working copy.</p></div>
<div class="paragraph"><p>The subset of files is chosen by providing a list of directories in
-cone mode (which is recommended), or by providing a list of patterns
-in non-cone mode.</p></div>
+cone mode (the default), or by providing a list of patterns in
+non-cone mode.</p></div>
<div class="paragraph"><p>When in a sparse-checkout, other Git commands behave a bit differently.
For example, switching branches will not update paths outside the
sparse-checkout directories/patterns, and <code>git commit -a</code> will not record
@@ -794,9 +794,9 @@ THE FUTURE.</p></div> Enable the necessary sparse-checkout config settings
(<code>core.sparseCheckout</code>, <code>core.sparseCheckoutCone</code>, and
<code>index.sparse</code>) if they are not already set to the desired values,
- and write a set of patterns to the sparse-checkout file from the
- list of arguments following the <em>set</em> subcommand. Update the
- working directory to match the new patterns.
+ populate the sparse-checkout file from the list of arguments
+ following the <em>set</em> subcommand, and update the working directory to
+ match.
</p>
<div class="paragraph"><p>To ensure that adjusting the sparse-checkout settings within a worktree
does not alter the sparse-checkout settings in other worktrees, the <em>set</em>
@@ -808,21 +808,19 @@ file. See <a href="git-worktree.html">git-worktree(1)</a> and the documentation <div class="paragraph"><p>When the <code>--stdin</code> option is provided, the directories or patterns are
read from standard in as a newline-delimited list instead of from the
arguments.</p></div>
-<div class="paragraph"><p>When <code>--cone</code> is passed or <code>core.sparseCheckoutCone</code> is enabled, the
-input list is considered a list of directories. This allows for
-better performance with a limited set of patterns (see <em>CONE PATTERN
-SET</em> below). The input format matches the output of <code>git ls-tree
---name-only</code>. This includes interpreting pathnames that begin with a
-double quote (") as C-style quoted strings. Note that the set command
-will write patterns to the sparse-checkout file to include all files
-contained in those directories (recursively) as well as files that are
-siblings of ancestor directories. This may become the default in the
-future; --no-cone can be passed to request non-cone mode.</p></div>
-<div class="paragraph"><p>When <code>--no-cone</code> is passed or <code>core.sparseCheckoutCone</code> is not enabled,
-the input list is considered a list of patterns. This mode is harder
-to use and less performant, and is thus not recommended. See the
-"Sparse Checkout" section of <a href="git-read-tree.html">git-read-tree(1)</a> and the "Pattern
-Set" sections below for more details.</p></div>
+<div class="paragraph"><p>By default, the input list is considered a list of directories, matching
+the output of <code>git ls-tree -d --name-only</code>. This includes interpreting
+pathnames that begin with a double quote (") as C-style quoted strings.
+Note that all files under the specified directories (at any depth) will
+be included in the sparse checkout, as well as files that are siblings
+of either the given directory or any of its ancestors (see <em>CONE PATTERN
+SET</em> below for more details). In the past, this was not the default,
+and <code>--cone</code> needed to be specified or <code>core.sparseCheckoutCone</code> needed
+to be enabled.</p></div>
+<div class="paragraph"><p>When <code>--no-cone</code> is passed, the input list is considered a list of
+patterns. This mode has a number of drawbacks, including not working
+with some options like <code>--sparse-index</code>. As explained in the
+"Non-cone Problems" section below, we do not recommend using it.</p></div>
<div class="paragraph"><p>Use the <code>--[no-]sparse-index</code> option to use a sparse index (the
default is to not use it). A sparse index reduces the size of the
index to be more closely aligned with your sparse-checkout
@@ -903,7 +901,65 @@ decreased in utility.</p></div> </div>
</div>
<div class="sect1">
-<h2 id="_sparse_checkout">SPARSE CHECKOUT</h2>
+<h2 id="_examples">EXAMPLES</h2>
+<div class="sectionbody">
+<div class="dlist"><dl>
+<dt class="hdlist1">
+<code>git sparse-checkout set MY/DIR1 SUB/DIR2</code>
+</dt>
+<dd>
+<p>
+ Change to a sparse checkout with all files (at any depth) under
+ MY/DIR1/ and SUB/DIR2/ present in the working copy (plus all
+ files immediately under MY/ and SUB/ and the toplevel
+ directory). If already in a sparse checkout, change which files
+ are present in the working copy to this new selection. Note
+ that this command will also delete all ignored files in any
+ directory that no longer has either tracked or
+ non-ignored-untracked files present.
+</p>
+</dd>
+<dt class="hdlist1">
+<code>git sparse-checkout disable</code>
+</dt>
+<dd>
+<p>
+ Repopulate the working directory with all files, disabling sparse
+ checkouts.
+</p>
+</dd>
+<dt class="hdlist1">
+<code>git sparse-checkout add SOME/DIR/ECTORY</code>
+</dt>
+<dd>
+<p>
+ Add all files under SOME/DIR/ECTORY/ (at any depth) to the
+ sparse checkout, as well as all files immediately under
+ SOME/DIR/ and immediately under SOME/. Must already be in a
+ sparse checkout before using this command.
+</p>
+</dd>
+<dt class="hdlist1">
+<code>git sparse-checkout reapply</code>
+</dt>
+<dd>
+<p>
+ It is possible for commands to update the working tree in a
+ way that does not respect the selected sparsity directories.
+ This can come from tools external to Git writing files, or
+ even affect Git commands because of either special cases (such
+ as hitting conflicts when merging/rebasing), or because some
+ commands didn’t fully support sparse checkouts (e.g. the old
+ <code>recursive</code> merge backend had only limited support). This
+ command reapplies the existing sparse directory specifications
+ to make the working directory match.
+</p>
+</dd>
+</dl></div>
+</div>
+</div>
+<div class="sect1">
+<h2 id="_internals_8201_8212_8201_sparse_checkout">INTERNALS — SPARSE CHECKOUT</h2>
<div class="sectionbody">
<div class="paragraph"><p>"Sparse checkout" allows populating the working directory sparsely. It
uses the skip-worktree bit (see <a href="git-update-index.html">git-update-index(1)</a>) to tell Git
@@ -918,36 +974,212 @@ skip-worktree reference bitmap. When Git updates the working directory, it updates the skip-worktree bits in the index based
on this file. The files matching the patterns in the file will
appear in the working directory, and the rest will not.</p></div>
-<div class="paragraph"><p>To enable the sparse-checkout feature, run <code>git sparse-checkout set</code> to
-set the patterns you want to use.</p></div>
-<div class="paragraph"><p>To repopulate the working directory with all files, use the
-<code>git sparse-checkout disable</code> command.</p></div>
</div>
</div>
<div class="sect1">
-<h2 id="_full_pattern_set">FULL PATTERN SET</h2>
+<h2 id="_internals_8201_8212_8201_non_cone_problems">INTERNALS — NON-CONE PROBLEMS</h2>
<div class="sectionbody">
-<div class="paragraph"><p>By default, the sparse-checkout file uses the same syntax as <code>.gitignore</code>
-files.</p></div>
-<div class="paragraph"><p>While <code>$GIT_DIR/info/sparse-checkout</code> is usually used to specify what
-files are included, you can also specify what files are <em>not</em> included,
-using negative patterns. For example, to remove the file <code>unwanted</code>:</p></div>
-<div class="listingblock">
+<div class="paragraph"><p>The <code>$GIT_DIR/info/sparse-checkout</code> file populated by the <code>set</code> and
+<code>add</code> subcommands is defined to be a bunch of patterns (one per line)
+using the same syntax as <code>.gitignore</code> files. In cone mode, these
+patterns are restricted to matching directories (and users only ever
+need supply or see directory names), while in non-cone mode any
+gitignore-style pattern is permitted. Using the full gitignore-style
+patterns in non-cone mode has a number of shortcomings:</p></div>
+<div class="ulist"><ul>
+<li>
+<p>
+Fundamentally, it makes various worktree-updating processes (pull,
+ merge, rebase, switch, reset, checkout, etc.) require O(N*M) pattern
+ matches, where N is the number of patterns and M is the number of
+ paths in the index. This scales poorly.
+</p>
+</li>
+<li>
+<p>
+Avoiding the scaling issue has to be done via limiting the number
+ of patterns via specifying leading directory name or glob.
+</p>
+</li>
+<li>
+<p>
+Passing globs on the command line is error-prone as users may
+ forget to quote the glob, causing the shell to expand it into all
+ matching files and pass them all individually along to
+ sparse-checkout set/add. While this could also be a problem with
+ e.g. "git grep — *.c", mistakes with grep/log/status appear in
+ the immediate output. With sparse-checkout, the mistake gets
+ recorded at the time the sparse-checkout command is run and might
+ not be problematic until the user later switches branches or rebases
+ or merges, thus putting a delay between the user’s error and when
+ they have a chance to catch/notice it.
+</p>
+</li>
+<li>
+<p>
+Related to the previous item, sparse-checkout has an <em>add</em>
+ subcommand but no <em>remove</em> subcommand. Even if a <em>remove</em>
+ subcommand were added, undoing an accidental unquoted glob runs
+ the risk of "removing too much", as it may remove entries that had
+ been included before the accidental add.
+</p>
+</li>
+<li>
+<p>
+Non-cone mode uses gitignore-style patterns to select what to
+ <strong>include</strong> (with the exception of negated patterns), while
+ .gitignore files use gitignore-style patterns to select what to
+ <strong>exclude</strong> (with the exception of negated patterns). The
+ documentation on gitignore-style patterns usually does not talk in
+ terms of matching or non-matching, but on what the user wants to
+ "exclude". This can cause confusion for users trying to learn how
+ to specify sparse-checkout patterns to get their desired behavior.
+</p>
+</li>
+<li>
+<p>
+Every other git subcommand that wants to provide "special path
+ pattern matching" of some sort uses pathspecs, but non-cone mode
+ for sparse-checkout uses gitignore patterns, which feels
+ inconsistent.
+</p>
+</li>
+<li>
+<p>
+It has edge cases where the "right" behavior is unclear. Two examples:
+</p>
+<div class="literalblock">
<div class="content">
-<pre><code>/*
-!unwanted</code></pre>
+<pre><code>First, two users are in a subdirectory, and the first runs
+ git sparse-checkout set '/toplevel-dir/*.c'
+while the second runs
+ git sparse-checkout set relative-dir
+Should those arguments be transliterated into
+ current/subdirectory/toplevel-dir/*.c
+and
+ current/subdirectory/relative-dir
+before inserting into the sparse-checkout file? The user who typed
+the first command is probably aware that arguments to set/add are
+supposed to be patterns in non-cone mode, and probably would not be
+happy with such a transliteration. However, many gitignore-style
+patterns are just paths, which might be what the user who typed the
+second command was thinking, and they'd be upset if their argument
+wasn't transliterated.</code></pre>
</div></div>
+<div class="literalblock">
+<div class="content">
+<pre><code>Second, what should bash-completion complete on for set/add commands
+for non-cone users? If it suggests paths, is it exacerbating the
+problem above? Also, if it suggests paths, what if the user has a
+file or directory that begins with either a '!' or '#' or has a '*',
+'\', '?', '[', or ']' in its name? And if it suggests paths, will
+it complete "/pro" to "/proc" (in the root filesytem) rather than to
+"/progress.txt" in the current directory? (Note that users are
+likely to want to start paths with a leading '/' in non-cone mode,
+for the same reason that .gitignore files often have one.)
+Completing on files or directories might give nasty surprises in
+all these cases.</code></pre>
+</div></div>
+</li>
+<li>
+<p>
+The excessive flexibility made other extensions essentially
+ impractical. <code>--sparse-index</code> is likely impossible in non-cone
+ mode; even if it is somehow feasible, it would have been far more
+ work to implement and may have been too slow in practice. Some
+ ideas for adding coupling between partial clones and sparse
+ checkouts are only practical with a more restricted set of paths
+ as well.
+</p>
+</li>
+</ul></div>
+<div class="paragraph"><p>For all these reasons, non-cone mode is deprecated. Please switch to
+using cone mode.</p></div>
+</div>
+</div>
+<div class="sect1">
+<h2 id="_internals_8201_8212_8201_cone_mode_handling">INTERNALS — CONE MODE HANDLING</h2>
+<div class="sectionbody">
+<div class="paragraph"><p>The "cone mode", which is the default, lets you specify only what
+directories to include. For any directory specified, all paths below
+that directory will be included, and any paths immediately under
+leading directories (including the toplevel directory) will also be
+included. Thus, if you specified the directory
+ Documentation/technical/
+then your sparse checkout would contain:</p></div>
+<div class="ulist"><ul>
+<li>
+<p>
+all files in the toplevel-directory
+</p>
+</li>
+<li>
+<p>
+all files immediately under Documentation/
+</p>
+</li>
+<li>
+<p>
+all files at any depth under Documentation/technical/
+</p>
+</li>
+</ul></div>
+<div class="paragraph"><p>Also, in cone mode, even if no directories are specified, then the
+files in the toplevel directory will be included.</p></div>
+<div class="paragraph"><p>When changing the sparse-checkout patterns in cone mode, Git will inspect each
+tracked directory that is not within the sparse-checkout cone to see if it
+contains any untracked files. If all of those files are ignored due to the
+<code>.gitignore</code> patterns, then the directory will be deleted. If any of the
+untracked files within that directory is not ignored, then no deletions will
+occur within that directory and a warning message will appear. If these files
+are important, then reset your sparse-checkout definition so they are included,
+use <code>git add</code> and <code>git commit</code> to store them, then remove any remaining files
+manually to ensure Git can behave optimally.</p></div>
+<div class="paragraph"><p>See also the "Internals — Cone Pattern Set" section to learn how the
+directories are transformed under the hood into a subset of the
+Full Pattern Set of sparse-checkout.</p></div>
</div>
</div>
<div class="sect1">
-<h2 id="_cone_pattern_set">CONE PATTERN SET</h2>
+<h2 id="_internals_8201_8212_8201_full_pattern_set">INTERNALS — FULL PATTERN SET</h2>
<div class="sectionbody">
<div class="paragraph"><p>The full pattern set allows for arbitrary pattern matches and complicated
inclusion/exclusion rules. These can result in O(N*M) pattern matches when
updating the index, where N is the number of patterns and M is the number
of paths in the index. To combat this performance issue, a more restricted
pattern set is allowed when <code>core.sparseCheckoutCone</code> is enabled.</p></div>
-<div class="paragraph"><p>The accepted patterns in the cone pattern set are:</p></div>
+<div class="paragraph"><p>The sparse-checkout file uses the same syntax as <code>.gitignore</code> files;
+see <a href="gitignore.html">gitignore(5)</a> for details. Here, though, the patterns are
+usually being used to select which files to include rather than which
+files to exclude. (However, it can get a bit confusing since
+gitignore-style patterns have negations defined by patterns which
+begin with a <em>!</em>, so you can also select files to <em>not</em> include.)</p></div>
+<div class="paragraph"><p>For example, to select everything, and then to remove the file
+<code>unwanted</code> (so that every file will appear in your working tree except
+the file named <code>unwanted</code>):</p></div>
+<div class="literalblock">
+<div class="content">
+<pre><code>git sparse-checkout set --no-cone '/*' '!unwanted'</code></pre>
+</div></div>
+<div class="paragraph"><p>These patterns are just placed into the
+<code>$GIT_DIR/info/sparse-checkout</code> as-is, so the contents of that file
+at this point would be</p></div>
+<div class="listingblock">
+<div class="content">
+<pre><code>/*
+!unwanted</code></pre>
+</div></div>
+<div class="paragraph"><p>See also the "Sparse Checkout" section of <a href="git-read-tree.html">git-read-tree(1)</a> to
+learn more about the gitignore-style patterns used in sparse
+checkouts.</p></div>
+</div>
+</div>
+<div class="sect1">
+<h2 id="_internals_8201_8212_8201_cone_pattern_set">INTERNALS — CONE PATTERN SET</h2>
+<div class="sectionbody">
+<div class="paragraph"><p>In cone mode, only directories are accepted, but they are translated into
+the same gitignore-style patterns used in the full pattern set. We refer
+to the particular patterns used in those mode as being of one of two types:</p></div>
<div class="olist arabic"><ol class="arabic">
<li>
<p>
@@ -960,23 +1192,22 @@ pattern set is allowed when <code>core.sparseCheckoutCone</code> is enabled.</p> </p>
</li>
</ol></div>
-<div class="paragraph"><p>In addition to the above two patterns, we also expect that all files in the
-root directory are included. If a recursive pattern is added, then all
-leading directories are added as parent patterns.</p></div>
-<div class="paragraph"><p>By default, when running <code>git sparse-checkout init</code>, the root directory is
-added as a parent pattern. At this point, the sparse-checkout file contains
-the following patterns:</p></div>
+<div class="paragraph"><p>Since cone mode always includes files at the toplevel, when running
+<code>git sparse-checkout set</code> with no directories specified, the toplevel
+directory is added as a parent pattern. At this point, the
+sparse-checkout file contains the following patterns:</p></div>
<div class="listingblock">
<div class="content">
<pre><code>/*
!/*/</code></pre>
</div></div>
-<div class="paragraph"><p>This says "include everything in root, but nothing two levels below root."</p></div>
-<div class="paragraph"><p>When in cone mode, the <code>git sparse-checkout set</code> subcommand takes a list of
-directories instead of a list of sparse-checkout patterns. In this mode,
-the command <code>git sparse-checkout set A/B/C</code> sets the directory <code>A/B/C</code> as
-a recursive pattern, the directories <code>A</code> and <code>A/B</code> are added as parent
-patterns. The resulting sparse-checkout file is now</p></div>
+<div class="paragraph"><p>This says "include everything immediately under the toplevel
+directory, but nothing at any level below that."</p></div>
+<div class="paragraph"><p>When in cone mode, the <code>git sparse-checkout set</code> subcommand takes a
+list of directories. The command <code>git sparse-checkout set A/B/C</code> sets
+the directory <code>A/B/C</code> as a recursive pattern, the directories <code>A</code> and
+<code>A/B</code> are added as parent patterns. The resulting sparse-checkout file
+is now</p></div>
<div class="listingblock">
<div class="content">
<pre><code>/*
@@ -989,13 +1220,17 @@ patterns. The resulting sparse-checkout file is now</p></div> </div></div>
<div class="paragraph"><p>Here, order matters, so the negative patterns are overridden by the positive
patterns that appear lower in the file.</p></div>
-<div class="paragraph"><p>If <code>core.sparseCheckoutCone=true</code>, then Git will parse the sparse-checkout file
-expecting patterns of these types. Git will warn if the patterns do not match.
-If the patterns do match the expected format, then Git will use faster hash-
-based algorithms to compute inclusion in the sparse-checkout.</p></div>
-<div class="paragraph"><p>In the cone mode case, the <code>git sparse-checkout list</code> subcommand will list the
-directories that define the recursive patterns. For the example sparse-checkout
-file above, the output is as follows:</p></div>
+<div class="paragraph"><p>Unless <code>core.sparseCheckoutCone</code> is explicitly set to <code>false</code>, Git will
+parse the sparse-checkout file expecting patterns of these types. Git will
+warn if the patterns do not match. If the patterns do match the expected
+format, then Git will use faster hash-based algorithms to compute inclusion
+in the sparse-checkout. If they do not match, git will behave as though
+<code>core.sparseCheckoutCone</code> was false, regardless of its setting.</p></div>
+<div class="paragraph"><p>In the cone mode case, despite the fact that full patterns are written
+to the $GIT_DIR/info/sparse-checkout file, the <code>git sparse-checkout
+list</code> subcommand will list the directories that define the recursive
+patterns. For the example sparse-checkout file above, the output is as
+follows:</p></div>
<div class="listingblock">
<div class="content">
<pre><code>$ git sparse-checkout list
@@ -1005,19 +1240,10 @@ A/B/C</code></pre> case-insensitive check. This corrects for case mismatched filenames in the
<em>git sparse-checkout set</em> command to reflect the expected cone in the working
directory.</p></div>
-<div class="paragraph"><p>When changing the sparse-checkout patterns in cone mode, Git will inspect each
-tracked directory that is not within the sparse-checkout cone to see if it
-contains any untracked files. If all of those files are ignored due to the
-<code>.gitignore</code> patterns, then the directory will be deleted. If any of the
-untracked files within that directory is not ignored, then no deletions will
-occur within that directory and a warning message will appear. If these files
-are important, then reset your sparse-checkout definition so they are included,
-use <code>git add</code> and <code>git commit</code> to store them, then remove any remaining files
-manually to ensure Git can behave optimally.</p></div>
</div>
</div>
<div class="sect1">
-<h2 id="_submodules">SUBMODULES</h2>
+<h2 id="_internals_8201_8212_8201_submodules">INTERNALS — SUBMODULES</h2>
<div class="sectionbody">
<div class="paragraph"><p>If your repository contains one or more submodules, then submodules
are populated based on interactions with the <code>git submodule</code> command.
@@ -1063,7 +1289,7 @@ of these restrictions.</p></div> <div id="footer">
<div id="footer-text">
Last updated
- 2022-03-09 14:17:38 PST
+ 2022-06-03 15:24:31 PDT
</div>
</div>
</body>
diff --git a/git-sparse-checkout.txt b/git-sparse-checkout.txt index 88e55f432..3776705bf 100644 --- a/git-sparse-checkout.txt +++ b/git-sparse-checkout.txt @@ -15,15 +15,15 @@ SYNOPSIS DESCRIPTION ----------- -This command is used to create sparse checkouts, which means that it -changes the working tree from having all tracked files present, to only -have a subset of them. It can also switch which subset of files are -present, or undo and go back to having all tracked files present in the -working copy. +This command is used to create sparse checkouts, which change the +working tree from having all tracked files present to only having a +subset of those files. It can also switch which subset of files are +present, or undo and go back to having all tracked files present in +the working copy. The subset of files is chosen by providing a list of directories in -cone mode (which is recommended), or by providing a list of patterns -in non-cone mode. +cone mode (the default), or by providing a list of patterns in +non-cone mode. When in a sparse-checkout, other Git commands behave a bit differently. For example, switching branches will not update paths outside the @@ -44,9 +44,9 @@ COMMANDS Enable the necessary sparse-checkout config settings (`core.sparseCheckout`, `core.sparseCheckoutCone`, and `index.sparse`) if they are not already set to the desired values, - and write a set of patterns to the sparse-checkout file from the - list of arguments following the 'set' subcommand. Update the - working directory to match the new patterns. + populate the sparse-checkout file from the list of arguments + following the 'set' subcommand, and update the working directory to + match. + To ensure that adjusting the sparse-checkout settings within a worktree does not alter the sparse-checkout settings in other worktrees, the 'set' @@ -60,22 +60,20 @@ When the `--stdin` option is provided, the directories or patterns are read from standard in as a newline-delimited list instead of from the arguments. + -When `--cone` is passed or `core.sparseCheckoutCone` is enabled, the -input list is considered a list of directories. This allows for -better performance with a limited set of patterns (see 'CONE PATTERN -SET' below). The input format matches the output of `git ls-tree ---name-only`. This includes interpreting pathnames that begin with a -double quote (") as C-style quoted strings. Note that the set command -will write patterns to the sparse-checkout file to include all files -contained in those directories (recursively) as well as files that are -siblings of ancestor directories. This may become the default in the -future; --no-cone can be passed to request non-cone mode. +By default, the input list is considered a list of directories, matching +the output of `git ls-tree -d --name-only`. This includes interpreting +pathnames that begin with a double quote (") as C-style quoted strings. +Note that all files under the specified directories (at any depth) will +be included in the sparse checkout, as well as files that are siblings +of either the given directory or any of its ancestors (see 'CONE PATTERN +SET' below for more details). In the past, this was not the default, +and `--cone` needed to be specified or `core.sparseCheckoutCone` needed +to be enabled. + -When `--no-cone` is passed or `core.sparseCheckoutCone` is not enabled, -the input list is considered a list of patterns. This mode is harder -to use and less performant, and is thus not recommended. See the -"Sparse Checkout" section of linkgit:git-read-tree[1] and the "Pattern -Set" sections below for more details. +When `--no-cone` is passed, the input list is considered a list of +patterns. This mode has a number of drawbacks, including not working +with some options like `--sparse-index`. As explained in the +"Non-cone Problems" section below, we do not recommend using it. + Use the `--[no-]sparse-index` option to use a sparse index (the default is to not use it). A sparse index reduces the size of the @@ -137,8 +135,45 @@ paths to pass to a subsequent 'set' or 'add' command. However, the disable command, so the easy restore of calling a plain `init` decreased in utility. -SPARSE CHECKOUT ---------------- +EXAMPLES +-------- +`git sparse-checkout set MY/DIR1 SUB/DIR2`:: + + Change to a sparse checkout with all files (at any depth) under + MY/DIR1/ and SUB/DIR2/ present in the working copy (plus all + files immediately under MY/ and SUB/ and the toplevel + directory). If already in a sparse checkout, change which files + are present in the working copy to this new selection. Note + that this command will also delete all ignored files in any + directory that no longer has either tracked or + non-ignored-untracked files present. + +`git sparse-checkout disable`:: + + Repopulate the working directory with all files, disabling sparse + checkouts. + +`git sparse-checkout add SOME/DIR/ECTORY`:: + + Add all files under SOME/DIR/ECTORY/ (at any depth) to the + sparse checkout, as well as all files immediately under + SOME/DIR/ and immediately under SOME/. Must already be in a + sparse checkout before using this command. + +`git sparse-checkout reapply`:: + + It is possible for commands to update the working tree in a + way that does not respect the selected sparsity directories. + This can come from tools external to Git writing files, or + even affect Git commands because of either special cases (such + as hitting conflicts when merging/rebasing), or because some + commands didn't fully support sparse checkouts (e.g. the old + `recursive` merge backend had only limited support). This + command reapplies the existing sparse directory specifications + to make the working directory match. + +INTERNALS -- SPARSE CHECKOUT +---------------------------- "Sparse checkout" allows populating the working directory sparsely. It uses the skip-worktree bit (see linkgit:git-update-index[1]) to tell Git @@ -155,64 +190,196 @@ directory, it updates the skip-worktree bits in the index based on this file. The files matching the patterns in the file will appear in the working directory, and the rest will not. -To enable the sparse-checkout feature, run `git sparse-checkout set` to -set the patterns you want to use. +INTERNALS -- NON-CONE PROBLEMS +------------------------------ + +The `$GIT_DIR/info/sparse-checkout` file populated by the `set` and +`add` subcommands is defined to be a bunch of patterns (one per line) +using the same syntax as `.gitignore` files. In cone mode, these +patterns are restricted to matching directories (and users only ever +need supply or see directory names), while in non-cone mode any +gitignore-style pattern is permitted. Using the full gitignore-style +patterns in non-cone mode has a number of shortcomings: + + * Fundamentally, it makes various worktree-updating processes (pull, + merge, rebase, switch, reset, checkout, etc.) require O(N*M) pattern + matches, where N is the number of patterns and M is the number of + paths in the index. This scales poorly. + + * Avoiding the scaling issue has to be done via limiting the number + of patterns via specifying leading directory name or glob. + + * Passing globs on the command line is error-prone as users may + forget to quote the glob, causing the shell to expand it into all + matching files and pass them all individually along to + sparse-checkout set/add. While this could also be a problem with + e.g. "git grep -- *.c", mistakes with grep/log/status appear in + the immediate output. With sparse-checkout, the mistake gets + recorded at the time the sparse-checkout command is run and might + not be problematic until the user later switches branches or rebases + or merges, thus putting a delay between the user's error and when + they have a chance to catch/notice it. + + * Related to the previous item, sparse-checkout has an 'add' + subcommand but no 'remove' subcommand. Even if a 'remove' + subcommand were added, undoing an accidental unquoted glob runs + the risk of "removing too much", as it may remove entries that had + been included before the accidental add. + + * Non-cone mode uses gitignore-style patterns to select what to + *include* (with the exception of negated patterns), while + .gitignore files use gitignore-style patterns to select what to + *exclude* (with the exception of negated patterns). The + documentation on gitignore-style patterns usually does not talk in + terms of matching or non-matching, but on what the user wants to + "exclude". This can cause confusion for users trying to learn how + to specify sparse-checkout patterns to get their desired behavior. + + * Every other git subcommand that wants to provide "special path + pattern matching" of some sort uses pathspecs, but non-cone mode + for sparse-checkout uses gitignore patterns, which feels + inconsistent. + + * It has edge cases where the "right" behavior is unclear. Two examples: + + First, two users are in a subdirectory, and the first runs + git sparse-checkout set '/toplevel-dir/*.c' + while the second runs + git sparse-checkout set relative-dir + Should those arguments be transliterated into + current/subdirectory/toplevel-dir/*.c + and + current/subdirectory/relative-dir + before inserting into the sparse-checkout file? The user who typed + the first command is probably aware that arguments to set/add are + supposed to be patterns in non-cone mode, and probably would not be + happy with such a transliteration. However, many gitignore-style + patterns are just paths, which might be what the user who typed the + second command was thinking, and they'd be upset if their argument + wasn't transliterated. + + Second, what should bash-completion complete on for set/add commands + for non-cone users? If it suggests paths, is it exacerbating the + problem above? Also, if it suggests paths, what if the user has a + file or directory that begins with either a '!' or '#' or has a '*', + '\', '?', '[', or ']' in its name? And if it suggests paths, will + it complete "/pro" to "/proc" (in the root filesytem) rather than to + "/progress.txt" in the current directory? (Note that users are + likely to want to start paths with a leading '/' in non-cone mode, + for the same reason that .gitignore files often have one.) + Completing on files or directories might give nasty surprises in + all these cases. + + * The excessive flexibility made other extensions essentially + impractical. `--sparse-index` is likely impossible in non-cone + mode; even if it is somehow feasible, it would have been far more + work to implement and may have been too slow in practice. Some + ideas for adding coupling between partial clones and sparse + checkouts are only practical with a more restricted set of paths + as well. + +For all these reasons, non-cone mode is deprecated. Please switch to +using cone mode. + + +INTERNALS -- CONE MODE HANDLING +------------------------------- + +The "cone mode", which is the default, lets you specify only what +directories to include. For any directory specified, all paths below +that directory will be included, and any paths immediately under +leading directories (including the toplevel directory) will also be +included. Thus, if you specified the directory + Documentation/technical/ +then your sparse checkout would contain: + + * all files in the toplevel-directory + * all files immediately under Documentation/ + * all files at any depth under Documentation/technical/ + +Also, in cone mode, even if no directories are specified, then the +files in the toplevel directory will be included. -To repopulate the working directory with all files, use the -`git sparse-checkout disable` command. +When changing the sparse-checkout patterns in cone mode, Git will inspect each +tracked directory that is not within the sparse-checkout cone to see if it +contains any untracked files. If all of those files are ignored due to the +`.gitignore` patterns, then the directory will be deleted. If any of the +untracked files within that directory is not ignored, then no deletions will +occur within that directory and a warning message will appear. If these files +are important, then reset your sparse-checkout definition so they are included, +use `git add` and `git commit` to store them, then remove any remaining files +manually to ensure Git can behave optimally. +See also the "Internals -- Cone Pattern Set" section to learn how the +directories are transformed under the hood into a subset of the +Full Pattern Set of sparse-checkout. -FULL PATTERN SET ----------------- -By default, the sparse-checkout file uses the same syntax as `.gitignore` -files. +INTERNALS -- FULL PATTERN SET +----------------------------- -While `$GIT_DIR/info/sparse-checkout` is usually used to specify what -files are included, you can also specify what files are _not_ included, -using negative patterns. For example, to remove the file `unwanted`: +The full pattern set allows for arbitrary pattern matches and complicated +inclusion/exclusion rules. These can result in O(N*M) pattern matches when +updating the index, where N is the number of patterns and M is the number +of paths in the index. To combat this performance issue, a more restricted +pattern set is allowed when `core.sparseCheckoutCone` is enabled. + +The sparse-checkout file uses the same syntax as `.gitignore` files; +see linkgit:gitignore[5] for details. Here, though, the patterns are +usually being used to select which files to include rather than which +files to exclude. (However, it can get a bit confusing since +gitignore-style patterns have negations defined by patterns which +begin with a '!', so you can also select files to _not_ include.) + +For example, to select everything, and then to remove the file +`unwanted` (so that every file will appear in your working tree except +the file named `unwanted`): + + git sparse-checkout set --no-cone '/*' '!unwanted' + +These patterns are just placed into the +`$GIT_DIR/info/sparse-checkout` as-is, so the contents of that file +at this point would be ---------------- /* !unwanted ---------------- +See also the "Sparse Checkout" section of linkgit:git-read-tree[1] to +learn more about the gitignore-style patterns used in sparse +checkouts. -CONE PATTERN SET ----------------- -The full pattern set allows for arbitrary pattern matches and complicated -inclusion/exclusion rules. These can result in O(N*M) pattern matches when -updating the index, where N is the number of patterns and M is the number -of paths in the index. To combat this performance issue, a more restricted -pattern set is allowed when `core.sparseCheckoutCone` is enabled. +INTERNALS -- CONE PATTERN SET +----------------------------- -The accepted patterns in the cone pattern set are: +In cone mode, only directories are accepted, but they are translated into +the same gitignore-style patterns used in the full pattern set. We refer +to the particular patterns used in those mode as being of one of two types: 1. *Recursive:* All paths inside a directory are included. 2. *Parent:* All files immediately inside a directory are included. -In addition to the above two patterns, we also expect that all files in the -root directory are included. If a recursive pattern is added, then all -leading directories are added as parent patterns. - -By default, when running `git sparse-checkout init`, the root directory is -added as a parent pattern. At this point, the sparse-checkout file contains -the following patterns: +Since cone mode always includes files at the toplevel, when running +`git sparse-checkout set` with no directories specified, the toplevel +directory is added as a parent pattern. At this point, the +sparse-checkout file contains the following patterns: ---------------- /* !/*/ ---------------- -This says "include everything in root, but nothing two levels below root." +This says "include everything immediately under the toplevel +directory, but nothing at any level below that." -When in cone mode, the `git sparse-checkout set` subcommand takes a list of -directories instead of a list of sparse-checkout patterns. In this mode, -the command `git sparse-checkout set A/B/C` sets the directory `A/B/C` as -a recursive pattern, the directories `A` and `A/B` are added as parent -patterns. The resulting sparse-checkout file is now +When in cone mode, the `git sparse-checkout set` subcommand takes a +list of directories. The command `git sparse-checkout set A/B/C` sets +the directory `A/B/C` as a recursive pattern, the directories `A` and +`A/B` are added as parent patterns. The resulting sparse-checkout file +is now ---------------- /* @@ -227,14 +394,18 @@ patterns. The resulting sparse-checkout file is now Here, order matters, so the negative patterns are overridden by the positive patterns that appear lower in the file. -If `core.sparseCheckoutCone=true`, then Git will parse the sparse-checkout file -expecting patterns of these types. Git will warn if the patterns do not match. -If the patterns do match the expected format, then Git will use faster hash- -based algorithms to compute inclusion in the sparse-checkout. +Unless `core.sparseCheckoutCone` is explicitly set to `false`, Git will +parse the sparse-checkout file expecting patterns of these types. Git will +warn if the patterns do not match. If the patterns do match the expected +format, then Git will use faster hash-based algorithms to compute inclusion +in the sparse-checkout. If they do not match, git will behave as though +`core.sparseCheckoutCone` was false, regardless of its setting. -In the cone mode case, the `git sparse-checkout list` subcommand will list the -directories that define the recursive patterns. For the example sparse-checkout -file above, the output is as follows: +In the cone mode case, despite the fact that full patterns are written +to the $GIT_DIR/info/sparse-checkout file, the `git sparse-checkout +list` subcommand will list the directories that define the recursive +patterns. For the example sparse-checkout file above, the output is as +follows: -------------------------- $ git sparse-checkout list @@ -246,19 +417,9 @@ case-insensitive check. This corrects for case mismatched filenames in the 'git sparse-checkout set' command to reflect the expected cone in the working directory. -When changing the sparse-checkout patterns in cone mode, Git will inspect each -tracked directory that is not within the sparse-checkout cone to see if it -contains any untracked files. If all of those files are ignored due to the -`.gitignore` patterns, then the directory will be deleted. If any of the -untracked files within that directory is not ignored, then no deletions will -occur within that directory and a warning message will appear. If these files -are important, then reset your sparse-checkout definition so they are included, -use `git add` and `git commit` to store them, then remove any remaining files -manually to ensure Git can behave optimally. - -SUBMODULES ----------- +INTERNALS -- SUBMODULES +----------------------- If your repository contains one or more submodules, then submodules are populated based on interactions with the `git submodule` command. diff --git a/howto/coordinate-embargoed-releases.html b/howto/coordinate-embargoed-releases.html index 30fc3bb55..c2238da72 100644 --- a/howto/coordinate-embargoed-releases.html +++ b/howto/coordinate-embargoed-releases.html @@ -873,7 +873,7 @@ Thanks, <div id="footer">
<div id="footer-text">
Last updated
- 2022-06-01 12:01:12 PDT
+ 2022-06-03 15:25:00 PDT
</div>
</div>
</body>
diff --git a/howto/keep-canonical-history-correct.html b/howto/keep-canonical-history-correct.html index dd843339d..6f9af043e 100644 --- a/howto/keep-canonical-history-correct.html +++ b/howto/keep-canonical-history-correct.html @@ -938,7 +938,7 @@ tip of your <em>master</em> again and redo the two merges:</p></div> <div id="footer">
<div id="footer-text">
Last updated
- 2022-06-01 12:01:12 PDT
+ 2022-06-03 15:25:00 PDT
</div>
</div>
</body>
diff --git a/howto/maintain-git.html b/howto/maintain-git.html index 237dd2562..038a42df3 100644 --- a/howto/maintain-git.html +++ b/howto/maintain-git.html @@ -1469,7 +1469,7 @@ $ git update-ref -d $mf/ai/topic</code></pre> <div id="footer">
<div id="footer-text">
Last updated
- 2022-06-01 12:01:12 PDT
+ 2022-06-03 15:25:00 PDT
</div>
</div>
</body>
diff --git a/howto/new-command.html b/howto/new-command.html index 4fade47d1..35a8510f4 100644 --- a/howto/new-command.html +++ b/howto/new-command.html @@ -863,7 +863,7 @@ letter [PATCH 0/n]. <div id="footer">
<div id="footer-text">
Last updated
- 2022-06-01 12:01:09 PDT
+ 2022-06-03 15:24:58 PDT
</div>
</div>
</body>
diff --git a/howto/rebase-from-internal-branch.html b/howto/rebase-from-internal-branch.html index c484c7f02..48423a805 100644 --- a/howto/rebase-from-internal-branch.html +++ b/howto/rebase-from-internal-branch.html @@ -895,7 +895,7 @@ the #1' commit.</p></div> <div id="footer">
<div id="footer-text">
Last updated
- 2022-06-01 12:01:12 PDT
+ 2022-06-03 15:25:00 PDT
</div>
</div>
</body>
diff --git a/howto/rebuild-from-update-hook.html b/howto/rebuild-from-update-hook.html index c3898b2bd..5d76d2d2f 100644 --- a/howto/rebuild-from-update-hook.html +++ b/howto/rebuild-from-update-hook.html @@ -847,7 +847,7 @@ This is still crude and does not protect against simultaneous <div id="footer">
<div id="footer-text">
Last updated
- 2022-06-01 12:01:12 PDT
+ 2022-06-03 15:25:00 PDT
</div>
</div>
</body>
diff --git a/howto/recover-corrupted-blob-object.html b/howto/recover-corrupted-blob-object.html index 0bc574f11..76b19cec6 100644 --- a/howto/recover-corrupted-blob-object.html +++ b/howto/recover-corrupted-blob-object.html @@ -880,7 +880,7 @@ thing.</p></div> <div id="footer">
<div id="footer-text">
Last updated
- 2022-06-01 12:01:11 PDT
+ 2022-06-03 15:25:00 PDT
</div>
</div>
</body>
diff --git a/howto/recover-corrupted-object-harder.html b/howto/recover-corrupted-object-harder.html index 1846641ee..8c321ca3c 100644 --- a/howto/recover-corrupted-object-harder.html +++ b/howto/recover-corrupted-object-harder.html @@ -1189,7 +1189,7 @@ int main(int argc, char **argv) <div id="footer">
<div id="footer-text">
Last updated
- 2022-06-01 12:01:12 PDT
+ 2022-06-03 15:25:00 PDT
</div>
</div>
</body>
diff --git a/howto/revert-a-faulty-merge.html b/howto/revert-a-faulty-merge.html index 12780ba9b..a1756c27f 100644 --- a/howto/revert-a-faulty-merge.html +++ b/howto/revert-a-faulty-merge.html @@ -1025,7 +1025,7 @@ P---o---o---M---x---x---W---x---M2 <div id="footer">
<div id="footer-text">
Last updated
- 2022-06-01 12:01:11 PDT
+ 2022-06-03 15:24:59 PDT
</div>
</div>
</body>
diff --git a/howto/revert-branch-rebase.html b/howto/revert-branch-rebase.html index c370786e0..14560dac4 100644 --- a/howto/revert-branch-rebase.html +++ b/howto/revert-branch-rebase.html @@ -907,7 +907,7 @@ Committed merge 7fb9b7262a1d1e0a47bbfdcbbcf50ce0635d3f8f <div id="footer">
<div id="footer-text">
Last updated
- 2022-06-01 12:01:09 PDT
+ 2022-06-03 15:24:58 PDT
</div>
</div>
</body>
diff --git a/howto/separating-topic-branches.html b/howto/separating-topic-branches.html index 2257d8ce9..0fa7393b4 100644 --- a/howto/separating-topic-branches.html +++ b/howto/separating-topic-branches.html @@ -841,7 +841,7 @@ o---o"master"</code></pre> <div id="footer">
<div id="footer-text">
Last updated
- 2022-06-01 12:01:11 PDT
+ 2022-06-03 15:24:59 PDT
</div>
</div>
</body>
diff --git a/howto/setup-git-server-over-http.html b/howto/setup-git-server-over-http.html index 8173eb8b4..9f21829ce 100644 --- a/howto/setup-git-server-over-http.html +++ b/howto/setup-git-server-over-http.html @@ -1071,7 +1071,7 @@ help diagnosing the problem, but removes security checks.</p></div> <div id="footer">
<div id="footer-text">
Last updated
- 2022-06-01 12:01:10 PDT
+ 2022-06-03 15:24:59 PDT
</div>
</div>
</body>
diff --git a/howto/update-hook-example.html b/howto/update-hook-example.html index e30d70635..ee0bd950e 100644 --- a/howto/update-hook-example.html +++ b/howto/update-hook-example.html @@ -930,7 +930,7 @@ that JC can make non-fast-forward pushes on it.</p></div> <div id="footer">
<div id="footer-text">
Last updated
- 2022-06-01 12:01:10 PDT
+ 2022-06-03 15:24:59 PDT
</div>
</div>
</body>
diff --git a/howto/use-git-daemon.html b/howto/use-git-daemon.html index edc1ac110..5fe57d526 100644 --- a/howto/use-git-daemon.html +++ b/howto/use-git-daemon.html @@ -791,7 +791,7 @@ a good practice to put the paths after a "--" separator.</p></div> <div id="footer">
<div id="footer-text">
Last updated
- 2022-06-01 12:01:10 PDT
+ 2022-06-03 15:24:59 PDT
</div>
</div>
</body>
diff --git a/howto/using-merge-subtree.html b/howto/using-merge-subtree.html index ad678cd0f..d3b84ed73 100644 --- a/howto/using-merge-subtree.html +++ b/howto/using-merge-subtree.html @@ -848,7 +848,7 @@ Please note that if the other project merges from you, then it will <div id="footer">
<div id="footer-text">
Last updated
- 2022-06-01 12:01:09 PDT
+ 2022-06-03 15:24:58 PDT
</div>
</div>
</body>
diff --git a/howto/using-signed-tag-in-pull-request.html b/howto/using-signed-tag-in-pull-request.html index 8ef62f3d2..9b694d8e2 100644 --- a/howto/using-signed-tag-in-pull-request.html +++ b/howto/using-signed-tag-in-pull-request.html @@ -952,7 +952,7 @@ as part of the merge commit.</p></div> <div id="footer">
<div id="footer-text">
Last updated
- 2022-06-01 12:01:09 PDT
+ 2022-06-03 15:24:59 PDT
</div>
</div>
</body>
diff --git a/technical/cruft-packs.html b/technical/cruft-packs.html new file mode 100644 index 000000000..4b99f6986 --- /dev/null +++ b/technical/cruft-packs.html @@ -0,0 +1,900 @@ +<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN"
+ "http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
+<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en">
+<head>
+<meta http-equiv="Content-Type" content="application/xhtml+xml; charset=UTF-8" />
+<meta name="generator" content="AsciiDoc 10.1.4" />
+<title>Cruft packs</title>
+<style type="text/css">
+/* Shared CSS for AsciiDoc xhtml11 and html5 backends */
+
+/* Default font. */
+body {
+ font-family: Georgia,serif;
+}
+
+/* Title font. */
+h1, h2, h3, h4, h5, h6,
+div.title, caption.title,
+thead, p.table.header,
+#toctitle,
+#author, #revnumber, #revdate, #revremark,
+#footer {
+ font-family: Arial,Helvetica,sans-serif;
+}
+
+body {
+ margin: 1em 5% 1em 5%;
+}
+
+a {
+ color: blue;
+ text-decoration: underline;
+}
+a:visited {
+ color: fuchsia;
+}
+
+em {
+ font-style: italic;
+ color: navy;
+}
+
+strong {
+ font-weight: bold;
+ color: #083194;
+}
+
+h1, h2, h3, h4, h5, h6 {
+ color: #527bbd;
+ margin-top: 1.2em;
+ margin-bottom: 0.5em;
+ line-height: 1.3;
+}
+
+h1, h2, h3 {
+ border-bottom: 2px solid silver;
+}
+h2 {
+ padding-top: 0.5em;
+}
+h3 {
+ float: left;
+}
+h3 + * {
+ clear: left;
+}
+h5 {
+ font-size: 1.0em;
+}
+
+div.sectionbody {
+ margin-left: 0;
+}
+
+hr {
+ border: 1px solid silver;
+}
+
+p {
+ margin-top: 0.5em;
+ margin-bottom: 0.5em;
+}
+
+ul, ol, li > p {
+ margin-top: 0;
+}
+ul > li { color: #aaa; }
+ul > li > * { color: black; }
+
+.monospaced, code, pre {
+ font-family: "Courier New", Courier, monospace;
+ font-size: inherit;
+ color: navy;
+ padding: 0;
+ margin: 0;
+}
+pre {
+ white-space: pre-wrap;
+}
+
+#author {
+ color: #527bbd;
+ font-weight: bold;
+ font-size: 1.1em;
+}
+#email {
+}
+#revnumber, #revdate, #revremark {
+}
+
+#footer {
+ font-size: small;
+ border-top: 2px solid silver;
+ padding-top: 0.5em;
+ margin-top: 4.0em;
+}
+#footer-text {
+ float: left;
+ padding-bottom: 0.5em;
+}
+#footer-badges {
+ float: right;
+ padding-bottom: 0.5em;
+}
+
+#preamble {
+ margin-top: 1.5em;
+ margin-bottom: 1.5em;
+}
+div.imageblock, div.exampleblock, div.verseblock,
+div.quoteblock, div.literalblock, div.listingblock, div.sidebarblock,
+div.admonitionblock {
+ margin-top: 1.0em;
+ margin-bottom: 1.5em;
+}
+div.admonitionblock {
+ margin-top: 2.0em;
+ margin-bottom: 2.0em;
+ margin-right: 10%;
+ color: #606060;
+}
+
+div.content { /* Block element content. */
+ padding: 0;
+}
+
+/* Block element titles. */
+div.title, caption.title {
+ color: #527bbd;
+ font-weight: bold;
+ text-align: left;
+ margin-top: 1.0em;
+ margin-bottom: 0.5em;
+}
+div.title + * {
+ margin-top: 0;
+}
+
+td div.title:first-child {
+ margin-top: 0.0em;
+}
+div.content div.title:first-child {
+ margin-top: 0.0em;
+}
+div.content + div.title {
+ margin-top: 0.0em;
+}
+
+div.sidebarblock > div.content {
+ background: #ffffee;
+ border: 1px solid #dddddd;
+ border-left: 4px solid #f0f0f0;
+ padding: 0.5em;
+}
+
+div.listingblock > div.content {
+ border: 1px solid #dddddd;
+ border-left: 5px solid #f0f0f0;
+ background: #f8f8f8;
+ padding: 0.5em;
+}
+
+div.quoteblock, div.verseblock {
+ padding-left: 1.0em;
+ margin-left: 1.0em;
+ margin-right: 10%;
+ border-left: 5px solid #f0f0f0;
+ color: #888;
+}
+
+div.quoteblock > div.attribution {
+ padding-top: 0.5em;
+ text-align: right;
+}
+
+div.verseblock > pre.content {
+ font-family: inherit;
+ font-size: inherit;
+}
+div.verseblock > div.attribution {
+ padding-top: 0.75em;
+ text-align: left;
+}
+/* DEPRECATED: Pre version 8.2.7 verse style literal block. */
+div.verseblock + div.attribution {
+ text-align: left;
+}
+
+div.admonitionblock .icon {
+ vertical-align: top;
+ font-size: 1.1em;
+ font-weight: bold;
+ text-decoration: underline;
+ color: #527bbd;
+ padding-right: 0.5em;
+}
+div.admonitionblock td.content {
+ padding-left: 0.5em;
+ border-left: 3px solid #dddddd;
+}
+
+div.exampleblock > div.content {
+ border-left: 3px solid #dddddd;
+ padding-left: 0.5em;
+}
+
+div.imageblock div.content { padding-left: 0; }
+span.image img { border-style: none; vertical-align: text-bottom; }
+a.image:visited { color: white; }
+
+dl {
+ margin-top: 0.8em;
+ margin-bottom: 0.8em;
+}
+dt {
+ margin-top: 0.5em;
+ margin-bottom: 0;
+ font-style: normal;
+ color: navy;
+}
+dd > *:first-child {
+ margin-top: 0.1em;
+}
+
+ul, ol {
+ list-style-position: outside;
+}
+ol.arabic {
+ list-style-type: decimal;
+}
+ol.loweralpha {
+ list-style-type: lower-alpha;
+}
+ol.upperalpha {
+ list-style-type: upper-alpha;
+}
+ol.lowerroman {
+ list-style-type: lower-roman;
+}
+ol.upperroman {
+ list-style-type: upper-roman;
+}
+
+div.compact ul, div.compact ol,
+div.compact p, div.compact p,
+div.compact div, div.compact div {
+ margin-top: 0.1em;
+ margin-bottom: 0.1em;
+}
+
+tfoot {
+ font-weight: bold;
+}
+td > div.verse {
+ white-space: pre;
+}
+
+div.hdlist {
+ margin-top: 0.8em;
+ margin-bottom: 0.8em;
+}
+div.hdlist tr {
+ padding-bottom: 15px;
+}
+dt.hdlist1.strong, td.hdlist1.strong {
+ font-weight: bold;
+}
+td.hdlist1 {
+ vertical-align: top;
+ font-style: normal;
+ padding-right: 0.8em;
+ color: navy;
+}
+td.hdlist2 {
+ vertical-align: top;
+}
+div.hdlist.compact tr {
+ margin: 0;
+ padding-bottom: 0;
+}
+
+.comment {
+ background: yellow;
+}
+
+.footnote, .footnoteref {
+ font-size: 0.8em;
+}
+
+span.footnote, span.footnoteref {
+ vertical-align: super;
+}
+
+#footnotes {
+ margin: 20px 0 20px 0;
+ padding: 7px 0 0 0;
+}
+
+#footnotes div.footnote {
+ margin: 0 0 5px 0;
+}
+
+#footnotes hr {
+ border: none;
+ border-top: 1px solid silver;
+ height: 1px;
+ text-align: left;
+ margin-left: 0;
+ width: 20%;
+ min-width: 100px;
+}
+
+div.colist td {
+ padding-right: 0.5em;
+ padding-bottom: 0.3em;
+ vertical-align: top;
+}
+div.colist td img {
+ margin-top: 0.3em;
+}
+
+@media print {
+ #footer-badges { display: none; }
+}
+
+#toc {
+ margin-bottom: 2.5em;
+}
+
+#toctitle {
+ color: #527bbd;
+ font-size: 1.1em;
+ font-weight: bold;
+ margin-top: 1.0em;
+ margin-bottom: 0.1em;
+}
+
+div.toclevel0, div.toclevel1, div.toclevel2, div.toclevel3, div.toclevel4 {
+ margin-top: 0;
+ margin-bottom: 0;
+}
+div.toclevel2 {
+ margin-left: 2em;
+ font-size: 0.9em;
+}
+div.toclevel3 {
+ margin-left: 4em;
+ font-size: 0.9em;
+}
+div.toclevel4 {
+ margin-left: 6em;
+ font-size: 0.9em;
+}
+
+span.aqua { color: aqua; }
+span.black { color: black; }
+span.blue { color: blue; }
+span.fuchsia { color: fuchsia; }
+span.gray { color: gray; }
+span.green { color: green; }
+span.lime { color: lime; }
+span.maroon { color: maroon; }
+span.navy { color: navy; }
+span.olive { color: olive; }
+span.purple { color: purple; }
+span.red { color: red; }
+span.silver { color: silver; }
+span.teal { color: teal; }
+span.white { color: white; }
+span.yellow { color: yellow; }
+
+span.aqua-background { background: aqua; }
+span.black-background { background: black; }
+span.blue-background { background: blue; }
+span.fuchsia-background { background: fuchsia; }
+span.gray-background { background: gray; }
+span.green-background { background: green; }
+span.lime-background { background: lime; }
+span.maroon-background { background: maroon; }
+span.navy-background { background: navy; }
+span.olive-background { background: olive; }
+span.purple-background { background: purple; }
+span.red-background { background: red; }
+span.silver-background { background: silver; }
+span.teal-background { background: teal; }
+span.white-background { background: white; }
+span.yellow-background { background: yellow; }
+
+span.big { font-size: 2em; }
+span.small { font-size: 0.6em; }
+
+span.underline { text-decoration: underline; }
+span.overline { text-decoration: overline; }
+span.line-through { text-decoration: line-through; }
+
+div.unbreakable { page-break-inside: avoid; }
+
+
+/*
+ * xhtml11 specific
+ *
+ * */
+
+div.tableblock {
+ margin-top: 1.0em;
+ margin-bottom: 1.5em;
+}
+div.tableblock > table {
+ border: 3px solid #527bbd;
+}
+thead, p.table.header {
+ font-weight: bold;
+ color: #527bbd;
+}
+p.table {
+ margin-top: 0;
+}
+/* Because the table frame attribute is overridden by CSS in most browsers. */
+div.tableblock > table[frame="void"] {
+ border-style: none;
+}
+div.tableblock > table[frame="hsides"] {
+ border-left-style: none;
+ border-right-style: none;
+}
+div.tableblock > table[frame="vsides"] {
+ border-top-style: none;
+ border-bottom-style: none;
+}
+
+
+/*
+ * html5 specific
+ *
+ * */
+
+table.tableblock {
+ margin-top: 1.0em;
+ margin-bottom: 1.5em;
+}
+thead, p.tableblock.header {
+ font-weight: bold;
+ color: #527bbd;
+}
+p.tableblock {
+ margin-top: 0;
+}
+table.tableblock {
+ border-width: 3px;
+ border-spacing: 0px;
+ border-style: solid;
+ border-color: #527bbd;
+ border-collapse: collapse;
+}
+th.tableblock, td.tableblock {
+ border-width: 1px;
+ padding: 4px;
+ border-style: solid;
+ border-color: #527bbd;
+}
+
+table.tableblock.frame-topbot {
+ border-left-style: hidden;
+ border-right-style: hidden;
+}
+table.tableblock.frame-sides {
+ border-top-style: hidden;
+ border-bottom-style: hidden;
+}
+table.tableblock.frame-none {
+ border-style: hidden;
+}
+
+th.tableblock.halign-left, td.tableblock.halign-left {
+ text-align: left;
+}
+th.tableblock.halign-center, td.tableblock.halign-center {
+ text-align: center;
+}
+th.tableblock.halign-right, td.tableblock.halign-right {
+ text-align: right;
+}
+
+th.tableblock.valign-top, td.tableblock.valign-top {
+ vertical-align: top;
+}
+th.tableblock.valign-middle, td.tableblock.valign-middle {
+ vertical-align: middle;
+}
+th.tableblock.valign-bottom, td.tableblock.valign-bottom {
+ vertical-align: bottom;
+}
+
+
+/*
+ * manpage specific
+ *
+ * */
+
+body.manpage h1 {
+ padding-top: 0.5em;
+ padding-bottom: 0.5em;
+ border-top: 2px solid silver;
+ border-bottom: 2px solid silver;
+}
+body.manpage h2 {
+ border-style: none;
+}
+body.manpage div.sectionbody {
+ margin-left: 3em;
+}
+
+@media print {
+ body.manpage div#toc { display: none; }
+}
+
+
+</style>
+<script type="text/javascript">
+/*<+'])');
+ // Function that scans the DOM tree for header elements (the DOM2
+ // nodeIterator API would be a better technique but not supported by all
+ // browsers).
+ var iterate = function (el) {
+ for (var i = el.firstChild; i != null; i = i.nextSibling) {
+ if (i.nodeType == 1 /* Node.ELEMENT_NODE */) {
+ var mo = re.exec(i.tagName);
+ if (mo && (i.getAttribute("class") || i.getAttribute("className")) != "float") {
+ result[result.length] = new TocEntry(i, getText(i), mo[1]-1);
+ }
+ iterate(i);
+ }
+ }
+ }
+ iterate(el);
+ return result;
+ }
+
+ var toc = document.getElementById("toc");
+ if (!toc) {
+ return;
+ }
+
+ // Delete existing TOC entries in case we're reloading the TOC.
+ var tocEntriesToRemove = [];
+ var i;
+ for (i = 0; i < toc.childNodes.length; i++) {
+ var entry = toc.childNodes[i];
+ if (entry.nodeName.toLowerCase() == 'div'
+ && entry.getAttribute("class")
+ && entry.getAttribute("class").match(/^toclevel/))
+ tocEntriesToRemove.push(entry);
+ }
+ for (i = 0; i < tocEntriesToRemove.length; i++) {
+ toc.removeChild(tocEntriesToRemove[i]);
+ }
+
+ // Rebuild TOC entries.
+ var entries = tocEntries(document.getElementById("content"), toclevels);
+ for (var i = 0; i < entries.length; ++i) {
+ var entry = entries[i];
+ if (entry.element.id == "")
+ entry.element.id = "_toc_" + i;
+ var a = document.createElement("a");
+ a.href = "#" + entry.element.id;
+ a.appendChild(document.createTextNode(entry.text));
+ var div = document.createElement("div");
+ div.appendChild(a);
+ div.className = "toclevel" + entry.toclevel;
+ toc.appendChild(div);
+ }
+ if (entries.length == 0)
+ toc.parentNode.removeChild(toc);
+},
+
+
+/////////////////////////////////////////////////////////////////////
+// Footnotes generator
+/////////////////////////////////////////////////////////////////////
+
+/* Based on footnote generation code from:
+ * http://www.brandspankingnew.net/archive/2005/07/format_footnote.html
+ */
+
+footnotes: function () {
+ // Delete existing footnote entries in case we're reloading the footnodes.
+ var i;
+ var noteholder = document.getElementById("footnotes");
+ if (!noteholder) {
+ return;
+ }
+ var entriesToRemove = [];
+ for (i = 0; i < noteholder.childNodes.length; i++) {
+ var entry = noteholder.childNodes[i];
+ if (entry.nodeName.toLowerCase() == 'div' && entry.getAttribute("class") == "footnote")
+ entriesToRemove.push(entry);
+ }
+ for (i = 0; i < entriesToRemove.length; i++) {
+ noteholder.removeChild(entriesToRemove[i]);
+ }
+
+ // Rebuild footnote entries.
+ var cont = document.getElementById("content");
+ var spans = cont.getElementsByTagName("span");
+ var refs = {};
+ var n = 0;
+ for (i=0; i<spans.length; i++) {
+ if (spans[i].className == "footnote") {
+ n++;
+ var note = spans[i].getAttribute("data-note");
+ if (!note) {
+ // Use [\s\S] in place of . so multi-line matches work.
+ // Because JavaScript has no s (dotall) regex flag.
+ note = spans[i].innerHTML.match(/\s*\[([\s\S]*)]\s*/)[1];
+ spans[i].innerHTML =
+ "[<a id='_footnoteref_" + n + "' href='#_footnote_" + n +
+ "' title='View footnote' class='footnote'>" + n + "</a>]";
+ spans[i].setAttribute("data-note", note);
+ }
+ noteholder.innerHTML +=
+ "<div class='footnote' id='_footnote_" + n + "'>" +
+ "<a href='#_footnoteref_" + n + "' title='Return to text'>" +
+ n + "</a>. " + note + "</div>";
+ var id =spans[i].getAttribute("id");
+ if (id != null) refs["#"+id] = n;
+ }
+ }
+ if (n == 0)
+ noteholder.parentNode.removeChild(noteholder);
+ else {
+ // Process footnoterefs.
+ for (i=0; i<spans.length; i++) {
+ if (spans[i].className == "footnoteref") {
+ var href = spans[i].getElementsByTagName("a")[0].getAttribute("href");
+ href = href.match(/#.*/)[0]; // Because IE return full URL.
+ n = refs[href];
+ spans[i].innerHTML =
+ "[<a href='#_footnote_" + n +
+ "' title='View footnote' class='footnote'>" + n + "</a>]";
+ }
+ }
+ }
+},
+
+install: function(toclevels) {
+ var timerId;
+
+ function reinstall() {
+ asciidoc.footnotes();
+ if (toclevels) {
+ asciidoc.toc(toclevels);
+ }
+ }
+
+ function reinstallAndRemoveTimer() {
+ clearInterval(timerId);
+ reinstall();
+ }
+
+ timerId = setInterval(reinstall, 500);
+ if (document.addEventListener)
+ document.addEventListener("DOMContentLoaded", reinstallAndRemoveTimer, false);
+ else
+ window.onload = reinstallAndRemoveTimer;
+}
+
+}
+asciidoc.install();
+/*]]>*/
+</script>
+</head>
+<body class="article">
+<div id="header">
+<h1>Cruft packs</h1>
+</div>
+<div id="content">
+<div id="preamble">
+<div class="sectionbody">
+<div class="paragraph"><p>The cruft packs feature offer an alternative to Git’s traditional mechanism of
+removing unreachable objects. This document provides an overview of Git’s
+pruning mechanism, and how a cruft pack can be used instead to accomplish the
+same.</p></div>
+</div>
+</div>
+<div class="sect1">
+<h2 id="_background">Background</h2>
+<div class="sectionbody">
+<div class="paragraph"><p>To remove unreachable objects from your repository, Git offers <code>git repack -Ad</code>
+(see <a href="../git-repack.html">git-repack(1)</a>). Quoting from the documentation:</p></div>
+<div class="quoteblock">
+<div class="content">[…] unreachable objects in a previous pack become loose, unpacked objects,
+instead of being left in the old pack. […] loose unreachable objects will be
+pruned according to normal expiry rules with the next <em>git gc</em> invocation.</div>
+<div class="attribution">
+</div></div>
+<div class="paragraph"><p>Unreachable objects aren’t removed immediately, since doing so could race with
+an incoming push which may reference an object which is about to be deleted.
+Instead, those unreachable objects are stored as loose objects and stay that way
+until they are older than the expiration window, at which point they are removed
+by <a href="../git-prune.html">git-prune(1)</a>.</p></div>
+<div class="paragraph"><p>Git must store these unreachable objects loose in order to keep track of their
+per-object mtimes. If these unreachable objects were written into one big pack,
+then either freshening that pack (because an object contained within it was
+re-written) or creating a new pack of unreachable objects would cause the pack’s
+mtime to get updated, and the objects within it would never leave the expiration
+window. Instead, objects are stored loose in order to keep track of the
+individual object mtimes and avoid a situation where all cruft objects are
+freshened at once.</p></div>
+<div class="paragraph"><p>This can lead to undesirable situations when a repository contains many
+unreachable objects which have not yet left the grace period. Having large
+directories in the shards of <code>.git/objects</code> can lead to decreased performance in
+the repository. But given enough unreachable objects, this can lead to inode
+starvation and degrade the performance of the whole system. Since we
+can never pack those objects, these repositories often take up a large amount of
+disk space, since we can only zlib compress them, but not store them in delta
+chains.</p></div>
+</div>
+</div>
+<div class="sect1">
+<h2 id="_cruft_packs">Cruft packs</h2>
+<div class="sectionbody">
+<div class="paragraph"><p>A cruft pack eliminates the need for storing unreachable objects in a loose
+state by including the per-object mtimes in a separate file alongside a single
+pack containing all loose objects.</p></div>
+<div class="paragraph"><p>A cruft pack is written by <code>git repack --cruft</code> when generating a new pack.
+<a href="../git-pack-objects.html">git-pack-objects(1)</a>'s <code>--cruft</code> option. Note that <code>git repack --cruft</code>
+is a classic all-into-one repack, meaning that everything in the resulting pack is
+reachable, and everything else is unreachable. Once written, the <code>--cruft</code>
+option instructs <code>git repack</code> to generate another pack containing only objects
+not packed in the previous step (which equates to packing all unreachable
+objects together). This progresses as follows:</p></div>
+<div class="olist arabic"><ol class="arabic">
+<li>
+<p>
+Enumerate every object, marking any object which is (a) not contained in a
+ kept-pack, and (b) whose mtime is within the grace period as a traversal
+ tip.
+</p>
+</li>
+<li>
+<p>
+Perform a reachability traversal based on the tips gathered in the previous
+ step, adding every object along the way to the pack.
+</p>
+</li>
+<li>
+<p>
+Write the pack out, along with a <code>.mtimes</code> file that records the per-object
+ timestamps.
+</p>
+</li>
+</ol></div>
+<div class="paragraph"><p>This mode is invoked internally by <a href="../git-repack.html">git-repack(1)</a> when instructed to
+write a cruft pack. Crucially, the set of in-core kept packs is exactly the set
+of packs which will not be deleted by the repack; in other words, they contain
+all of the repository’s reachable objects.</p></div>
+<div class="paragraph"><p>When a repository already has a cruft pack, <code>git repack --cruft</code> typically only
+adds objects to it. An exception to this is when <code>git repack</code> is given the
+<code>--cruft-expiration</code> option, which allows the generated cruft pack to omit
+expired objects instead of waiting for <a href="../git-gc.html">git-gc(1)</a> to expire those objects
+later on.</p></div>
+<div class="paragraph"><p>It is <a href="../git-gc.html">git-gc(1)</a> that is typically responsible for removing expired
+unreachable objects.</p></div>
+</div>
+</div>
+<div class="sect1">
+<h2 id="_caution_for_mixed_version_environments">Caution for mixed-version environments</h2>
+<div class="sectionbody">
+<div class="paragraph"><p>Repositories that have cruft packs in them will continue to work with any older
+version of Git. Note, however, that previous versions of Git which do not
+understand the <code>.mtimes</code> file will use the cruft pack’s mtime as the mtime for
+all of the objects in it. In other words, do not expect older (pre-cruft pack)
+versions of Git to interpret or even read the contents of the <code>.mtimes</code> file.</p></div>
+<div class="paragraph"><p>Note that having mixed versions of Git GC-ing the same repository can lead to
+unreachable objects never being completely pruned. This can happen under the
+following circumstances:</p></div>
+<div class="ulist"><ul>
+<li>
+<p>
+An older version of Git running GC explodes the contents of an existing
+ cruft pack loose, using the cruft pack’s mtime.
+</p>
+</li>
+<li>
+<p>
+A newer version running GC collects those loose objects into a cruft pack,
+ where the .mtime file reflects the loose object’s actual mtimes, but the
+ cruft pack mtime is "now".
+</p>
+</li>
+</ul></div>
+<div class="paragraph"><p>Repeating this process will lead to unreachable objects not getting pruned as a
+result of repeatedly resetting the objects' mtimes to the present time.</p></div>
+<div class="paragraph"><p>If you are GC-ing repositories in a mixed version environment, consider omitting
+the <code>--cruft</code> option when using <a href="../git-repack.html">git-repack(1)</a> and <a href="../git-gc.html">git-gc(1)</a>, and
+leaving the <code>gc.cruftPacks</code> configuration unset until all writers understand
+cruft packs.</p></div>
+</div>
+</div>
+<div class="sect1">
+<h2 id="_alternatives">Alternatives</h2>
+<div class="sectionbody">
+<div class="paragraph"><p>Notable alternatives to this design include:</p></div>
+<div class="ulist"><ul>
+<li>
+<p>
+The location of the per-object mtime data, and
+</p>
+</li>
+<li>
+<p>
+Storing unreachable objects in multiple cruft packs.
+</p>
+</li>
+</ul></div>
+<div class="paragraph"><p>On the location of mtime data, a new auxiliary file tied to the pack was chosen
+to avoid complicating the <code>.idx</code> format. If the <code>.idx</code> format were ever to gain
+support for optional chunks of data, it may make sense to consolidate the
+<code>.mtimes</code> format into the <code>.idx</code> itself.</p></div>
+<div class="paragraph"><p>Storing unreachable objects among multiple cruft packs (e.g., creating a new
+cruft pack during each repacking operation including only unreachable objects
+which aren’t already stored in an earlier cruft pack) is significantly more
+complicated to construct, and so aren’t pursued here. The obvious drawback to
+the current implementation is that the entire cruft pack must be re-written from
+scratch.</p></div>
+</div>
+</div>
+</div>
+<div id="footnotes"><hr /></div>
+<div id="footer">
+<div id="footer-text">
+Last updated
+ 2022-06-03 15:24:31 PDT
+</div>
+</div>
+</body>
+</html>
diff --git a/technical/cruft-packs.txt b/technical/cruft-packs.txt new file mode 100644 index 000000000..d81f3a898 --- /dev/null +++ b/technical/cruft-packs.txt @@ -0,0 +1,123 @@ += Cruft packs + +The cruft packs feature offer an alternative to Git's traditional mechanism of +removing unreachable objects. This document provides an overview of Git's +pruning mechanism, and how a cruft pack can be used instead to accomplish the +same. + +== Background + +To remove unreachable objects from your repository, Git offers `git repack -Ad` +(see linkgit:git-repack[1]). Quoting from the documentation: + +[quote] +[...] unreachable objects in a previous pack become loose, unpacked objects, +instead of being left in the old pack. [...] loose unreachable objects will be +pruned according to normal expiry rules with the next 'git gc' invocation. + +Unreachable objects aren't removed immediately, since doing so could race with +an incoming push which may reference an object which is about to be deleted. +Instead, those unreachable objects are stored as loose objects and stay that way +until they are older than the expiration window, at which point they are removed +by linkgit:git-prune[1]. + +Git must store these unreachable objects loose in order to keep track of their +per-object mtimes. If these unreachable objects were written into one big pack, +then either freshening that pack (because an object contained within it was +re-written) or creating a new pack of unreachable objects would cause the pack's +mtime to get updated, and the objects within it would never leave the expiration +window. Instead, objects are stored loose in order to keep track of the +individual object mtimes and avoid a situation where all cruft objects are +freshened at once. + +This can lead to undesirable situations when a repository contains many +unreachable objects which have not yet left the grace period. Having large +directories in the shards of `.git/objects` can lead to decreased performance in +the repository. But given enough unreachable objects, this can lead to inode +starvation and degrade the performance of the whole system. Since we +can never pack those objects, these repositories often take up a large amount of +disk space, since we can only zlib compress them, but not store them in delta +chains. + +== Cruft packs + +A cruft pack eliminates the need for storing unreachable objects in a loose +state by including the per-object mtimes in a separate file alongside a single +pack containing all loose objects. + +A cruft pack is written by `git repack --cruft` when generating a new pack. +linkgit:git-pack-objects[1]'s `--cruft` option. Note that `git repack --cruft` +is a classic all-into-one repack, meaning that everything in the resulting pack is +reachable, and everything else is unreachable. Once written, the `--cruft` +option instructs `git repack` to generate another pack containing only objects +not packed in the previous step (which equates to packing all unreachable +objects together). This progresses as follows: + + 1. Enumerate every object, marking any object which is (a) not contained in a + kept-pack, and (b) whose mtime is within the grace period as a traversal + tip. + + 2. Perform a reachability traversal based on the tips gathered in the previous + step, adding every object along the way to the pack. + + 3. Write the pack out, along with a `.mtimes` file that records the per-object + timestamps. + +This mode is invoked internally by linkgit:git-repack[1] when instructed to +write a cruft pack. Crucially, the set of in-core kept packs is exactly the set +of packs which will not be deleted by the repack; in other words, they contain +all of the repository's reachable objects. + +When a repository already has a cruft pack, `git repack --cruft` typically only +adds objects to it. An exception to this is when `git repack` is given the +`--cruft-expiration` option, which allows the generated cruft pack to omit +expired objects instead of waiting for linkgit:git-gc[1] to expire those objects +later on. + +It is linkgit:git-gc[1] that is typically responsible for removing expired +unreachable objects. + +== Caution for mixed-version environments + +Repositories that have cruft packs in them will continue to work with any older +version of Git. Note, however, that previous versions of Git which do not +understand the `.mtimes` file will use the cruft pack's mtime as the mtime for +all of the objects in it. In other words, do not expect older (pre-cruft pack) +versions of Git to interpret or even read the contents of the `.mtimes` file. + +Note that having mixed versions of Git GC-ing the same repository can lead to +unreachable objects never being completely pruned. This can happen under the +following circumstances: + + - An older version of Git running GC explodes the contents of an existing + cruft pack loose, using the cruft pack's mtime. + - A newer version running GC collects those loose objects into a cruft pack, + where the .mtime file reflects the loose object's actual mtimes, but the + cruft pack mtime is "now". + +Repeating this process will lead to unreachable objects not getting pruned as a +result of repeatedly resetting the objects' mtimes to the present time. + +If you are GC-ing repositories in a mixed version environment, consider omitting +the `--cruft` option when using linkgit:git-repack[1] and linkgit:git-gc[1], and +leaving the `gc.cruftPacks` configuration unset until all writers understand +cruft packs. + +== Alternatives + +Notable alternatives to this design include: + + - The location of the per-object mtime data, and + - Storing unreachable objects in multiple cruft packs. + +On the location of mtime data, a new auxiliary file tied to the pack was chosen +to avoid complicating the `.idx` format. If the `.idx` format were ever to gain +support for optional chunks of data, it may make sense to consolidate the +`.mtimes` format into the `.idx` itself. + +Storing unreachable objects among multiple cruft packs (e.g., creating a new +cruft pack during each repacking operation including only unreachable objects +which aren't already stored in an earlier cruft pack) is significantly more +complicated to construct, and so aren't pursued here. The obvious drawback to +the current implementation is that the entire cruft pack must be re-written from +scratch. diff --git a/technical/pack-format.html b/technical/pack-format.html index eae23ff1b..2149c9ddb 100644 --- a/technical/pack-format.html +++ b/technical/pack-format.html @@ -1169,6 +1169,44 @@ A trailer, containing a: </div>
</div>
<div class="sect1">
+<h2 id="_pack_mtimes_files_have_the_format">pack-*.mtimes files have the format:</h2>
+<div class="sectionbody">
+<div class="paragraph"><p>All 4-byte numbers are in network byte order.</p></div>
+<div class="ulist"><ul>
+<li>
+<p>
+A 4-byte magic number <em>0x4d544d45</em> (<em>MTME</em>).
+</p>
+</li>
+<li>
+<p>
+A 4-byte version identifier (= 1).
+</p>
+</li>
+<li>
+<p>
+A 4-byte hash function identifier (= 1 for SHA-1, 2 for SHA-256).
+</p>
+</li>
+<li>
+<p>
+A table of 4-byte unsigned integers. The ith value is the
+ modification time (mtime) of the ith object in the corresponding
+ pack by lexicographic (index) order. The mtimes count standard
+ epoch seconds.
+</p>
+</li>
+<li>
+<p>
+A trailer, containing a checksum of the corresponding packfile,
+ and a checksum of all of the above (each having length according
+ to the specified hash function).
+</p>
+</li>
+</ul></div>
+</div>
+</div>
+<div class="sect1">
<h2 id="_multi_pack_index_midx_files_have_the_following_format">multi-pack-index (MIDX) files have the following format:</h2>
<div class="sectionbody">
<div class="paragraph"><p>The multi-pack-index files refer to multiple pack-files and loose objects.</p></div>
@@ -1375,7 +1413,7 @@ the MIDX itself.</p></div> <div id="footer">
<div id="footer-text">
Last updated
- 2022-02-16 17:29:08 PST
+ 2022-06-03 15:24:31 PDT
</div>
</div>
</body>
diff --git a/technical/pack-format.txt b/technical/pack-format.txt index 6d3efb7d1..b520aa9c4 100644 --- a/technical/pack-format.txt +++ b/technical/pack-format.txt @@ -294,6 +294,25 @@ Pack file entry: <+ All 4-byte numbers are in network order. +== pack-*.mtimes files have the format: + +All 4-byte numbers are in network byte order. + + - A 4-byte magic number '0x4d544d45' ('MTME'). + + - A 4-byte version identifier (= 1). + + - A 4-byte hash function identifier (= 1 for SHA-1, 2 for SHA-256). + + - A table of 4-byte unsigned integers. The ith value is the + modification time (mtime) of the ith object in the corresponding + pack by lexicographic (index) order. The mtimes count standard + epoch seconds. + + - A trailer, containing a checksum of the corresponding packfile, + and a checksum of all of the above (each having length according + to the specified hash function). + == multi-pack-index (MIDX) files have the following format: The multi-pack-index files refer to multiple pack-files and loose objects. |